Holt is thought to derive from the Anglo-Saxon word for woodland and Holt is located on wooded high ground of the Cromer ridge at the crossing point of two ancient by-ways and as such was a natural point for a settlement to grow.

The town has a mention in the great survey of 1086 known as the Domesday Book.