Robert of Mortain - William the Conqueror's half-brother - was probably the man responsible for managing the construction. Once it was built, he became the owner. The castle was a classic Norman motte-and-bailey: a high circular mound (the motte) topped by a wooden keep, attached to a lower enclosed courtyard (the bailey) ringed by a wooden palisade. The whole complex was surrounded by earthworks and ringed by twin moats - some of which still hold water. The castle was carefully positioned next to a Crown deer park for hunting; a fossarius (a specialised ditch digger) is recorded as being employed here in 1086. Radiocarbon dating of organic remains within the motte confirms a post-1066 construction date. The old Anglo-Saxon manorial centre was moved to the new castle, and the village of Berkhamsted apparently shifted west from Northchurch to be closer to the Norman power base.

Robert of Mortain's son William rebelled against Henry I and lost the castle to confiscation. Henry granted Berkhamsted to his chancellor Ranulf in the early 1100s. Ranulf had a short tenure: in 1123, riding with Henry to the castle, he became overly exhilarated at the view from the hill above the valley and fell off his horse, dying from his injuries. The castle was subsequently given by Henry II to Thomas Becket when Becket became chancellor in 1155. Becket extended the castle to accommodate his enormous household, rebuilt it in stone, and added a shell keep on the motte and an outer stone wall. He fell from royal favour in 1164 and was killed in Canterbury Cathedral in 1170, but his renovation of Berkhamsted remained the bones of the structure that visitors see today. Henry II liked the castle and used it extensively after Becket's death - perhaps an uncomfortable memorial to the friendship that had ended in murder.

In December 1216 the future Louis VIII of France besieged the castle with siege engines - probably trebuchets - and attacked it for twenty days, throwing what chroniclers called innumerable damnable stones at the defenders. The siege was part of the First Barons' War, in which rebellious English barons had invited the French prince to take the throne from King John. John had died in October; his nine-year-old son Henry III was the legitimate heir but lacked military strength. The earthworks visible today around the outside of the castle walls are probably the firing platforms Louis built for his trebuchets. Having put up a strong defence, the Berkhamsted garrison was eventually allowed to surrender on honourable terms - to leave with their weapons and armour intact. The following year, royal forces defeated the rebels and the castle returned to royal hands. A 13th-century crossbow stave was dug up from the inner moat during clearance work in 1930, thought to be a relic of the siege. It is now in the British Museum.

Henry III gave the castle to his brother Richard, Earl of Cornwall, in 1227. Berkhamsted was Richard's favourite home, partly because it was conveniently close to London. He built an impressive three-storey tower onto the property in 1254 and restored much of the rest. Berkhamsted became the centre of the earldom's administration: Richard's nine stewards submitted their annual financial reports here. The chroniclers of nearby Dunstable complained that Richard's building works required so many carts to carry the timber that local trade in other goods was badly affected. Richard died at the castle in 1272. From him the castle passed through generations of royal owners. King Edward III gave it to his son Edward the Black Prince in 1337 - tying the castle into the new Duchy of Cornwall, where it has formally remained ever since. The Black Prince held King John II of France here as a prisoner after the Battle of Poitiers in 1356. Geoffrey Chaucer, later famous for The Canterbury Tales, supervised renovation work on the castle in his role as Clerk of the King's Works under Henry IV.

By the late 1400s the castle was unfashionable. By the mid-1500s it was in ruins. Stone was carted away to build Berkhamsted Place, the local school, and other buildings in the town. Then in the 1830s the engineers of the London and Birmingham Railway designed a route that would have run straight through the castle. The 1833 Act of Parliament that authorised the railway included a clause specifically protecting the castle - the first building in Britain to receive statutory protection from development. The railway was forced to take a route across the valley floor instead. Even so, the chosen route passed through the outer fortifications, and the outer moat was partially infilled and the gatehouse demolished in 1834. The first protection was incomplete, but it set the principle: that ancient monuments could not simply be erased for convenience. In 1929 the Office of Works acquired the castle from the Duchy of Cornwall, and English Heritage manages it today. Entry is free. Local volunteers staff the site. The motte and the moats survive in remarkable condition, and the historian Isobel Thompson considers Berkhamsted one of the best surviving motte-and-bailey castles in England.