By 1165, Henry II had a Bigod problem. Hugh Bigod, Earl of Norfolk, had picked the wrong side during the Anarchy under King Stephen, and the family's four Suffolk castles at Framlingham, Bungay, Walton and Thetford gave them a stranglehold on East Anglia that no king could comfortably ignore. Henry confiscated the lot, then handed Framlingham and Bungay back as a calculated gesture. The real answer was Orford. Henry chose a flat site two miles from the sea, with marshes draining down to the River Ore, and ordered a royal castle built in the middle of Bigod country. The total cost recorded in the Pipe Rolls came to £1,413, with the master mason possibly a man named Alnoth. Caen limestone arrived from Normandy. Timbers came from as far as Scarborough. By 1173, the keep was finished, and the Bigods were boxed in.

The historian R. Allen Brown called it one of the most remarkable keeps in England, and the design is genuinely strange. A 90-foot central tower, circular in cross-section, with three rectangular clasping towers projecting from a 49-foot core. The proportions follow a one-to-the-root-of-two ratio borrowed from contemporary English churches. Broad five-foot staircases climb through ashlar masonry. The best chambers face the morning sun. Doors and carefully placed windows seal out the North Sea draughts. The roof above the upper hall once formed a dome topped by a tall steeple, and the chapel above the entrance has a shape so irregular that historian Stephen Brindle suggests it would normally have been considered unseemly for a room dedicated to God. The model may have been imperial. The banded, angular silhouette resembles the Theodosian Walls of Constantinople, and the keep as a whole may echo a hall John II Komnenos had recently built in the Byzantine capital. Henry was making a statement, and he was making it in the language of empire.

The Bigods rose in revolt in 1173-74 and were crushed. Framlingham was confiscated for good, and Orford settled into the long quiet that follows a successful piece of statecraft. Prince Louis of France took the castle in 1216, invited in by barons fed up with King John. The de Valoines family held it, then the de Uffords by marriage, then in 1336 Edward III handed it in perpetuity to the first Earl of Suffolk. The castle drifted out of royal hands. The port at the foot of the keep grew briefly busier than Ipswich, then declined as the shingle of Orford Ness crept south and choked the harbour. By the 1840s the curtain walls and mural towers had been quarried away, and only the keep still stood, a strange polygonal silhouette above the marshes. The 18th-century government refused to let owners demolish it because Dutch shipping used the tower as a landmark for avoiding the sandbanks offshore.

Sir Arthur Churchman bought the castle in 1928 and gave it to the Orford Town Trust, which opened it to the public in 1930. During the Second World War the keep was wrapped in barbed wire and Nissen huts, originally for anti-aircraft work but eventually as a radar station with a concrete floor poured in the south-east tower to support the equipment. The wartime fittings came out at the end of the conflict, and in 1962 the castle passed to the Ministry of Works and then English Heritage. The mudstone walls, called septaria, have been weathering since at least the sixteenth century. In 2022 English Heritage completed a £1 million project to cover the castle in lime render to slow the decay. The upper hall now houses the Orford Museum, with archaeological finds from the surrounding parish. The wild man's century is long gone, but the polygonal silhouette still rises above the marsh exactly where Henry put it.