Earlier Henrician castles favoured rounded bastions, semi-circular walls smoothed against the angle of incoming shot. Yarmouth chose geometry instead. The arrow-head let defenders sweep their own walls with crossfire, eliminating the dead ground where attackers could hide. It was a deliberate copy of Italian fortification ideas just beginning to circulate north of the Alps, and it would soon become standard everywhere. Walls of ashlar stone (red brick patches the south side) enclose a courtyard once filled with buildings. Half of that courtyard was filled in not long after construction to create a solid artillery platform, where eight heavy guns could fire over the sea without obstruction. The thinking was straightforward: more guns, fewer dead angles, more time before the enemy got close.

Yarmouth Castle's recorded history is mostly a chronicle of decay punctuated by panic. By 1586 it was already in poor condition. A 1623 survey found a garrison of just four gunners and the captain, with the buildings declared 'ruinous'. The 1632 repairs raised the parapets and added storage rooms, partly with stone scavenged from nearby Sandown Castle, whose walls had been claimed by the sea. When the Civil War broke out in 1642, a Royalist captain named Barnaby Burley held the castle for King Charles with a tiny garrison; the Crown then tried to offload the running costs onto the town of Yarmouth, which mostly declined. By 1688, when James II faced William of Orange's invasion, the locals and the garrison both sided with William and quietly prevented their commander, Robert Holmes, from holding the place for the wrong king.

Things stayed quiet for over a century. The harbour silted up. The design fell behind the times. But the Crimean War of the 1850s set off a new invasion scare, and in 1855 the entire south coast of England was rearmed. Yarmouth Castle got considerable repairs that year: four naval guns went onto the artillery platform, mounted on traversing rails that still survive, weathered and rusted into place. Visitors can walk along those rails today and trace where the guns swung to cover the Solent. A regular county army unit was stationed there, the first time in decades the place had a proper garrison. It would be the castle's last serious military upgrade.

After 1855, the role kept shrinking. The coastguard moved in as a signalling station in 1898. In 1912, parts of the castle were leased to the Pier Hotel - later the George Hotel - which still occupies what was once the moat on the south and east sides. The Office of Works took over in 1913 and began the careful repairs that would define the modern visitor experience. Both world wars pulled the castle back into use as a military outpost, but it was finally retired in the 1950s. Today it is Grade I listed and a scheduled monument, managed by English Heritage. The barrel-vaulted lodgings on the ground floor, the Long Room running across the upper storey with its original 17th-century roof intact, the sharp angle of the arrow-head bastion - all still here, still pointing inland, still ready for an attack that never came.