When William Rufus rode into the north-west in 1091, he was looking for ground that would hold a castle. He found it at Verterae, a three-acre Roman fort that had guarded the Stainmore Pass for centuries before being abandoned in the fifth. The earthworks were still there, the road that the Romans had run between Carlisle and York still ran past it, and the north side of the site dropped away to the River Eden. Around 1092 Rufus's masons set a motte-and-bailey castle into the northern quarter of the old fort, reusing what the Romans had left. A planned village - Church Brough - grew up alongside, part of the Norman colonisation of the lowlands. The castle was timber on stone foundations, palisade-and-earth where it was not stone. It would not last.

In 1173 William the Lion of Scotland invaded as part of the Great Revolt against Henry II. His army tried Wark and failed, tried Carlisle and failed, took Appleby, and then arrived at Brough. The castle was held by six knights. Six. They put up enough of a fight that William had to take the outer defences before besieging the keep, then set the keep itself on fire to force the garrison out. The chronicler Jordan Fantosme records that one knight kept fighting after the fire - first with spears, then with wooden stakes, until he was finally overwhelmed. William razed what was left using Flemish mercenaries. Henry II's forces caught up with William at the Battle of Alnwick later that year and ended his war. Brough was recovered, and over the 1180s Henry's men - first Theobald de Valoignes, then Hugh de Morville - rebuilt it with a square stone keep set into the bailey wall, where it could brace the outer defences from within.

By the late 1260s the Clifford family controlled Brough, and they kept it for almost four hundred years. Robert Clifford rebuilt the east wall around 1308 and added a new hall and a circular tower of apartments - Clifford's Tower - before dying at Bannockburn in 1314. The region around the castle was raided by the Scots in 1314 and 1319. Around the same time the Cliffords laid out Market Brough along the road overlooked by the castle, hoping to siphon trade and rents from the valley; it received its royal charter in 1330 and quickly eclipsed the older Church Brough. Then came the disastrous Christmas feast of 1521. Henry Clifford's celebration ended in a fire that gutted the castle. For nearly a century and a half, Brough was a ruin.

Lady Anne Clifford was a royalist who somehow kept her northern estates through the English Civil War, protected by powerful Parliamentary friends. Between 1659 and 1661 she rebuilt Brough, doing more restoration here than anywhere else on her properties. Her approach was strikingly traditional: she drew on existing northern castle styles and deliberately tried to recreate twelfth-century features in the keep. She also renamed the keep "the Roman Tower," believing - wrongly, but evocatively - that the Romans had built it. By 1665 the castle had twenty-four fireplaces and she was finally able to spend Christmas there, splitting her time between Clifford's Tower and the keep as work continued. The following year, in 1666, another fire broke out. The keep that Lady Anne had spent her later life restoring became uninhabitable. She died in 1676. The castle was stripped of fittings, then of stonework, and its masonry began collapsing around 1800. In 1921 the ruins were given to the state. English Heritage runs them now, and the view from the keep's top, with the Eden flowing far below and the Pennines piled to the east, is largely the view William Rufus once decided was worth a castle.