The castle's origins trace to the aftermath of the Norman Conquest. In 1069, the Saxon rebel Eadric the Wild led a failed uprising that culminated in defeat at the Battle of Stafford. William the Conqueror rewarded his follower Robert de Stafford with 131 manors, predominantly in Staffordshire, and Robert built a wooden castle on the hilltop sometime in the 1070s. The Domesday Book of 1086 records the family's extensive holdings. From this base, Robert controlled and taxed the native Anglo-Saxon population. His descendants, through the female line, took the name de Stafford and rose steadily through the ranks of English nobility: feudal barons by custom, Barons Stafford by writ in 1299, Earls of Stafford in 1351. Ralph de Stafford, the first Earl, was one of Edward III's leading commanders in the Hundred Years' War. In 1347, he commissioned a master mason to replace the wooden castle with stone.

The stone castle reached its peak under Humphrey Stafford, first Duke of Buckingham, who was killed at the Battle of Northampton in 1460 during the Wars of the Roses. His grandson Henry, the second Duke, initially supported Richard III but then switched allegiance to the invading Henry Tudor in 1483. The rebellion failed, and Henry paid with his head. His son Edward, the third Duke, was restored to favour by the grateful Henry VII, but made the fatal error of possessing royal blood under Henry VIII. The king had him executed in 1521 on trumped-up treason charges. Three dukes, three violent ends. The pattern illustrates how proximity to the English throne was often a death sentence. The later Staffords recovered a small parcel of land, but never their former grandeur. The castle slid into decay.

When the English Civil War erupted in 1643, Lady Isabel Stafford, a devout Catholic and Royalist, refused to surrender the castle to Parliamentary forces. Colonel William Brereton rode up and demanded her capitulation. She refused. The Parliamentarians burned the outhouses, but from inside the walls, Lady Stafford's garrison shot back, killing soldiers and horses. A relief column under Colonel Hastings lifted the siege briefly, but the Royalist garrison eventually fled when they learned a large Parliamentary army with siege cannon was approaching. On 22 December 1643, the Parliamentarian Committee of Stafford ordered the castle demolished. When the diarist Celia Fiennes passed through in 1698, she noted only a grassy hill with overgrown trenches where a castle had once stood.

By the 1790s, a single low wall was all that remained visible above ground. Workmen hired to shore it up accidentally discovered buried basements and foundations extending from the wall. Sir William Jerningham ordered the entire mound excavated and cleared. Beginning in 1813, the family rebuilt part of the castle in the Gothic Revival style, intending it as a habitable residence, not a folly, despite what some critics called it. The Jerninghams were motivated partly by ambition, partly by their elevation to the peerage. But funds ran short, and the project stalled. The rebuilt keep was occupied into the twentieth century, but after surrounding woodland was felled in the post-war years, high winds began to strip masonry from the exposed towers. By 1949 the structure was declared unsafe, and the last caretakers moved out. Today the ruins stand on a heritage trail opened in 1988, the hilltop offering views across the Staffordshire countryside that once made this position worth fighting over.