William Peverel was among the Conqueror's closest followers, granted the new castle at Nottingham in 1068 as William subdued the Midlands. By 1086, Peverel held extensive lands in Nottinghamshire and Derbyshire, and his castle at Castleton, recorded in the Domesday Book as standing at 'Pechesers,' a name translated variously as 'Peak's Tail' or 'Peak's Arse,' was the seat of his feudal barony. What made Peveril unusual was that it appears to have been built in stone from the outset, while most early Norman castles were timber. The site's natural defences, steep slopes falling away on three sides, made it a formidable position. The winding approach from the north was the only practical route up, and the castle's prominence ensured it served as a visible symbol of Norman power over the conquered landscape. Sections of the curtain walls still show herringbone stonework, the distinctive opus spicatum of Norman construction, alongside repairs from every subsequent century.

The Peverel family lost the castle in 1155 when the younger William Peverel's estates were confiscated by Henry II. The king visited three times, once hosting Malcolm IV of Scotland, and during the baronial revolt of 1173-1174, he spent 116 pounds fortifying Peveril and Bolsover. The garrison was expanded from two watchmen and a porter to a force of twenty knights shared with Bolsover and Nottingham. Henry II built the keep between 1175 and 1177 at a cost of 184 pounds, a significant investment given his annual income of roughly 10,000 pounds. The castle's political life grew tangled. The Earls of Derby claimed it through marriage to Peverel's daughter Margaret. William de Ferrers, the fourth Earl, paid 2,000 marks for the lordship of the Peak in 1199 but the Crown kept the castle. The closest Peveril came to actual battle was in 1216, when King John gave it to Ferrers but the castellan Brian de Lisle refused to hand it over. John authorised force, but de Lisle apparently negotiated rather than fight.

By the late fourteenth century, Peveril had passed to John of Gaunt, the richest nobleman in England and Edward III's third surviving son. Gaunt had no use for a relatively minor Peak District castle when he owned several grander ones. In 1374 he ordered the lead stripped from Peveril's roofs for reuse at Pontefract Castle, effectively signing its death warrant. His son Henry Bolingbroke, who became Henry IV, retained ownership through the Duchy of Lancaster, but administrative functions gradually moved elsewhere. By 1609, a survey found the castle 'very ruinous and serveth for no use.' It housed animals for a time. The Duchy of Lancaster, which still technically owns it, has administered Peveril from the medieval period to the present day, a continuous thread of ownership spanning more than six centuries.

Railways brought tourists to Castleton in the nineteenth century, and the castle became a destination. Walter Scott set his 1823 novel Peveril of the Peak partly amid the ruins. The Duchy undertook stabilisation work, clearing rubble and adding mortar to prevent further collapse without attempting restoration. In 1932, custody passed to the Office of Works, and English Heritage cares for the site today. The keep, square and compact, rises fifteen metres above its base on one side and ten and a half on the other, smaller than the royal keeps at Dover or Scarborough but perfectly proportioned against the hillside. Roman tiles scavenged from the fort of Navio, two miles away, appear in the surviving tower walls, a reminder that this landscape has drawn fortress-builders for millennia. Standing on the keep's remnants, the view across the Hope Valley makes clear why the site was chosen. This was never just a military position. It was a statement, visible for miles, that power had arrived and intended to stay.