Shell keeps were a brief architectural fashion of the 12th and early 13th centuries. The basic idea was simple: take an existing wooden motte-and-bailey castle, replace the outer palisade with a stone wall, fill the interior with domestic buildings clustered against the inside of that wall, and you have a fortified residence that looks more like a luxurious courtyard than a barracks. The buildings curve to fit the geometry, an extreme example of the trend toward circular planning. At Restormel the wall is up to 2.4 metres thick, sunk unusually deep into the original motte, and surrounded by a ditch four metres deep and fifteen metres wide. A square gate tower, now ruined, guards the entrance. The slate for the construction was quarried from the scarp face just to the north-east, so the castle is built quite literally from the hill it sits on.

The first fortification at Restormel went up around 1100, probably built as a hunting lodge by Baldwin Fitz Turstin, the local Norman sheriff. The transformation came in the 13th century. Robert de Cardinham, lord of the manor between 1192 and 1225, rebuilt the inner curtain walls and converted the gatehouse to stone. Then in 1270 the castle passed to Richard of Cornwall, brother of Henry III. Richard died the next year and his son Edmund made Restormel his main administrative base. Edmund called it his duchy palace, and the description was not entirely vain. He fitted the castle with luxurious quarters, built inner chambers along the curved wall, and installed something genuinely unusual for the period: piped running water, pressurized from a nearby spring. While most medieval lords still hauled buckets, Edmund's household turned a tap. From here the earl oversaw the stannary administration, supervising the tin mines that made Cornwall rich.

After Edmund's death in 1300, Restormel reverted to the Crown and became one of the 17 antiqua maneria of the Duchy of Cornwall. With an absent lord, it slid quietly into disrepair. By a 1337 survey it was already failing, briefly patched up by order of the Black Prince, then neglected again after his death in 1376. The castle saw armed conflict only once in its 800-year history. During the English Civil War a Parliamentary garrison occupied the ruins and made basic repairs. On 21 August 1644, Sir Richard Grenville, a Royalist commander who had been MP for Fowey before the war, stormed the castle as part of a manoeuvre to encircle Parliamentary forces. By a 1649 Parliamentary survey it was recorded as utterly ruined, with only the outer walls standing. The official judgment: too damaged to repair, too worthless to demolish. So they left it.

By the 19th century, neglect had aged into atmosphere. The French writer Henri-François-Alphonse Esquiros visited in 1865 and described the place as what the English call a romantic scene, with ivy climbing the circular wall and visitors arriving for picnics and parties of pleasure. In 1846 Queen Victoria and Prince Albert came up the Fowey in their yacht and walked the wall walk, the same circuit Edmund's household had used six centuries earlier. In 1925, Prince Edward, Duke of Cornwall and briefly later King Edward VIII, entrusted the ruin to the Office of Works. A 1971 proposal to restore the castle was dropped after public opposition. It was designated a scheduled monument a decade later. The Great Western Railway named a Castle class locomotive after it in 1927, number 5010, withdrawn from service in 1959 but still part of the place's memory. The ring of stone keeps doing what it has done for nine hundred years: holding its shape, indifferent to who walks through it.