The de la Pomeroy family had held the feudal barony here since shortly after the Norman Conquest, with a presence on the rolls of the Domesday Book of 1086. Their barony was substantial, almost thirty-two knight's fees by 1166, each roughly equivalent to a manor. Henry Pomeroy enclosed a deer park in 1207, and for the next three centuries the family hunted, married, and held their seat somewhere in the neighbourhood, but not, as it turns out, at the castle. The castle proper does not appear in the records until 1496, when Elizabeth, widow of Richard Pomeroy, was assigned a third share of both castle and capital messuage. They were on separate sites. Archaeological digs between 1980 and 1996 swept away centuries of romantic guesswork: the curtain wall, gatehouse, and St. Margaret's Tower are late fifteenth-century, making Berry Pomeroy one of the last traditional personal castles built in England. Beneath the medieval surface, the diggers found only five sherds of older pottery. Nothing else.

There is one extraordinary survival from the Pomeroy years. In 1978, hidden behind a thick mat of vegetation in the upper storey of the gatehouse, conservators uncovered a wall painting. Even faint and weather-eaten, the subject was identifiable: an Adoration of the Magi, dated on stylistic grounds to around 1490 to 1500, the same window in which the castle was built. Imagine a chamber above the gatehouse where the family heard mass with kings, queens, and a journeying star painted across the wall in their own lifetime, the colours fresh, the gilding bright. Five hundred years of damp and neglect later, it had become a ghost of itself. A pigment enhancement in the published photograph brings it back. Standing in the actual room, all you see is shadow and lime.

In 1547, Sir Thomas Pomeroy sold up. The buyer was Edward Seymour, 1st Duke of Somerset, Lord Protector to the boy-king Edward VI and effectively ruler of England. He may never even have visited his new Devon property. Five years later, he was beheaded for treason and the lands forfeited to the Crown. It took complex dealings until 1558 for his son, also Edward, to recover the castle. Between 1560 and 1580 this second Edward swept away the older Pomeroy interior buildings and put up a fashionable four-storey Tudor house at the north end of the courtyard. A grandson added the North Range around 1600. The dynasty kept building. By 1659 a Royalist Sir Edward Seymour, 2nd Baronet, had died at the castle after weathering the Civil War, Parliamentarian raids, and the sequestration of his estates. His son, the 3rd Baronet, did Royal Navy service after the Restoration and became Vice-Admiral and MP for Totnes. When he died in 1688, an inventory listed roughly fifty rooms, by then likely in poor repair after the staggering expense of his father's cause.

The 4th Baronet, Sir Edward Seymour, was already fifty-five when he inherited. He had been a serious politician, Speaker of the House of Commons from 1673, and he had also inherited Bradley House in Wiltshire, which was closer to London. He chose Wiltshire. Berry Pomeroy was probably stripped for usable materials to fund the rebuilding of Bradley House, finished in 1710. Lady Anne, his mother, stayed at Berry Pomeroy until her death in 1694, and after that the castle was abandoned. The vicar John Prince, who had known it in its prime, wrote of it in 1701: "The apartments within were very splendid; especially the dining room, which was adorn'd, besides paint, with statues and figures cut in alabaster... 'tis now demolished, and all this glory lieth in the dust."

By the late eighteenth century, the picturesque movement had discovered that ruins were beautiful. Berry Pomeroy, by then the haunt of jackdaws and tangled with ivy, fit the mood exactly, and visitors began to come. Around 1830 the Duke of Somerset paid for some of the crumbling walls to be repaired, one of the earliest examples in England of architectural conservation work done because tourists might be hurt. With the ruin came the ghosts. Two female apparitions are said to haunt the castle: the White Lady and the Blue Lady. The Blue Lady is said to be the daughter of a Norman lord who beckons travellers from her tower; those who go to help are said to fall to their deaths. She is also said to wander the dungeons mourning a baby she murdered after being raped by her own father. These are dark stories, told carefully here because they describe an imagined woman trapped by terrible cruelty, not a curiosity to be entertained. The wishing tree is gentler: a beech against whose trunk visitors whispered wishes, then walked backwards around it several times. The castle has appeared on BBC's Timewatch and on Most Haunted, and it has been a fictional setting for Elizabeth Goudge's novel The Castle on the Hill and an episode of Channel 4's Five Go Mad in Dorset. Today it is a Grade I listed building, still owned by John Seymour, 19th Duke of Somerset, and administered by English Heritage. The drive still winds through the wood. The shell still rises out of the limestone above the brook.