Long before the king came running, White Ladies had been a small Augustinian convent of canonesses regular dedicated to St Leonard of Brewood. The earliest documentary reference is from 1186, but the surviving Romanesque work in the ruined nave is typically late 12th century, so the foundation was probably in the 1170s or 1180s. We do not know who founded it. The FitzAlans, the Lacys, or the Zouche family are the strongest candidates, but no charter survives that names a benefactor. The priory was small: normally just a prioress and five canonesses, plus servants and chaplains. They were called the White Ladies because the linen rochets they wore over their habits were white. A short ride to the east, in the Staffordshire village of Brewood itself, was a Benedictine house known by contrast as Black Ladies Priory. John Leland, antiquary to Henry VIII, visited shortly after the Dissolution and recorded the place as Cistercian, an error that propagated through later antiquarian writers. The bishop's register makes it clear: White Ladies were Augustinians.

By the late 15th century White Ladies was in financial trouble. Most of its income came from leases granted at fixed rents in earlier centuries, and 16th-century inflation made those rents almost worthless. In 1471 Prioress Joan Shirley had granted a 99-year lease on a property in Overton at six shillings and eightpence a year. Nearly seventy years later, when the Crown finally dissolved the priory in 1536 under the Suppression of Religious Houses Act, that lease still had three decades to run. The Valor Ecclesiasticus of 1535 found total revenues of just thirty-one pounds, one shilling, and fourpence. The convent's dormitory had been reported in bad repair as far back as 1524. When the dissolution came there were only four canonesses left in residence. The site and its lands were eventually bought in 1540 by William Whorwood, Solicitor General to the King. A house was built on the priory ruins, probably incorporating part of the old prioress's lodging. By the early 17th century the estate was being lived in and farmed by Catholic tenants connected to the Giffard family of Chillington.

On 3 September 1651 Oliver Cromwell's army won what he called his crowning mercy at the Battle of Worcester. The young Charles II, twenty-one years old, escaped the field with a small group of officers. They rode all night, north out of Worcestershire and into the wooded country of east Shropshire, looking for shelter. Charles Giffard, a cousin of the Giffard family who owned the White Ladies estate, knew exactly where to take them. They arrived in the dawn of 4 September. George Penderell, a servant living at the house, woke his brothers. Richard Penderell came up from his nearby farm; William Penderell came over from Boscobel House about a mile away. Charles was cut his long hair, given the woodman's clothes off Richard's back, and his face stained brown with walnut juice. The royal party went on without him. Charles spent the day with Richard hiding in nearby Spring Coppice in the rain, trying to figure out how to reach Wales. The attempt to cross the Severn failed because the river was guarded. On 6 September he returned to the area and spent the day in the trees at Boscobel House, perched in a great oak with Major William Careless while a search party passed below. That tree became, in later legend, the Royal Oak.

The Penderell brothers, Richard, William, Humphrey, John and George, were small Catholic farmers who worked for the Giffards. A sixth brother had been killed at Edgehill in 1642 fighting for the King's father. They were exactly the kind of family who paid the highest price in 17th-century England: too poor to matter, too Catholic to be safe, and now in possession of the most dangerous secret in the country. They could have collected a thousand pounds and lived comfortably for life. They chose not to. After the Restoration of 1660, Charles II remembered. The Penderell family was granted a perpetual annuity, paid annually from Crown funds to their descendants, and it is still being paid today. The Catholic Giffards too, eventually, were quietly tolerated. The Royal Oak became a national symbol, planted on coins, embedded in pub names, baked into Restoration mythology. Oak Apple Day, 29 May, was celebrated annually as the king's birthday and the anniversary of his triumphant return to London in 1660.

The post-Reformation house built into the priory was largely lost by the early 19th century, although the artist Edward Williams sketched what was left in 1791. In 1938 Edward Fitzherbert, the 13th Baron Stafford, whose family had eventually inherited the site through marriage from the Giffards, placed White Ladies in the care of the Office of Works, predecessor of English Heritage. What survives now is essentially the late-12th-century priory church: the north wall of the nave and chancel still standing, the layout of the transepts and east end legible on the ground, parts of the cloister visible against the church. The ruin is small, quiet, and open to the public without charge. Boscobel House, a mile to the east, is also under English Heritage and contains a descendant of the Royal Oak; the original was reportedly destroyed by souvenir-hunters in the 18th century. Standing in the roofless nave of White Ladies on a grey afternoon, with the same wind blowing off the same Shropshire fields that the king felt in 1651, it is difficult not to imagine a young man in dirty clothes being hurried into a back room and told to keep quiet.