Bolton Castle was built between 1378 and 1399 by Richard Scrope, the 1st Baron Scrope of Bolton, on the north side of Wensleydale. The contract with the master mason John Lewyn survives from September 1378. The royal licence to crenellate followed in July 1379. The reported cost was 18,000 marks, an enormous sum equivalent to roughly £12,000 in fourteenth-century money. The result was a quadrangular castle - a square arrangement of buildings around a central courtyard with four corner towers - rather than the older motte-and-keep design. Sir Francis Knollys later described it as having the highest walls of any house he had ever seen. The sixteenth-century antiquary John Leland recorded an astronomical clock in the courtyard and a clever flue system that carried smoke from the great hall hearth through tunnels in the walls. The Scrope family lived there for two centuries before Mary arrived.

In 1536 John Scrope, the 8th Baron, threw the castle's weight behind the Pilgrimage of Grace - the great northern Catholic rebellion against Henry VIII's dissolution of the monasteries. When the rising collapsed, Scrope gave sanctuary inside Bolton to Adam Sedbar, the Abbot of nearby Jervaulx Abbey. The king's men came for them both. Scrope escaped west to Skipton. Sedbar was caught, dragged to Tyburn, and hanged. Henry's revenge on the castle was fire: he ordered Bolton torched, and the damage was extensive. Yet within a few years much of it had been repaired and the chastened Sir John resumed his seat in Parliament. The Scropes had decided the long view was worth the price of accommodation, and they kept their castle when other rebellious families lost everything.

Mary spent six months at Bolton, from July 1568 to January 1569. She was given Henry Scrope's own apartments in the south-west tower. Of her retinue of 51 people, only 30 men and 6 ladies-in-waiting could be lodged inside. The rest took rooms in the village. She brought a hairdresser, embroiderer, apothecary, physician, surgeon, and a goldsmith whom Knollys worried might counterfeit seals for forged correspondence. The castle was not equipped to house royalty, so tapestries, rugs, and furniture were borrowed from nearby houses and from Barnard Castle in County Durham. Elizabeth herself sent pewter vessels and a copper kettle. Mary obtained an old cloth of estate from Scotland and hung it above her chair in the great chamber, making it a makeshift throne room. She was allowed to walk the grounds and to hunt. She had her hair dressed daily by Mary Seton. She met local Catholics, for which her keepers were severely reprimanded. Knollys, whom she nicknamed Schoolmaster, taught her English - she had previously spoken only French, Latin, and Scots - and she wrote him a letter in the new language, apologising that she had never used it before.

Local legend says Mary tried to escape and fled toward Leyburn, dropping her shawl on the cliff edge that runs west out of the town. The escarpment is still called The Shawl and remains a favourite walking route with sweeping views down Wensleydale. The story is almost certainly later embellishment, but the name stuck. Mary was moved south to Tutbury in January 1569 and eventually executed at Fotheringhay in 1587 after nineteen years of confinement. Bolton itself survived the English Civil War of the 1640s, was slighted afterwards to render it militarily useless, and was inherited in 1630 by the Powlett family through Mary Scrope, illegitimate daughter of the last Earl of Sunderland. The Powletts became Lords Bolton. The current owner, Thomas Peter Algar Orde-Powlett, the 9th Baron Bolton, inherited from his father in June 2023. The castle has never been sold. It is still a family seat, still partly a ruin and partly restored, and still has the same impossibly tall walls Knollys described in 1568.