The Talbot family had been lords of Worksop since the 14th century, and in 1568 their manor house was where Mary, Queen of Scots was held during one of her long captivities in English custody. The house she knew was already old. In the 1580s the wealthy George Talbot, 6th Earl of Shrewsbury - husband of Bess of Hardwick - had a new one built on the same site, almost certainly to designs by Robert Smythson. It was one of the great prodigy houses of the age, a deliberate piece of architectural showing-off, much admired for its long gallery on the top floor and for the dated chimneypiece of 1585. Smythson at the same time designed the smaller Worksop Manor Lodge nearby, a building so striking it was compared to the Medici villa at Pratolino. The lodge survived in something close to its original form until it too burned, in 2007.

After the Union of the Crowns in 1603, the Stuart court used Worksop Manor on its journeys south. King James VI and I stayed here that summer on his way to take the English throne. Anne of Denmark arrived in June, holding court on the king's birthday with kitchens that recorded Polish and Bolognese sausages, Westphalia bacon, and two Frenchmen employed simply to fold napkins. She gave a jewel to a young William Cecil and tied it in his ear, then watched him dance with the seven-year-old Princess Elizabeth. The crowd of hangers-on grew so disorderly that the Duke of Lennox and the Earls of Shrewsbury and Cumberland issued a formal proclamation telling them to leave or behave. The young Prince Charles, future Charles I, stayed here in August 1604, where his physician noted four days of music and the boy's initiation into the proper management of a deer hunt.

The house passed by marriage to the Dukes of Norfolk and stayed in that family for over a century. Thomas Howard, 8th Duke, doubled its size in 1701. Edward Howard, the 9th, refined the gardens. In 1761 the renovated mansion of Mary Howard, Duchess of Norfolk, burned to the ground. James Paine was commissioned to design a replacement - a square palace with an enormous central courtyard that would, had it been completed, have been one of the largest country houses ever built in England. Only one wing was finished by 1767, when the money ran out. That wing alone was palatial. The 9th Duke died in 1777, and the estate passed to a distant cousin in Surrey who never came north. For sixty years the half-house slumbered. Then in 1838 the Earl of Surrey sold the estate to Henry Pelham-Clinton, 4th Duke of Newcastle of nearby Clumber, for £375,000. The duke wanted the land, not the building. The Howards were leading Catholic aristocrats and the new owner appears to have been animated by anti-Catholic sentiment; he stripped the lead from the roof, sold off the fittings, and demolished the main wing with gunpowder. Even with the salvage receipts he lost heavily on the deal.

Out of the rubble a reduced but still substantial mansion was assembled from the stable block, the service wing, and what remained of the east end. It was leased to Lord Foley, then to a lead manufacturer, then sold at auction in 1890 to a Nottingham businessman who promptly felled many of the mature trees for sale. Since at least that year the estate has been the home of Worksop Manor Stud, breeding thoroughbreds. The house that stands today is large - 25 bays by 14, in ashlar with hipped slate roofs - but it is a ghost of what Paine drew and the Talbots built. The Lord of the Manor of Worksop retains an ancient ceremonial role at every British coronation: the right to assist the sovereign by providing a glove for the right hand and supporting the arm that holds the sceptre. The house may be diminished, but its title still walks at the head of kings.