Sudbury was in the Domesday Book by 1086, recorded among the manors of the new Norman order. The Vernon family did not reach the village until the 16th century, when the Sudbury heiress Ellen Montgomery married Sir John Vernon, son of Sir Henry Vernon of Haddon Hall. They brought with them centuries of Derbyshire pedigree but no special distinction in architecture. That changed with George Vernon, grandfather of the future first Baron Vernon, who set out after the Restoration of Charles II to build something serious. The result, completed between 1660 and 1680, is one of the finest Restoration mansions in England, though its design is curiously old-fashioned: the layout follows Jacobean conventions, with state rooms on one side and servants' quarters opposite, while the entrance porch is firmly Baroque, with paired columns rising over carvings by William Wilson.

The collection of portraits Sudbury accumulated reads like a guest list for Restoration London. Two of King Charles II's mistresses look down from the walls: Nell Gwyn, the orange-seller turned actress, painted around 1675 by Sir Peter Lely, and Louise de Keroualle, Duchess of Portsmouth, the French Catholic from whom Charles II's modern Dukes of Richmond descend, attributed to Sir Godfrey Kneller and dated 1670. Catherine Vernon, painted by John Riley in 1681, watches over them in a more domestic capacity. King George III and Queen Charlotte appear together from the studio of Sir Joshua Reynolds in 1779. By the late 19th century the Sudbury estate ran from Cubley down to Marchington in Staffordshire, generations of Vernons accumulating land and pictures and the kind of quiet wealth that built rooms like the Long Gallery to put it all in.

In 2022 the National Trust did something unusual with Sudbury. They took down the ropes. They added speech bubbles to the portraits. They installed activities and games and reframed the entire house as an experience designed primarily for children. The redesigned property became known as the Children's Country House. Reaction divided sharply. Joanna FitzAlan Howard, daughter of the 10th Baron Vernon, told The Times that her ancestral home had been turned into "a child-centred theme park" and accused the Trust of dumbing down. The pressure group Restore Trust complained that adult visitors without children were being discouraged, and that house contents had been moved aside for games. In May 2023 the redesigned house won Permanent Exhibition of the Year at the Museum and Heritage Awards. The judges praised its "participatory and imaginative" approach. Both sides held their ground.

There is no separating Sudbury from Pride and Prejudice now. The 1995 adaptation rewrote how a generation pictured Jane Austen's Pemberley, and the rooms doing that work were these rooms, this staircase, these windows. Older viewers may remember the centrally-placed domed cap-house from a different source entirely: it appeared in the title shot of Yorkshire Television's children's programme The Book Tower. Adjacent to the main house, in the 19th-century servants' wing, the National Trust Museum of Childhood holds toys and games and the small ephemera that Victorian children left behind. The wing became a museum decades before the main house was reimagined for the same audience. Looked at one way, the redesign simply admits what Sudbury has always been good at: hosting the imaginations of children. Looked at another, it removes something irreplaceable. Both views are sincerely held by people who care about the place.