The Worsleys bought the manor of Hovingham in 1563. The Palladian house you see now was built between 1750 and 1774 by Thomas Worsley VI, who designed it himself - and he was no amateur. He held the office of Surveyor-General to the Board of Works under George III, which made him responsible for the king's official buildings. Building Hovingham was therefore both a personal project and a professional manifesto. He wrapped the new house around the existing stable block instead of demolishing it, an unusual sequence that left the riding school as the structural and ceremonial heart of the building. The hall is built of limestone ashlar with Westmoreland slate roofs, laid out on an L-shaped plan. It is Grade I listed; a Tuscan temple in the grounds, an ornamental bridge over a waterfall, and the pigeoncote and walls are listed Grade II.

Thomas Worsley was George III's friend before he was his courtier. He had taught the future king to ride in his youth, and one of his accommodations as designer of Hovingham was to provide an indoor school suitable for royal use should the king visit. George III never actually came, but the room remained the architectural keystone - and the Worsley family's portrait collection eventually included items connected with the royal household. Thomas also sat in the House of Commons, as Whig MP for Orford from 1761 to 1768 and for Callington from 1768 to 1774. The estate passed to his son Edward, then to his nephew William, who became the first Baronet Worsley of Hovingham in 1838. The baronetcy is still in the family. The current owner, Sir William Worsley, 6th Baronet, took over from his father Sir Marcus, who died in 2012.

Stretched out in front of the hall is a cricket pitch said to be possibly the oldest private cricket ground in England. Colonel Sir William Worsley, the 4th Baronet, captained Yorkshire County Cricket Club in 1928 and 1929. He went on to serve as Lord Lieutenant of the North Riding from 1951 to 1965. His daughter, born in 1933, grew up running on this lawn. Her name was Katharine Lucy Mary Worsley, and in 1961 she married Prince Edward, Duke of Kent, in York Minster - becoming Katharine, Duchess of Kent, a working royal for most of the second half of the 20th century. The cricket pitch is still in use. The riding school is still here. The 5th Baronet, Sir Marcus, was MP for Keighley and then Chelsea before serving as Lord Lieutenant of North Yorkshire. The current Baronet, Sir William, and his wife Marie-Noelle open the house and gardens for about four weeks each June - if you want to see inside, that is the window.

The Hovingham Festival began in 1887 when the family started inviting professional musicians to perform in the great indoor space they had inherited. Thirteen festivals were held in the original run, ending in 1906. The festival was revived in the 1950s after a 45-year break, and continues in various forms. There is something perfect about a room originally designed for horses being used for string quartets - the acoustics, it turns out, are remarkable. The grounds spread out around the house: walled garden, lawns, the Tuscan temple sitting at the end of a walk, the ornamental bridge over its small waterfall. The village of Hovingham clusters around the gates, a few hundred people in a place that has been quietly important to its inhabitants for centuries, and which most travellers between York and Scarborough drive past without ever knowing the king's riding master built a Palladian palace here.