The ground on which Holker stands belonged first to Cartmel Priory, founded in 1190 about two kilometres up the road. When Henry VIII dissolved the monasteries in the 16th century, the land was bought by the Preston family - local landowners who built the original house in the early 1500s. The Preston era ended messily. In 1644 the estate was confiscated from Thomas Preston by Parliament during the Civil War, then later returned. On Thomas Preston's death the property passed to the Lowther family through the marriage of his heiress Catherine to Sir William Lowther, 1st Baronet of Marske. In 1756 another marriage moved Holker again, this time to Lord George Augustus Cavendish. It has remained with the Cavendishes ever since.

George Webster of Kendal rebuilt much of the hall in the 19th century in a Jacobean Revival style, working in roughcast stone with ashlar dressings and a slate roof. Edward Graham Paley then began further renovations. Only about a decade after Paley's work, in 1871, a major fire tore through the building and destroyed the entire west wing along with a number of notable artworks. Paley had by then taken his pupil Hubert Austin into partnership, and the firm of Paley and Austin rebuilt the lost wing in Elizabethan Revival style, in variegated red sandstone. The new entrance front bristled with turrets, domed pinnacles, a copper-covered ogee cupola, and a square tower with a lead-covered pyramidal roof. Holker is described as Paley and Austin's 'most important country house commission.'

Inside, the hall reads like a roll call of the European art world. Four panels in the billiards room are attributed to Jean-Baptiste Oudry, alongside a caricature by Joshua Reynolds and works by Jan Wyck and Matthias Reed. The silk-walled drawing room contains a Carrara marble fireplace and paintings by Claude Joseph Vernet, Salvatore Rosa, and Douglas Anderson; Vernet's companion piece was lost in the 1871 fire. The dining room has Chippendale chairs and a self-portrait by Anthony van Dyck. A cantilevered oak staircase rises through limestone arches; its hundred-plus balusters are each carved with a different design. Queen Mary slept here in 1937; the Duke of Gloucester and his wife in 1939; both rooms now bear their names. A Wedgwood bedroom carries a marble fireplace inlaid with blue and white Jasperware, with a four-poster bed by George Hepplewhite.

The grounds run to ninety hectares - ten of formal gardens, eighty of parkland, deer park, and woodland. The pleasure gardens include a cedar planted by Lord George Cavendish in the late 18th century and an araucaria from 1844. Thomas Hayton Mawson, the great Arts and Crafts garden designer, worked here in 1901 and redesigned the formal garden in 1910. North of the hall stands a lead statue of Inigo Jones, made by John Michael Rysbrack in the 1740s and moved here from Chiswick House in the 19th century. There is even a geological footnote: in chronostratigraphy, a sub-stage of the Carboniferous period is named the 'Holkerian' after this very estate.

The hall remains the home of Lord Cavendish and his wife. The older wing - the part that survived the 1871 fire - is private. Paley and Austin's west wing, with its turrets and tracery, is the part the public sees. The former stable buildings, U-shaped in plan with a timber bell turret and a weathervane dated 1864, now contain a café and gift shop. Events run through the year. Visitors wander between the Elliptical Garden, the Summer Garden, and the Sunken Garden with its pair of summer houses, past the two-tier circular ice house (in use since at least 1732) and eventually back to that vast lime tree. Standing under it, the house feels less like the centre of the estate than another lovely thing the lime has been quietly outliving.