George, 1st Baron Lyttelton, was a poet, a man of letters, briefly Chancellor of the Exchequer, and secretary to Frederick, Prince of Wales. He began landscaping the Hagley grounds in the new Picturesque style during the 1740s, while his father was still alive, then between 1754 and 1760 commissioned Sanderson Miller, the gentleman-architect of the West Midlands, to build him a new Neo-Palladian house. With assistance from the London architect John Sanderson, Miller produced one of the finest provincial Palladian houses in England: a plain exterior, corner towers with pyramidal roofs in the manner Inigo Jones had pioneered at Wilton House, and Venetian windows for light. Inside, Francesco Vassali laid down some of the best Rococo plasterwork in the country, and the rooms filled with Chippendale furniture and family portraits by Van Dyck, Joshua Reynolds, Cornelius Johnson, and Peter Lely.

Hagley's 350 acres of deer park, grazed by fallow deer, were laid out between roughly 1739 and 1764, and the follies in it read like a roll-call of the era. The Hagley Obelisk, built in 1764 for Sir Richard Lyttelton, stands on Wychbury Hill and is visible for miles. The Temple of Theseus, a reconstruction of the actual building in Athens (now identified as the Temple of Hephaestus), was put up between 1759 and 1762 as a gift from Lyttelton's half-brother Admiral Smith. It is currently on Historic England's Heritage at Risk Register. A sham ruined castle in mediaeval style was designed by Sanderson Miller himself. Visitors came from everywhere: Alexander Pope, William Shenstone (both of whom got monuments in the park after their deaths), and James Thomson, who revised the Spring section of The Seasons after visiting in 1743. Horace Walpole, who rarely admired anything, wrote in 1753: I wore out my eyes with gazing, my feet with climbing, and my tongue and vocabulary with commending. In April 1786 John Adams and Thomas Jefferson visited together, and Adams confided to his diary that Stowe, Hagley, and Blenheim are superb.

Fifty years before the Palladian house went up, Hagley Park was already a place of fugitives. In the panicked days after the Gunpowder Plot collapsed in November 1605, two of the conspirators, Robert Wintour and Stephen Littleton, fled south from the disaster at Holbeche House and sought help from Humphrey Littleton, who had the use of Hagley Park at the time. The widow Meriel Littleton was the formal occupant, but Humphrey was running the place. The plotters were captured here on 9 January 1606 after the cook, John Fynwood, became suspicious of how much food was disappearing into the back of the house. When officials searched the building, another servant, David Bate, gave them away by pointing to the courtyard the two were trying to escape from. The Littletons were Catholics in a country that had just survived an attempt to blow up its Parliament, and the family paid for the association for generations.

In April 1943, four boys looking for birds' nests on Wychbury Hill, on the Hagley estate, climbed up to an old pollarded wych-elm and looked down into the hollow trunk. Inside was the skeleton of a woman. The autopsy estimated she had been dead about eighteen months, since 1941. She had been gagged with taffeta cut from her own clothes and stuffed head-first into the tree. Police never identified her. Then, starting in late 1944, anonymous graffiti began appearing on walls around the West Midlands, including the Wychbury Obelisk that the Lytteltons had built two centuries earlier. The message, in chalk and in different handwriting over the years, asked the same question: Who put Bella in the wych-elm? Theories have piled up over the decades, from a Nazi spy ring to a wartime affair gone wrong, but no one knows. The original tree died long ago, but every few years someone repaints the graffiti on the obelisk, and the question reappears.

The 9th Viscount and his wife rebuilt the house painstakingly after the 1925 fire, though the servants' top floor was never replaced and the roofline between the towers still sits lower than Sanderson Miller intended. By the 1970s the estate was in serious debt. The 11th Viscount Cobham was forced to sell off large tracts of land to keep Hagley solvent, complicated by an expensive divorce. His brother and successor, Christopher Charles Lyttelton, the 12th Viscount Cobham, has spent the last decade and a half restoring both the hall and the park. The Palladian Bridge has been rebuilt, the vista at the head of the valley reopened to the repaired Rotunda, and the Hagley Obelisk on Wychbury Hill conserved in 2010. The stable block, once full of horses, now hosts small businesses. The Great Western Railway named a 4-6-0 locomotive Hagley Hall in the 1920s; in June 2007 the engine, preserved on the Severn Valley Railway, was hauled here on a low-loader to be parked briefly on the forecourt of its namesake.