The building was not designed for art. Thomas Baldwin drew up the first plans in 1794 - a two-storey building to serve the new Sydney Gardens, the only surviving eighteenth-century pleasure gardens in the country. Baldwin went bankrupt before construction began. The job passed to Charles Harcourt Masters, who built a three-storey hotel between 1796 and 1799. Visitors entered the gardens through the hotel itself. A semicircular orchestra projected from the rear at first-floor level, with two rows of supper boxes curving from the sides. Public breakfasts were served around midday: tea, coffee, rolls, and Bath's own Sally Lunn buns, followed by dancing. Three evening galas a summer marked the birthdays of George III and the Prince of Wales and the July races; on those nights thousands of lamps lit the gardens, and supper was eaten to music and fireworks. The hotel became a lodging house in 1836, and an extra storey of bedrooms was added. By the time the Holburne collection arrived, the building's original purpose had been forgotten.

Holburne's collecting taste was unfashionable in its breadth. He bought what he liked rather than what was in vogue - Italian maiolica when the Victorians were obsessing over Pre-Raphaelites, Old Masters when others chased modern landscapes, miniatures when miniatures had fallen out of polite interest. The result was a collection of unusual coherence: connoisseur's pieces across silver, porcelain, paintings, portrait miniatures, gems, books, furniture and Roman glass. Later acquisitions doubled down on what Holburne would have wanted. In 1955 Ernest E. Cook, grandson of the travel entrepreneur Thomas Cook, bequeathed ten paintings including works by Gainsborough, Stubbs and Turner. Sir Orme Sargent gave his family's Allan Ramsay portraits in 1962. The galleries today still show what Holburne himself would have recognised - and what he could not have imagined: Northern Renaissance portraits on long-term loan from the Schroder family, Hans Burgkmair's Jakob Fugger and Sybilla Artzt, Dürer's engravings of Christ's Passion.

By the late 2000s the museum needed more space, and the proposal was contentious. The architect Eric Parry RA designed a three-storey extension that pointedly refused to imitate Georgian Bath. Instead of Bath stone and pilasters, Parry used vertical mullions of brown-green mottled ceramic faience, intended to echo the rhythms and shifting foliage of the mature trees in Sydney Gardens behind. Local residents, councillors and conservationists fought it for more than a year before planning was approved in 2008. As built, the extension reads from below as transparent, from the middle as layered and semi-transparent, and at the top as solid - the building rising through states of opacity. The new ground-floor cafe gives a 180-degree view onto the gardens, and the day-lit top floor houses temporary exhibitions. The basement holds storage and education space. The museum reopened in May 2011, with three new floors that doubled its public space.

The Holburne has had a second career as a film set. It stood in for the Devonshire villa in the 2008 film The Duchess, with Keira Knightley sweeping through what was once a Georgian breakfast room. It played Steyne's mansion in Vanity Fair, the 2004 adaptation starring Reese Witherspoon. It has appeared in a German TV film, Four Seasons, with Tom Conti and Michael York, and in the Bollywood film Cheeni Kum in 2006. More recently it has been one of the visual anchors of Bath as Bridgerton's Mayfair - the building exterior is one of the most-photographed shots in the series. For a museum that began as a Georgian pleasure-garden hotel, becoming a backdrop for the fictional Georgian society Netflix invented feels rather like an old building returning to first principles.