Construction began in 1579, possibly to a design by the Italian architect Sebastiano Serlio - though no one is sure. What's certain is that the Red Lodge was never meant to be a house. It was a pavilion: a place to take wine, to admire the gardens, to keep guests entertained between courses. The Young family used it as an extension of their hospitality, the Tudor equivalent of a luxurious garden room. Sir John died in 1589, and Dame Joan - sister and co-heiress of Nicholas Wadham, who would later co-found Wadham College, Oxford with his wife Dorothy - completed the building in 1590. Joan lies in Bristol Cathedral now, her recumbent effigy at the west entrance, a Tudor matriarch of formidable family connections.

The Great Oak Room is the building's masterpiece - original sixteenth-century oak panelling, a moulded plaster ceiling, and a double-decker fireplace that the building's own museum describes as making it one of the finest rooms in the West Country. In the 1730s, John and Mary Henley bought the lodge and doubled its footprint, fitting Georgian windows and a grand new staircase designed to flood the rooms with light as Mary processed toward her receptions. Later still, the lodge was leased to surgeons working at Bristol Royal Infirmary. James Cowles Prichard, who wrote Researches into the Physical History of Man, lived here. Two of his colleagues - Francis Cheyne Bowles and Richard Smith - used the Great Oak Room as a dissection theatre. The same oak panelling that had welcomed Queen Elizabeth's friends now watched anatomists at work.

In 1854, Lady Byron - using money from the endowment of her husband, the poet Lord Byron - bought the Red Lodge and gave it to Mary Carpenter. Carpenter was a Unitarian reformer who had already opened a ragged school in Bristol for the city's poorest children. Now she founded something new: the first girls' reformatory in Britain. The girls sent here were the children the Victorian state did not know what to do with - some had been in prison, some were destitute, some had simply been swept up by the poor laws. Carpenter believed they needed not punishment but nurture. She used the lodge as both school and headquarters, travelling the world to study the plight of pauper children, and lobbying Parliament. The reform school operated here until 1917. The 1845 Broadwood piano she bought for the girls still stands in what is now called the Mary Carpenter Room.

In the Small Oak Room hangs a portrait of Florence Smyth, of the Smyth family of Ashton Court. Beside her stands a young Black boy, unnamed. The painting has no identification of the child - no record of who he was, where he came from, or how he came to be in the household. He may have been a servant, a peer, or an enslaved person. If the last, the portrait is among the earliest depictions of a slave in the United Kingdom. The museum treats the painting honestly, neither hiding the question nor pretending to answer it - a boy whose name has been lost to the archive but whose face remains, looking out from a Bristol household at the height of the city's involvement in the Atlantic trade. The Skinner chair in the parlour was sat on by Prince Albert in 1843, when he came to Bristol to launch Brunel's SS Great Britain a short walk away.

By 1919, the historic interior was at risk of being torn out and sold piecemeal. James Fuller Eberle stepped in, buying the building for the Bristol Savages (now called Bristol 1904 Arts) and the Bristol Corporation. The arts society took the Victorian laundry as their studio; the corporation, which became Bristol City Council, took the rest. The knot garden, visible from the parlour and the Great Oak Room, is a 1980s interpretation of an Elizabethan original. The box hedge design comes from the moulded plaster ceiling in the bedroom upstairs - one Tudor pattern given back to the garden, four hundred years late.