The Multangular Tower is ten-sided, almost thirty feet high, and still standing where the Ninth Legion put it sometime in the early fourth century. Behind it, an Anglian Tower rises from an even earlier Roman wall, the masonry seams telling a story of repeated patching across centuries. The Ninth Legion arrived at this confluence of the Ouse and Foss in AD 71 and threw up turf ramparts on a foundation of green wood. Stone followed wood. Limestone followed stone. Each successive empire built atop the last, until by the early fourth century the tower stood three storeys tall and housed a catapult aimed across the river. The legion that built the original fortress later vanished from the historical record entirely, but their corner remains, exactly where they planted it, two millennia later.

St Mary's Abbey was founded in 1086 when Alan, Count of Brittany, granted the land and St Olave's Church to a monk named Stephen of Whitby. By the time Henry VIII dissolved it on 25 November 1539, the Benedictine community had grown into the richest monastery in northern England, with revenues over £2,085 a year. The Gothic abbey church the monks raised between 1270 and 1279 was aligned on a northeast axis, off the standard east-pointing orientation, because the shape of the riverbank demanded it. After dissolution, the abbey crumbled for two centuries while locals quarried its stones for other buildings. What survives is the lacework of the north and west walls, dummy lancet windows opening onto sky, foliage capitals weathered nearly featureless. The foundations are traced into the lawn, so visitors can walk the ghost of a vanished nave.

In 1828, the British royal family granted these ten acres to the Yorkshire Philosophical Society on the condition that they be turned into botanical gardens. Landscape architect Sir John Murray Naysmith laid out the grounds in the fashionable gardenesque style, with a conservatory, a pond, and a menagerie that included a golden eagle, several monkeys, and the bear who would shortly bid for territorial supremacy. The Yorkshire Museum, designed by William Wilkins in Greek Revival style, opened in February 1830 as one of Britain's first purpose-built museums. The young Princess Victoria visited in 1835, the same year the gardens first opened to the public. On 26 September 1831, the inaugural meeting of the British Association for the Advancement of Science convened inside the museum, putting the place at the center of Victorian Britain's enthusiasm for organized inquiry.

The collections housed at the Yorkshire Museum are vast. The biology section holds 200,000 specimens, including two stuffed great auks (extinct since 1844), a nearly complete moa skeleton, and the bones of elephants, cave bears, and hyenas pulled from Kirkdale Cave and dated to roughly 125,000 years ago. The archaeology collection runs to nearly a million objects spanning half a million years, with showpieces including the Coppergate Helmet, an eighth-century Anglo-Saxon piece discovered in York in 1982, and the Ormside Bowl, an Anglian silversmith's masterwork. The geological collection contains 112,500 specimens. In the gardens themselves, an octagonal observatory built in 1832-33 still operates with an 1850 Thomas Cooke telescope; it is Yorkshire's oldest working observatory, opened to the public on Thursdays and Saturdays by volunteers.

The gardens host York's Saluting Station, one of only twelve in the United Kingdom. A 21-gun salute fires at noon on royal occasions throughout the year, the brass of a military band ringing off the abbey ruins. On 9 September 2022, ninety-six rounds were fired in honor of Queen Elizabeth II, one for each year of her life. Beyond pageantry, the gardens have become a serious conservation site: in 2012 they were chosen as a release point for the endangered tansy beetle, an iridescent green jewel found in barely a handful of locations worldwide. In 2018 a new leaf-mining fly species was discovered in the gardens, mining the sage. Roxy Music and Hawkwind played here in 1970. Judi Dench performed the Virgin Mary in the York Mystery Plays among the abbey ruins in 1957. Layered history simply keeps accumulating.