The property began as a hospital of St Mary, founded during the reign of Henry I, sometime between 1100 and 1135. Richard de Clare endowed it as an Augustinian priory in 1212. The priory was closed in 1536 during the Dissolution of the Monasteries, its stones carted off for use at Madingley Hall three years later. The Fowkes family acquired what remained in 1595 and converted the ruins into a Jacobean-style house, incorporating the walls of the old chapter house into the main domestic structure. The monks' day room, or calefactory, became an outbuilding. Subsequent owners included Thomas Hobson — the Cambridge carrier whose refusal to let customers choose their own horses gave the English language the phrase "Hobson's Choice" — and his descendants. By the time Thomas Parker died in 1643 the property had ceased to be called Priory and was known instead as Anglesey Abbey.

Urban Huttleston Broughton, 1st Baron Fairhaven, moved into Anglesey Abbey in 1926 with a fortune and an eye for quality. Over the following decades he added a library wing, a Tapestry Hall, and extended the house to accommodate a collection that grew to encompass paintings by Claude Lorrain and Dutch flower masters like Ambrosius Bosschaert, marble horses, a fifteenth-century wooden figure of St Jerome, a silver Shield of Achilles by John Flaxman, and bronzes by some of the leading sculptors of his era. The library wing, designed by Sidney Parvin, was added in 1937. Three generations of the Ayres family worked in the gardens from 1921 onward. Lord Fairhaven also purchased and restored Lode Mill in 1934; by 1978 it had been returned to working order by the Cambridgeshire Wind and Watermill Society, and today you can buy flour in the mill shop.

The 98 acres of gardens at Anglesey Abbey are Lord Fairhaven's greatest work. He inherited flat Cambridgeshire farmland and created formal avenues, a Winter Walk, a Temple Lawn (laid in 1953 to mark the coronation of Queen Elizabeth II), a rose garden, and a Coronation Avenue lined with caryatid sculptures. A Winter Garden was opened in 1998 in his memory, designed so that the grounds offer something striking in every season: snowdrops and daffodils in early spring, roses and borders in summer, the Winter Lights festival in the cold months. The gardens are bounded on the northwest by Bottisham Lode, and the quarry pool nearby is believed to mark the site of a nineteenth-century coprolite mine. This is a landscape that has been worked, imagined, and reimagined across centuries.

Anglesey Abbey appears in the National Trust's 2020 report on its properties' connections to colonialism and historic slavery. Samuel Shepheard, who bought the property in 1739 but never lived there or made improvements to it, had been one of the twenty-four directors of the East India Company from 1717 to 1721. The report does not suggest that the house or gardens were built on proceeds of that connection — Shepheard simply owned the property as an investment — but the Trust's transparency about these histories is part of its broader effort to tell complete stories about the properties in its care. Anglesey Abbey, in any case, has far more centuries of history than any single owner's biography, and the house Lord Fairhaven left behind remains one of the most vividly personal country houses in England.