The Scarisbrick family had been on the site since 1238, which is the kind of continuity most English landed families can only point to in fragments. The original hall lies a hundred and forty metres northwest of the present building, marked now by a tree-covered island, a still waterlogged moat, and the slow line of Eas Brook tracing the boundary. The whole site has scheduled monument status. The current house dates to the era of King Stephen at its earliest fabric, but the version that visitors see now is overwhelmingly the work of the Pugins, father and son, working at different times for different patrons. The hundred-foot tower, visible across the West Lancashire flatlands for many miles, is the building's signature; you see it before you see the village.

Ann's brother Charles, who held the estate from 1835 to 1860, was the one who hired Augustus Pugin in the first place. Charles was a recluse with a magnificent budget who wanted his house remade in the Gothic style that Pugin was then making fashionable through his work on the Houses of Parliament. Pugin obliged with rich, dark, ecclesiastically inflected interiors that pleased his client. When Charles died and Ann inherited, she summoned Pugin's son Edward, gave him more latitude than his father had ever enjoyed, and asked him to go further. He replaced the existing clock tower with a taller and grander one in French Gothic style. He added a whole new East Wing in memory of Ann's father, joining it to the older building with an octagonal tower decorated with eight doves for the Scarisbrick family connection. He worked on her inkstand and her notepaper. The hall, when Edward Pugin was done, looked like a French chateau translated into Lancashire brick.

Ann lived in serious comfort. During her occupancy the house was gas-lit for the first time, and a central heating system seems to have been installed. She was a society hostess of the kind that fills shire houses with gala dinners and weekend parties, and she had inherited only the building and not its furnishings, so she set about decorating the whole place from scratch on a grander scale than her brother had managed. She lived at Scarisbrick from 1861 until her death in 1872 at the age of eighty-three. Her daughter Eliza inherited, then Eliza's French son Emmanuel de Biaudos, Marquis de Casteja, brought the estate into a Continental family. The marquis built the chapel of St Elizabeth on the site of the older Catholic chapel as a memorial to his wife, who had died in 1878.

The Castejas held the hall until 1923, when they sold it to Charles Scarisbrick's grandson Sir Tom Talbot Leyland Scarisbrick, returning it briefly to the family name. By 1946 the family had sold it on to become a training college. In 1964 Charles Oxley founded a school in the building, and Scarisbrick Hall School continues there today as a co-educational independent institution. The school has changed hands more than once: Nord Anglia, then GeMs, then Friends of Kingswood, the parents-and-teachers group renting the building now. When GeMs sold the school unexpectedly, a generation of GCSE students were left scrambling for places elsewhere. The hall itself is Grade I listed and on the Buildings at Risk Register, with restoration estimated at £2.46 million.

The hundred-foot tower is the thing you remember. Drive the country lanes of West Lancashire on a clear day and it appears across the fields, an unexpected silhouette of pinnacles and crocketed stone that belongs more to a French river valley than to the moss-covered plains of the Lancashire coast. Augustus Pugin and his son were trying to do something specific with these buildings, to convince Victorian England that Gothic was not a borrowed costume but the right native style for an Anglican-Catholic civilisation that had lost its way after the Reformation. The argument is on every elevation of Scarisbrick Hall. Whether it persuades depends on the viewer, but it is hard to deny the conviction. The film crews who shot Channel 4's Utopia in the Great Hall and Oak Room in late 2012 found a building that still photographs as a place of strangeness.