Charles Gregory Fairfax, the 9th Viscount Fairfax of Emley, bought a townhouse at 27 Castlegate in 1759. He was a widower. His first wife was the heiress Elizabeth Clifford, daughter of the 2nd Baron Clifford of Chudleigh, and her inheritance gave him the money for the purchase. The house was for his daughter Ann from that first marriage. In 1761 he hired the Yorkshire architect John Carr - the man who would later design Buxton's Crescent and Harewood House - to remodel the interior. The work took four years. When Carr was finished, the ceilings, friezes and overdoors had been worked by James Henderson and Giuseppe Cortese in plasterwork that survives largely intact today. The wrought-iron gates and railings facing Castlegate were lost when the street was widened, but the York artist Ridsdale Tait had recorded them in time.

Viscount Fairfax died in 1772 and the title became extinct. The house began its long, slow descent through Yorkshire's gentry. Sir Walter Vavasour took it in 1780. William Danby in 1787. Peregrine Wentworth in 1792. Sir John Lister Kaye in 1820. Mrs Ann Mary Pemberton owned it from 1840 to 1865. After that the records get vague. By the late nineteenth century the building had been renamed St George's Hall and was being used as a dance hall - the gilded saloon converted to a function room, the elegant drawing rooms partitioned. Worse came in 1921, when St George's Cinema was built immediately adjacent and Fairfax House was annexed and expanded into the cinema complex. The Carr interiors disappeared behind cinema fittings, paint, and several decades of indifferent maintenance.

The cinema closed in 1970. York City Council acquired the dilapidated building and sold it to the York Civic Trust. Between 1982 and 1984, the architect Francis Johnson led a restoration that pulled the eighteenth-century house back out from under the cinema. The defunct cinema lobby became the new main entrance to Fairfax House, which is why a building that started as a Georgian townhouse now has a slightly odd theatrical entrance one door north of where Viscount Fairfax's front door used to be. The York Conservation Trust later bought the freehold and leased it back to the Civic Trust, the arrangement that still holds the building in place.

What turned Fairfax House from a restored townhouse into a museum was a collection. Noel Terry - of Terry's of York, the chocolate firm that made the Chocolate Orange - died in 1980. He had spent decades assembling Georgian furniture and clocks, and his collection was donated to the York Civic Trust to fill the period rooms of the house being restored at Castlegate. The result is unique: ninety-nine percent of the furniture on display is English-made, making it one of the finest collections of English Georgian furniture in Europe. The collection has grown by acquisition and gift ever since. In summer 2017 a rediscovered wooden panel carved by Grinling Gibbons while learning his trade in York was added. In 2023, a portrait of Elizabeth Clifford - the first wife of the Viscount whose inheritance bought the house - was acquired and hung in the building she had unknowingly paid for.

Today Castlegate is a narrow lane between Clifford's Tower and the great curve of the River Ouse - a five-minute walk from York Minster, two minutes from the Jorvik Viking Centre, one minute from the York Castle Museum. Fairfax House is at number 27. The frontage is restrained: red brick, sash windows, a doorway under a fanlight. Inside, the Saloon, the Drawing Room, the Library, the Dining Room and the Viscount's Bedroom are all dressed as they would have been in the 1760s, with the Terry furniture in place. The Georgian glass collection is one of the largest in any house museum. After everything the building has been through - the dancing, the building society, the cinema reels - it is now what it was always supposed to be: a Georgian house in York, occupied by people who care about it.