The structure at Strangers' Hall began modestly. A merchant named Ralph de Middleton purchased the site in 1286–87, and it was probably he or his descendants who raised the first building — a domestic house above a three-bayed undercroft, dating to around 1320, whose chamfered ribs and wall piers survive to this day.

Each century added its own layer. In the 15th century, William Barley and then Mayor Thomas Caus substantially remodelled the building: a hall was raised above the undercroft, 10.3 metres long by 6 metres wide, with walls of flint-rubble. The south-west range went up, connected by a newel staircase. A north range facing the street followed, with loading bays at ground level for the commercial traffic that moved through the courtyard — goods in, goods out, watched over from a porter's lodge equipped with a squint: a narrow opening in the wall that allowed the porter to see who was coming without being seen himself.

The family whose name runs through the building's history is the Sothertons. Nicholas Sotherton, grocer, alderman, and eventually mayor from 1539, bought the hall in the early 16th century and transformed it. Under his hand, the south wall was heightened to carry a new roof decorated with his merchant's mark and the cross of Saint George. A bay window — likely salvaged from a monastic house dissolved in the Reformation — was set into the wall, its spandrels carved with shields.

Nicholas died in 1540; his wife Agnes continued the work. Their son John occupied the house, and John's brother Thomas — also a mayor, in 1565 — was the one who brought the Protestant refugees to the city. Strangers' Hall thereby became something more than a merchant's home. It became a place of refuge during one of the great religious persecutions of the 16th century, a fact that its name has carried ever since.

After the Sotherton line faded and ownership passed through various hands, the next 150 years of the hall's history have been described, candidly, as 'a considerable mystery.' What is known: an Italian sculptor had a studio there in 1819, and a Norman priest who taught languages to the writer George Borrow rented rooms. In 1846 a meeting was held to consider demolishing the building. The motion failed.

Fifty years later the hall's fate was still precarious. In 1899, a solicitor named Leonard Bolingbroke — grandson of the Norwich painter James Stark — bought Strangers' Hall for £1,050 specifically to save it from destruction. He converted it into a folk museum, filled it with domestic objects, and eventually gave the entire property to the city of Norwich. It opened as a public museum in July 1923, the ceremony attended by H. Rider Haggard, among others. The city has run it as a museum of domestic history ever since.

The building requires constant vigilance. Woodworm and damp are its recurring enemies, and the annual deep-cleaning is thorough enough that in January 2023, volunteers discovered charred parish records from a church bombed in 1942 — possibly misfiled at the hall after a library fire in 1994 — tucked away in an overlooked corner. A few months later, extensive woodworm required moving several antique pieces of furniture to another site for freezing treatment that kills the larvae.

The undercroft from 1320 still stands beneath the hall. The bay window Nicholas Sotherton may have salvaged from a dissolved monastery is still in the south wall. The oak panelling Sir Joseph Paine installed in the 1660s to create the Oak Room — with its overmantel painting of Norwich as seen from Mousehold Heath — remains. Seven centuries of merchants, mayors, refugees, sculptors, language teachers, and volunteers have moved through these rooms. The building holds them all, and keeps accumulating.