Sudbury is older than its painters. The Anglo-Saxon Chronicle calls it Sudberie, the south-borough, distinguishing it from Bury St Edmunds and Norwich to the north. By the early eleventh century a market was already trading here, on the same Market Hill that still slopes up to St Peter's Church. The Domesday Book records it in 1086 as a market town where local people came to barter their goods. A defensive ditch ringed the settlement, fed by a diverted arm of the Stour. Three medieval churches still stand from those centuries of cloth wealth, and in one of them, St Gregory's, sits a curious relic: the skull of Simon of Sudbury, born here around 1316, who rose to Archbishop of Canterbury and Lord Chancellor of England, then lost his head to a mob during the Peasants' Revolt of 1381.

The cloth trade made Sudbury prosperous in the Late Middle Ages, and the wealth paid for the churches and the timber-framed merchants' houses that still lean over the lanes. Weaving carried on through the seventeenth and eighteenth centuries with ups and downs. Today the tradition is silk: Vanners, Stephen Walters and other Sudbury mills still weave the silks used for royal coronation gowns, House of Lords curtains, and luxury menswear linings. Most ties sold by Britain's grandest tailors began on a Sudbury loom. The town remains a working textile centre in a country where most have closed.

Walk down Gainsborough Street and the painter's birthplace is still there, restored as Gainsborough's House and now a museum. The collection holds more of his work than anywhere outside London. He left at thirteen for an apprenticeship, but the Stour Valley never left him. His friend and successor John Constable, born downriver at East Bergholt, worked the same landscape a generation later. The pair invented English landscape painting between them. A statue of Gainsborough, palette in hand, stands on Market Hill outside St Peter's, unveiled in 1913.

During the Second World War, RAF Sudbury became home to the 834th Squadron of the 486th Bomb Group, flying B-24 Liberators against targets in Germany and occupied Europe. The crews were young Americans, far from home, and their commander brought in a professional artist named Phil Brinkman to paint the nose of each bomber with one of the twelve signs of the zodiac. They became the Zodiac Squadron, and the painted Liberators flew from Sudbury's flat fields on bombing and reconnaissance missions until the war ended. Most of the airfield is gone now. Sections of perimeter track and runway have become footpaths between Chilton, Newmans Green and Great Waldingfield, and pillboxes still squat along the river. Walkers cross concrete that once thundered under four-engined bombers heading east.

Sudbury was once a working port. Horse-drawn lighters carried grain, brick, coal and even coconuts for mat-making up the Stour from Manningtree. In 1914, fearing German invasion, the fourteen remaining Stour Lighters were scuttled in Ballingdon Cut. One was raised a century later and rebuilt with electric propulsion. The River Stour Trust runs boats from the Granary in Quay Lane, and each September the river hosts the Sudbury to the Sea event, when hundreds of canoes paddle the twenty-four miles down to Cattawade. Dodie Smith, who wrote The Hundred and One Dalmatians, sent Pongo and Missis through Sudbury at midnight in her novel, drinking at the fountain on Market Hill while the church clocks struck twelve. The town has always been a place travellers pass through, and a place worth stopping in.