Lavenham appears in the Domesday Book of 1086 as the manor of Aubrey de Vere I, ancestor of the Earls of Oxford, who had already planted a vineyard there. The Vere family held the estate for over five hundred years, until 1604. The wool trade arrived in the fifteenth century and turned the place into a powerhouse. By the late 1400s Lavenham was among the twenty wealthiest settlements in England, and the Spring family, the most successful of the merchant houses, was building on a scale that matched anything in London. In the official inspector's returns for 1446 the Lavenham weavers produced 264 finished cloths in a single year. The wool church of St Peter and St Paul went up in 1525 with a tower 138 feet high - claimed as the tallest village church tower in Britain. The Guildhall of Corpus Christi was completed in 1529. Then the Dutch refugees settled in Colchester began producing cloth that was cheaper, lighter and more fashionable. Lavenham's industry collapsed inside a generation, and the town never recovered.

Many of Lavenham's buildings appear visibly crooked from the outside, leaning at odd angles, with windows and doorways slightly skewed. The traditional explanation, recounted with relish in every visitor's guide, is that the town grew so fast in its boom years that the houses were thrown up in haste using green, unseasoned timber. As the wood slowly dried in the decades that followed, the timbers warped and twisted, pulling the buildings out of true. By the time the warping became obvious, the cloth industry had collapsed and the families had no money to rebuild straight. So the houses stayed crooked. The Crooked House at 7 High Street is the most famous of them, thought to be the inspiration for the nursery rhyme There Was a Crooked Man and described by Country Life in 2022 as one of the world's most photographed homes. It has been a Grade II listed art gallery, a tearoom, and is now run by Alex and Oli Khalil-Martin as a venue for experiences and events.

In the late eighteenth century, the village was home to the poet Jane Taylor (1783-1824), and it may have been while living in Shilling Street that she wrote 'The Star', the poem from which Twinkle Twinkle Little Star takes its lyrics. Colchester and Ongar, both in Essex, also claim the honour, but it would be very Lavenham-like for the line about the diamond in the sky to have been written in one of the warped, jettied cottages of a town that had lost its wealth and gained a different kind of fame. The Taylors were a literary family. Jane's father Isaac was a celebrated engraver and writer of children's books; he also lived in Lavenham.

The crooked, time-frozen quality of the streets has made Lavenham an irresistible film set. The witch-burning scenes in the 1968 Vincent Price film Witchfinder General were staged in front of the Guildhall. In 1969 some of The Thirteen Chairs was filmed here, Sharon Tate's last role before her murder. In 1971 Pier Paolo Pasolini used the village as medieval London for The Canterbury Tales. Stanley Kubrick's Barry Lyndon (1975) included the Guildhall. In December 1969, John Lennon and Yoko Ono floated a hot-air balloon in the Market Place for their experimental film Apotheosis. And in 2010, under strict secrecy, the production of Harry Potter and the Deathly Hallows Part 1 came to the village's De Vere House to film exteriors for Godric's Hollow, although the cast did not actually visit. The 1990s BBC drama Lovejoy was set in the area; one 1994 episode was titled, inevitably, 'Last Tango in Lavenham'.

Lavenham sits about five miles northeast of Sudbury, on a ridge above the River Brett, with the church on the southernmost summit, the marketplace on the central one, and a windmill ruin on the northern. The southernmost valley contains a brook that has been covered by a culvert for 500 years; Water Street is built on top. There have been attempts to give the culvert Scheduled Monument status as an early example of municipal plumbing. A few miles north, on land near Alpheton, sits the former airfield of RAF Lavenham, where the American 487th Bomb Group flew B-24 Liberators and B-17 Flying Fortresses from 1944 to 1945. The off-duty crews came into the village to drink at the Swan Hotel, which still stands on the high street and still trades on the connection.