The name Aldeburgh is Old English for old fortification, ald-burh, but the fortification itself is gone, swallowed by the sea along with much of the Tudor town. In the 16th century Aldeburgh was a leading port and shipbuilding centre. The Sea Venture, flagship of the Virginia Company that wrecked memorably on Bermuda in 1609 and helped inspire Shakespeare's The Tempest, is believed to have been built here in 1608. The port silted up as the River Alde grew lazy, larger ships could no longer berth, and Aldeburgh fell back on fishing. By the 19th century it had reinvented itself as a seaside resort, and the whimsical Victorian and Edwardian architecture that defines so much of the town today dates from that long second act. The shingle beach still slopes down to where the lost streets used to be.

George Crabbe, the poet, was born in Aldeburgh in 1754 and grew up watching the men and women of the town work the sea. His long 1810 poem The Borough drew portraits of local types, including a fisherman called Peter Grimes who abused his apprentices, was shunned by the town, and in Crabbe's telling died in bed tormented by hallucinations of his victims. In 1942 Benjamin Britten, recently returned from America, read Crabbe's poem in California and decided to make it into an opera. Peter Grimes premiered at Sadler's Wells in 1945 and made Britten internationally famous. With Pears and Eric Crozier he founded the Aldeburgh Festival in 1948, and in 1957 he and Pears moved into The Red House in town. They are buried side by side in the churchyard of St Peter and St Paul. The opera made Aldeburgh part of the cultural world.

In 1908 Aldeburgh elected Elizabeth Garrett Anderson its mayor, making her the first female mayor in Britain. She had already been the first British woman to qualify as a physician and surgeon, the first to take a Doctor of Medicine in France, the first to be elected to a British school board, and a co-founder of the first British hospital staffed by women. She was the daughter of Newson Garrett, who had been mayor of Aldeburgh himself in 1889. The town later contributed two more notable women to the suffrage movement: Agnes Garrett, suffragist and designer, founded the Ladies Dwellings Company; and Dame Millicent Fawcett, Garrett Anderson's sister, who led the constitutional suffrage movement and set her one novel in the town. Elizabeth Garrett Anderson is buried in the town churchyard.

South of the town, where the shingle narrows toward Orford Ness, stands a unique quatrefoil Martello tower, the largest and northernmost of 103 such defensive towers built between 1808 and 1812 against a feared Napoleonic invasion. The Landmark Trust runs it as a holiday house. It is also the only surviving building of the lost fishing village of Slaughden, which the North Sea finished off by 1936. In the centre of Aldeburgh, the Moot Hall is a Grade I-listed timber-framed building from around 1520, used for council meetings for more than four hundred years and still housing the local museum. It once stood comfortably inland from the beach. Today it stands only a few yards from the waves at high water. The town has had to strengthen its sea defences after every major surge, including 1953.

Aldeburgh keeps an outsized cultural life for a town of around two thousand people. The Aldeburgh Bookshop, locally thought to occupy the site of George Crabbe's birthplace, has been in business more than seventy years and has run the Aldeburgh Literary Festival since 2002. There is a separate poetry festival, several food festivals, and an annual carnival held in August at least since 1892. M. R. James set his ghost story A Warning to the Curious in a town he called Seaburgh, clearly Aldeburgh, with the Martello Tower and the White Lion Hotel playing themselves. Ruth Rendell lived here; so did Imogen Holst and Joan Cross. Two family-run fish and chip shops have been rated among the country's best. The beach has a Blue Flag award. The shingle is still where Britten walked.