Britten had grown up on the Suffolk coast and lived in nearby Snape in the 1930s. The sea, the marshes, the wind off the North Sea, the particular long horizontal light of this coast: all of it pulled at him. When he returned from the United States in 1942 he settled in Aldeburgh, and his great breakthrough opera Peter Grimes, premiered at Sadler's Wells in 1945, was set in a fictional version of this very town, based on George Crabbe's 1810 poem The Borough about a brutal Aldeburgh fisherman. Grimes made Britten famous worldwide and gave him the moral standing to do something only an established composer could do: build a festival in his own town, on his own terms, programmed around the music he wanted to hear and to write.

By the 1960s the small halls of Aldeburgh, Orford, Blythburgh and Framlingham could not accommodate the ambition. Then one of East Anglia's largest mid-19th-century malthouses became available at Snape, on the River Alde just outside Aldeburgh. Britten had once lived in Snape. He saw what could be done. The largest malthouse was converted into a concert hall, keeping the square roof vents that had drawn off the heat of germinating barley, and the Queen opened it on 2 June 1967 for the twentieth Aldeburgh Festival. The acoustic was extraordinary. The visual quality of the place, brick and timber and clear glass looking onto the marsh, became part of the festival's identity. On the first night of the 1969 Festival, the hall burned to the ground. Only the outer walls remained.

The 1969 Festival continued in other venues, losing only a single performance. Within a year the hall was rebuilt to its earlier specification, and the Queen returned in June 1970 to open the new building. The fire could have killed the festival. Instead it concentrated minds. Britten and Pears were joined as artistic directors first by Imogen Holst, daughter of the composer Gustav Holst, who served on the directorate from 1956 until her death in 1984, and then after Britten's own death in 1976 by a rotating cast of musicians: Philip Ledger, Steuart Bedford, Mstislav Rostropovich, Murray Perahia, Simon Rattle, Oliver Knussen. The Russian connection was strong. Rostropovich and Sviatoslav Richter became regulars. In 1970 the British premiere of Shostakovich's Fourteenth Symphony, dedicated to Britten, took place at Snape.

By 1982 the festival's archivist Rosamund Strode had counted new work from over seventy-five composers, including fifteen world-premiere operas. Britten wrote new work for it himself: A Midsummer Night's Dream came in 1960, and on 16 June 1973 Pears sang the role of Aschenbach at the first performance of Death in Venice. Harrison Birtwistle's Punch and Judy premiered there in 1968. Lennox Berkeley, Elliott Carter, Hans Werner Henze, Alfred Schnittke, Toru Takemitsu and Mark-Anthony Turnage came as composers in residence. Imogen Holst opened the doors to early music and to European composers little heard in England, including Berg, Webern, Boulez and Schoenberg. In contrast, John Dankworth, Cleo Laine and Peggy Ashcroft performed regularly, and Grace Kelly, by then Princess Grace of Monaco, took part in a poetry recital.

The festival's unique character comes partly from its rural Suffolk setting, with concerts in tiny medieval churches and the converted maltings overlooking the marshes, and partly from the persistence of Britten's original vision: eclectic, demanding, devoted to new music. Thomas Ades, the most prominent British composer of the next generation, served as artistic director before Pierre-Laurent Aimard took over from 2009 to 2016. The 2009 expansion of Snape Maltings added the Hoffmann Building with its Britten Studio and the Jerwood Kiln Studio, opened with the premiere of Harrison Birtwistle's opera The Corridor. The 2013 Festival, the sixty-sixth, marked the Britten centenary with a new production of Peter Grimes. The 2020 Festival was cancelled because of the pandemic and replaced with a virtual one. By 2024 the audience was back on the marsh, in the brick and the wind.