The foundation that received the paintings was already two centuries old. Edward Alleyn (1566 to 1626) was one of the great actors of Elizabethan theatre, the leading man of the Rose and the Fortune Theatres, the company that rivalled Shakespeare's Globe. He grew wealthy enough on his commercial interests to buy the Manor of Dulwich in 1605, and he founded the College of God's Gift on the land: a school for boys, with almshouses for the local poor. The college would eventually splinter into three separate schools, Dulwich College, Alleyn's School, and James Allen's Girls' School. Alleyn bequeathed his own art collection along with the estate, mostly portraits of kings and queens. By the 18th century the gallery on the first floor of the Old College was depressing. Horace Walpole sniffed that he saw "a hundred mouldy portraits among apostles sibyls and kings of England."

Bourgeois died in 1811 and left his collection to the College of God's Gift, on the advice of the actor John Philip Kemble. His will required that Sir John Soane design a gallery to house them. Soane was the Regency architect best known for the Bank of England's interior; his eye was peculiar and exact. He designed a building of uninterrupted raw brick (rejecting the stucco porticos that other architects loved), a series of interlinked rooms lit by overhead skylights that bathed the paintings in indirect, even daylight. The American architect Philip Johnson would later say flatly: "Soane has taught us how to display paintings." Soane added a small mausoleum at the heart of the gallery for Bourgeois, Desenfans, and Desenfans's widow Margaret, who died in 1815 and is buried beside them. It opened to Royal Academy students in 1815, then to the public in 1817. It is, by date, the oldest public art gallery in England.

In the early hours of 31 December 1966, thieves broke into the gallery and walked out with eight paintings: three Rembrandts, three Rubens, a Gerrit Dou, and an Adam Elsheimer. They were worth at least three million pounds. The reward offered was a thousand. Detective Superintendent Charles Hewett (who had previously investigated the suspected serial killer John Bodkin Adams) led the investigation, and within days all eight paintings were recovered. The only thief caught was an unemployed ambulance driver named Michael Hall, who got five years. One small painting in that haul, Rembrandt's Portrait of Jacob de Gheyn III from 1632, has since been stolen and recovered four times in total, making it the most frequently stolen artwork in the world according to Guinness World Records. Last stolen in 1983, it was found in a left-luggage office in Germany in 1986, returned anonymously, discovered on the back of a bicycle, and finally found under a bench in a graveyard in Streatham. The gallery upgraded its security in 2010.

What the thieves were after, the visitors still come for. The collection runs deep in French, Italian, and Spanish Baroque, with strong British portraiture from the Tudor era forward. There are eleven Aelbert Cuyps, twelve Philip Wouwermans, ten Rubens, nineteen David Teniers the Younger. The English School includes seven Gainsboroughs, nine Reynolds, two Hogarths, a single John Constable. The Italian rooms hold two Raphaels, two Canalettos, three Tiepolos. Among the most beloved is Bartolome Esteban Murillo's painting of two flower girls, which earlier directors used to call "the Mona Lisa of Dulwich." Generations of artists trained here as students. John Constable came regularly. So did J.M.W. Turner, William Etty, and later Vincent van Gogh. Charles Dickens, who knew the gallery well, sent his hero Samuel Pickwick here to retire at the end of The Pickwick Papers.

On 12 July 1944, during the last desperate phase of the German V1 flying-bomb campaign on London, a buzz bomb hit the gallery and severely damaged the mausoleum and west wing. Bones from the three sarcophagi were scattered across the lawn. When Austin Vernon and Partners refurbished the building (the Queen Mother reopened it on 27 April 1953), they reassembled what they could; each sarcophagus now contains approximately one skeleton, though no one knows for certain whose. The British architect Rick Mather added a modernist cafe and education wing in 1999, with Queen Elizabeth II opening the latest refurbishment on 25 May 2000. The gallery marked its bicentenary in 2017 with a temporary summer pavilion, won the first major UK exhibition of Berthe Morisot since 1950 in spring 2023, and in 2023 announced a £4.9 million redevelopment by Carmody Groarke including a new sculpture garden. The Soane skylights remain the heart of the place.