Philip IV of Spain enjoyed retreating to rooms annexed to the monastery of San Jeronimo el Real, near what is now the Prado Museum. The land around it belonged to his chief minister, the Count-Duke of Olivares, who saw an opportunity. In 1629, Olivares began planning what would become the Buen Retiro, a series of pavilions and offices that expanded, building by building, until 1640. Nothing about the project was planned from the start. Each addition was a new improvisation, a new gesture of flattery from minister to monarch. The result was enormous but poorly built, its materials chosen for speed rather than durability. That haste would eventually doom it.

Despite the palace's shoddy construction, Olivares and Philip mounted an extraordinary campaign to fill it with art. Because quality antique paintings were scarce on the market, ambassadors were dispatched to commission new works from Rome and Naples. Claude Lorrain, Nicolas Poussin, and Gaspard Dughet contributed landscapes. Massimo Stanzione painted biblical scenes. Giovanni Lanfranco depicted ancient Rome. For the Salon de Reinos, the great reception room, a series glorifying Spanish military victories was commissioned from the country's finest painters. Diego Velazquez produced The Surrender of Breda for this room, one of the most celebrated paintings in Western art. Francisco Zurbaran, Antonio de Pereda, and Vicente Carducho contributed companion pieces. Many of these works survive today in the Prado Museum.

The palace served as a royal residence for over a century. After fire destroyed the Royal Alcazar of Madrid in 1734, the Buen Retiro became the court's primary seat until the new Royal Palace was completed in 1764. But the building's hastily laid foundations and cheap materials were already taking their toll. When Napoleon's troops arrived in 1808 during the Peninsular War, they converted the palace and its grounds into barracks. Powder kegs were stored in the gardens. A bunker was dug. The damage was so extensive that when Queen Isabella II later attempted restoration, her engineers concluded there was nothing to save. The palace was demolished almost entirely.

Two buildings from the original complex still stand, both heavily altered. The Salon de Reinos, where Velazquez's masterpiece once hung, served for decades as the army museum before that collection moved to the Alcazar de Toledo in 2010-2012. It is now being integrated into the Prado Museum complex. The Cason del Buen Retiro, originally the ballroom, houses the Prado's Study Center. Its ceiling preserves a magnificent fresco by the Italian artist Luca Giordano, painted around 1696-1697 for King Charles II, depicting The Apotheosis of the Spanish Monarchy. And then there are the gardens, or what became of them: today's Retiro Park covers roughly half the original grounds, its layout bearing no resemblance to the 17th-century design but preserving the essential idea of a place where a city comes to breathe.