The site first appears in the earliest known plans of Caracas, drawn up by Governor Juan de Pimentel around 1578, when it occupied a quarter of a city block as a modest house of bahareque and horcones. By 1689, the Caracas city council had acquired part of the lot from the heirs of Antonio de Tovar to build a new prison, which opened in 1696. The south side of the plot changed hands in 1704 through a land swap with Isabel Maria Xedler, and by December 1750, the new headquarters of the City Council stood complete. It was from the balconies of this House of the Cabildo, on April 19, 1810, that canon Jose Cortes de Madariaga signaled the people of Caracas to disavow Captain General Vicente Emparan - the act widely regarded as the first step toward Venezuelan independence.

The earthquake of March 26, 1812, left the building in ruins that persisted for decades. Reconstruction consumed four years, and in 1841 Congress approved the sale of both buildings to the national government. After renovation, the structure was inaugurated in 1842 under President Jose Antonio Paez. The transformation that gave the building its modern identity came in 1874, when President Antonio Guzman Blanco ordered architect Juan Hurtado Manrique to convert it into the Government Palace. Manrique added the 1810 memorial pavilion on the south side, and the palace was inaugurated on November 7, 1874, alongside the statue of Simon Bolivar in the plaza that now bears his name. Three years later, Congress designated it the official presidential mansion. General Francisco Linares Alcantara became its first resident - and someone painted it yellow, the color of the Liberal Party. The name stuck.

Castro's umbrella leap during the San Narciso earthquake of October 28, 1900, was more than an anecdote. His broken ankle convinced him that the presidential residence needed anti-seismic construction, and by decree on October 28, 1912, President Juan Vicente Gomez converted the Yellow House into the headquarters of the Ministry of Foreign Affairs - the role it has served ever since. The building's architecture reflects its layered history: a regular, square floor plan organized around a central courtyard, with Ionic columns supporting arcades on both floors. The Bolivar Room, occupying the entire front of the second floor, serves as the primary protocol hall. On February 16, 1979, the Yellow House was designated a National Historical Monument.

Not all of the building's wounds have healed gracefully. In 1989, fire consumed the second floor, destroying part of the Yellow House's artistic heritage. The loss was a reminder of how precarious the survival of historic buildings can be in a seismically active city with a turbulent political history. Yet the Casa Amarilla still stands on its original plot, facing the cathedral across the plaza, just as it did when it was a prison, a cabildo, and a flashpoint for revolution. From a colonial jail to the seat of Venezuelan diplomacy, few buildings anywhere in the Americas have held so many distinct purposes on the same piece of ground - or absorbed so many earthquakes in the process.