Governor La Gama began petitioning the Spanish emperor for defensive fortifications in 1529, warning that the island's vulnerability was causing colonists to emigrate. Construction started in 1533 using stone, authorized by Charles V, Holy Roman Emperor, as protection against Island Carib attacks and European imperial competitors. The structure was completed by 1540, but it had no guns and its position offered poor sightlines over the harbor. The Spanish built the San Felipe del Morro Fortress and later the San Cristobal Fortress to address the actual military need. La Fortaleza, meanwhile, quietly transitioned into a governor's palace. Initially, it was four stone walls enclosing a patio with a circular Homage Tower, from which governors swore fidelity oaths to the Spanish crown. A second tower, the Austral Tower, followed. In 1640, a chapel dedicated to Santa Catalina Alejandria was integrated into the walls, giving the palace its alternate name: Palacio de Santa Catalina.

La Fortaleza's military record is not distinguished by successful defense. In 1598, George Clifford, Earl of Cumberland, attacked and captured San Juan. In 1625, the Dutch general Boudewijn Hendricksz invaded the city and established himself in La Fortaleza. When the Dutch finally retreated, they set the fortress and much of the city ablaze. The third capture was permanent: in 1898, during the Spanish-American War, the U.S. Navy invaded and occupied Puerto Rico. According to tradition, the last Spanish governor, Ricardo De Ortega, struck a longcase clock in La Fortaleza with his sword at the moment Spain lost control of the island, stopping its hands forever. Whether or not the story is literally true, the image captures something real. After nearly four centuries of Spanish rule, time in La Fortaleza did stop, and when it resumed, the building served a new sovereign.

By 1846, the Spanish authorities had remodeled La Fortaleza from a medieval fortress into a Neoclassical palace, adapting its military architecture to a purely administrative function under Governor Rafael Aristegui. The complex today consists of several attached buildings with formal living quarters on the second floor and private quarters on the third. It overlooks the high city walls fronting San Juan Bay, and within its north perimeter lie sheltered gardens and a swimming pool. The exterior has alternated between white, light blue, and light gray paint over the years. In 1834, Colonel George Dawson Flinter described the building as having a chapel, stables, cistern, and spacious apartments. A century later, Herbert Hoover was received there as president. The building has been continuously inhabited and continuously renovated, each generation leaving its mark while the walls from the 1530s remain underneath.

On October 30, 1950, members of the Puerto Rican Nationalist Party attempted to storm La Fortaleza during the San Juan Nationalist revolt, intending to attack Governor Luis Munoz Marin. The five-minute shootout left four Nationalists dead: Domingo Hiraldo Resto, Carlos Hiraldo Resto, Manuel Torres Medina, and Raimundo Diaz Pacheco. Three building guards, including Lorenzo Ramos, were seriously injured. A decade later, on October 9, 1960, La Fortaleza was designated a National Historic Landmark. The designation itself became political. Ricardo Alegria, director of the Institute of Puerto Rican Culture, argued the plaque should read "national monument of Puerto Rico" rather than identifying it as a U.S. historic site. Restoration began in 1962 and continued through the Luis A. Ferre administration, with the Institute providing paintings and draftsmen. In 1983, UNESCO inscribed La Fortaleza alongside the San Juan National Historic Site as a World Heritage Site.

Today, Calle de la Fortaleza leads visitors through the cobblestoned heart of Old San Juan toward the palace. In recent years, the street has become famous for a canopy of colorful umbrellas suspended overhead, and on occasion, for displays of the Puerto Rican flag stretching the length of the block. The palace itself remains a working government building, the seat of the executive branch, situated about a mile from the Capitol building and two miles from the Supreme Court. It has housed more than 170 governors across Spanish colonial rule, U.S. territorial administration, and Puerto Rican self-governance. A fortress that failed at its original purpose, it found a more lasting one. Nearly five centuries after construction began, the building that blind men supposedly sited still stands on the western tip of San Juan Islet, overlooking the bay it was never quite able to defend.