Spain recognized the strategic value of this promontory in 1794, when it built the Castillo de San Joaquin on a high white cliff called Punta del Cantil Blanco. The adobe structure with nine to thirteen cannons guarded the narrowest point of the bay's entrance, serving as an outpost of the Presidio de San Francisco. When Mexico gained independence in 1821, the fort passed to new hands, but by 1835 the Mexican army had moved to Sonoma, leaving the castillo's walls to crumble in wind and rain. On July 1, 1846, Captain John Charles Fremont, Kit Carson, and ten followers captured the empty fort and spiked its cannons. The Spanish name "Punta del Castillo" eventually became, in rough American translation, Fort Point.

After California became a state in 1850, military officials recognized San Francisco Bay as commercially and strategically vital. The U.S. Army Corps of Engineers began work on a new fort in 1853, designing it so the lowest tier of artillery sat close to water level, allowing cannonballs to ricochet across the surface and strike enemy ships at the waterline. Workers blasted away the original cliff and built a structure with seven-foot-thick walls and multi-tiered casemated construction typical of Third System forts. Of more than thirty such forts on the East Coast, Fort Point became the only one on the West. In 1854, Inspector General Joseph K. Mansfield declared it "the key to the whole Pacific Coast." A crew of 200, many unemployed miners, labored eight years to complete it.

In 1861, with civil war looming, the army mounted the fort's first cannon. Colonel Albert Sidney Johnston, commander of the Department of the Pacific, prepared Bay Area defenses and ordered troops to Fort Point. The Kentucky-born Johnston then resigned his commission to join the Confederate Army and was killed at the Battle of Shiloh in 1862. Throughout the war, artillerymen at Fort Point stood guard for an enemy that never came. The Confederate raider CSS Shenandoah planned to attack San Francisco, but on the way to the harbor, the captain learned the war was over. It was August 1865, months after General Lee surrendered.

The Civil War rendered Fort Point's design obsolete before it saw action. Rifled artillery had devastated Fort Sumter and Fort Pulaski on the Atlantic Coast, proving that masonry walls could not withstand modern weapons. Troops moved out, and the fort was never again continuously occupied. In 1892, the army began constructing new Endicott System fortifications with steel breech-loading guns. Within eight years, all 103 smooth-bore cannons at Fort Point had been dismounted and sold for scrap. The 1906 earthquake caused moderate damage, and the fort served variously as barracks, training facility, and refugee camp. During World War II, soldiers from the 6th U.S. Coast Artillery guarded minefields and the anti-submarine net spanning the Golden Gate.

The American Institute of Architects proposed preserving Fort Point as early as 1926 for its outstanding military architecture, but funds were unavailable. When Golden Gate Bridge plans in the 1930s called for the fort's removal, Strauss intervened. "While the old fort has no military value now," he said, "it should be preserved and restored as a national monument." On October 16, 1970, President Richard Nixon signed a bill creating Fort Point National Historic Site, now administered by the National Park Service as part of the Golden Gate National Recreation Area. Today visitors walk through brick archways and climb iron staircases where soldiers once kept watch, all while the bridge rumbles overhead.