Long before Fremont, the Spanish Presidio of San Francisco constructed the Bateria de Yerba Buena here in 1797, artillery batteries protecting the Yerba Buena anchorage. They called the promontory Punta Medanos, later Punta de San Jose. After 1849, Americans renamed it Black Point for the dark California bay laurel trees covering the bluff. Today, the shoreline of Black Point remains the last section of original San Francisco coastline east of the Golden Gate Bridge. The Spanish battery was abandoned by 1806, leaving only the land itself as witness to the centuries of flags that would follow: Spanish, Mexican, Bear Republic, American.

The Civil War prompted hasty construction. In 1864, a temporary battery rose at Point San Jose, as Upper Fort Mason was then known: a breast-high brick wall mounting six Rodman cannons and six 42-pounder guns. These were wartime expedients, not permanent fortifications, meant to protect San Francisco from Confederate raiders who never came. Excavation in the early 1980s uncovered the western half of this battery remarkably preserved, and it has since been restored to Civil War condition. In 1882, the fort received its permanent name, honoring Richard Barnes Mason, military governor of California during the Gold Rush.

President Grover Cleveland's Endicott Board in 1885 ranked San Francisco second only to New York in strategic importance, triggering massive harbor improvements. By 1904, Fort Mason became headquarters of the Pacific Division under Major General Arthur MacArthur Jr., father of Douglas. The lower piers rose between 1912 and 1915, warehouses and docks for the Army Transport Service. The Fort Mason Tunnel burrowed under the headland to connect with the State Belt Railroad along the Embarcadero. What had been a coastal defense post transformed into a logistical hub feeding American military expansion across the Pacific: Hawaii, the Philippines, islands whose names would become famous in the next war.

During World War II, Fort Mason commanded the San Francisco Port of Embarkation, controlling thirteen installations spread across the Bay Area: piers at Alameda, the Richmond Parr Terminals, Hamilton Field for air shipments, the Presidio's animal depot. Lieutenant Ronald Reagan served here early in the war, assigned to a planning unit called The Backroom Boys. The statistics stagger: 1,647,174 passengers and 23,589,472 measured tons shipped into the Pacific theater. This represented two-thirds of all troops sent to the Pacific and more than half of all Army cargo through West Coast ports. August 1945 saw the peak, with 93,986 outbound passengers loaded in a single month.

The fort closed as a military installation after more than a century of service. Some officer housing still belongs to the Army, some rented to civilians. A youth hostel occupies one of the larger buildings. The lower portion became the Fort Mason Center for Arts and Culture, a campus of creative institutions: the San Francisco Museum of Modern Art Artists Gallery, Blue Bear School of Music, Magic Theatre, Greens Restaurant, the Mexican Museum. A path follows the harbor edge, rising along the headland with views north past Alcatraz and west to the Golden Gate. The National Park Service headquarters for both the Golden Gate National Recreation Area and the San Francisco Maritime National Historical Park call it home now, guardians of a different kind of territory.