The station was built by Captain Miles A. Johnson and his cousins Henry and James A. Johnson, together with twin brothers Alpheus and William Packard. The Johnsons were Portuguese-American, part of a larger community of Azorean and Cape Verdean whalers who had settled in American coastal cities and brought their maritime expertise with them. Alpheus and William Packard hunted whales for four years off the California coast. The station's crew reflected the multiethnic reality of nineteenth-century whaling: Portuguese, Africans, Irish, Spaniards, Mexicans, and Englishmen all worked the California whale fishery. The station processed gray whales to produce whale oil, which burned cleanly in oil lamps, served in miners' headlamps and lighthouse beacons, and found use in soaps, candles, and machinery lubrication.

California gray whales followed a predictable seasonal path: south to warm breeding lagoons in Baja California in winter, north to rich subarctic feeding grounds in summer, and back again. Ballast Point sat directly on this route. The whaling station operated from 1858 to 1873, when it closed — not because the whales ran out, though their numbers had declined dramatically, but because petroleum had made whale oil economically uncompetitive. Kerosene was cheaper and more abundant. A second whaling station operated at La Playa, not far away, during the same period. San Diego whaling ended in the 1880s. Today the gray whale has recovered and is no longer endangered, protected under the Marine Mammal Protection Act along with humpback, blue, and right whales.

The transformation of Ballast Point after the whaling station closed is a compressed history of American military expansion. In 1869, the United States government acquired the site for a quarantine station and lighthouse. Fort Rosecrans was established in 1873, named after Major General William Rosecrans of Civil War fame. Through World War I and World War II, Rosecrans maintained gun batteries defending San Diego Bay. The Ballast Point Light operated from 1890 to 1957. In 1946 the Navy took over the site for a submarine base; today it is Naval Base Point Loma, founded in its current form in 1959 and home to the Pacific Fleet's attack submarines. The whaling station's ruins are somewhere underneath or adjacent to all of this, accessible only to those with base access.

Even the whaling station was not the first human claim on Ballast Point. Before 1858, the site was Fort Guijarros — the Spanish fortification built in 1797 to guard the entrance to San Diego Bay. The fort fired its guns in anger only twice: once against an American trading vessel, the Lelia Byrd, in 1803, and once against an American smuggler in 1828. In 1846 during the Mexican-American War, U.S. Marines landed nearby, took abandoned guns from the fort, and used them against the Mexican-held Old Town. The Spanish fort, the whaling station, the military installations — Ballast Point is a palimpsest, each era writing over the previous one, with the whale oil era now the hardest to read.