Designed by John McComb Jr. with Lieutenant Colonel Jonathan Williams as consulting engineer, the fort was built from 1808 to 1811 on an artificial island just off the shore of Lower Manhattan. Officially named West Battery, it bristled with 28 thirty-two-pounder cannons pointed toward the harbor, ready for a British attack during the run-up to the War of 1812. That attack never came. The fort was garrisoned in 1812 but never saw combat. By 1821, the Army abandoned it, and Congress ceded the structure to New York City in 1822. The bridge connecting it to Manhattan became more popular with fishermen than with soldiers. The military chapter had lasted barely a decade, and the real story of Castle Clinton was only beginning.

In 1824, the city converted the fort into Castle Garden, a 6,000-seat entertainment venue. A roof was added around 1845, transforming the open-air fortification into a proper theater. The space attracted presidents Andrew Jackson and John Tyler, but its most famous night came on September 11, 1850, when Swedish soprano Jenny Lind gave her American debut. Tickets sold for up to $225, a staggering sum at the time. The following year, prices dropped to 50 cents for "popular" concerts. European dancing star Lola Montez performed her notorious "tarantula dance" on the same stage, and Louis-Antoine Jullien gave dozens of concerts mixing classical and light music. Castle Garden was the cultural heart of mid-century New York.

On August 1, 1855, Castle Garden transformed again, becoming the nation's first immigrant processing depot. Before Ellis Island existed, this was where the world came to enter America. Over 35 years, between 7.5 and 8 million immigrants passed through its rotunda, the vast majority from Denmark, England, Germany, Ireland, Italy, Russia, Scotland, and Sweden. Clerks registered each arrival, advised them on destinations, and sold train tickets for the journey inland. The name Castle Garden became so synonymous with the immigrant experience that German-speaking and Yiddish-speaking arrivals turned it into a generic word for any chaotic, multilingual situation. By 1874, the New York Times reported that Castle Garden was "so well known in Europe that few emigrants can be induced to sail to any other destination." The depot closed in 1890, its functions moving to the new facility on Ellis Island.

In 1896, the fort reinvented itself once more as the New York Aquarium. Engineers lined the circular interior with over 100 tanks on two levels, supplied by fresh water from the city system and salt water pumped from the Hudson River. The aquarium drew two million visitors per year by the early 1900s. Charles Haskins Townsend took over as director in 1902 under the New York Zoological Society, adding classrooms and upgrading the facilities. But the old fort was never properly adapted for its aquatic role: mechanical equipment flooded at high tide, coal bunkers needed manual replenishment every four days, and the second story's wooden frame was a fire hazard. Still, the aquarium thrived for 45 years, until 1941, when Robert Moses closed it to make way for the Brooklyn-Battery Tunnel.

Moses wanted Castle Clinton demolished, dismissing the fort that "never fired a shot." What followed was one of New York's fiercest preservation battles. The Board of Estimate voted to demolish the fort no fewer than six times between 1942 and 1948. Preservationist George McAneny fought for nine years to save it. A Brooklyn junkyard operator offered $1,120 to buy the structure. The state Supreme Court issued and then revoked injunctions. Finally, in 1946, Congress designated Castle Clinton a national monument, and President Truman signed the bill. But the city still owned it, and the legal tug-of-war continued until Governor Dewey and the state legislature finally transferred ownership to the federal government in 1949. Castle Clinton reopened on May 25, 1975, with a performance of Beethoven's Ninth Symphony by the American Symphony Orchestra. Since 1986, it has served as the departure point for ferries to the Statue of Liberty, welcoming over three million visitors each year.