Castle Pinckney was constructed in 1810 as part of the coastal defense network protecting Charleston Harbor, joining the larger and more strategically placed Forts Sumter and Moultrie. The fort saw its first flicker of relevance during the Nullification Crisis of 1832, when President Andrew Jackson prepared to use military force to collect a controversial tariff from South Carolina. A sea wall was completed and troops moved in. Then the crisis passed, and the fort went quiet again, relegated to storing gunpowder and supplies. By the late 1850s, Castle Pinckney bristled with fourteen 24-pounders, four 42-pounders, four 8-inch howitzers, and assorted mortars and field pieces. It looked formidable on paper. In practice, a skeleton crew watched over a fort that everyone regarded as secondary to its bigger neighbors across the harbor.

On December 27, 1860 - one week after South Carolina seceded from the Union - Castle Pinckney's small U.S. Army garrison surrendered to South Carolina militia without a fight. The soldiers withdrew to Fort Sumter to join Major Robert Anderson. With that quiet handover, Castle Pinckney became the first federal military installation seized forcefully by a Southern state government, beating the Charleston Arsenal by three days and Fort Sumter's famous bombardment by months. After the fall of Fort Sumter, the Charleston Zouave Cadets garrisoned the island. In September 1861, 154 Union prisoners of war captured at the First Battle of Bull Run arrived - 120 enlisted men and 34 officers. Richmond officials had selected, as the Charleston Mercury noted, those who had shown "the most insolent and insubordinate disposition." The prisoners wandered the island freely by day and were confined to the lower casemates at night.

Castle Pinckney's career as a prison lasted exactly six weeks. The fort proved far too small for permanent confinement, and on October 31, 1861, the prisoners were transferred to the Charleston City Jail. The fort was then strengthened with earthen embankments and additional heavy weapons on its upper tier. When a devastating fire swept through Charleston in December 1861, damaging the jail, the prisoners were briefly shuttled back to the island for just over a week, many sleeping on the open parade ground, before being moved again. For the rest of the war, Castle Pinckney served as an artillery position, but its guns never saw action against an enemy fleet. After the war, the fort was modernized for possible use during the Spanish-American War. Once again, it was not needed.

In 1890, parts of the old brick walls and casemates were torn down to make room for a harbor lighthouse, despite local efforts to convert the island into a retirement home for veterans. The lighthouse operated into the twentieth century. Castle Pinckney was declared a U.S. National Monument in 1924, but Congress stripped that designation in 1951, handing the island back to the Army Corps of Engineers. By 1956, the federal government declared it surplus property. The South Carolina State Ports Commission bought the island for $12,000 in 1958, hoping to use it as either a spoil area or a tourist attraction. Neither plan worked. The Commission tried to give it back to the federal government, which declined, citing costs that outweighed the island's historical value. Offers poured in to buy the island for a private residence, a nightclub, and a restaurant. All were refused.

A local Sons of Confederate Veterans post took over management in the late 1960s, attempting to establish a museum and preserve what remained. Castle Pinckney was listed on the National Register of Historic Places in 1970, but the preservation group could not raise enough money and eventually let the fort revert to state ownership. A mysterious fire in 1967 destroyed an abandoned house on the island, though a warehouse survived. Limited restoration efforts have been made over the years, but the fort's location on an isolated shoal in the middle of Charleston Harbor makes access difficult and maintenance nearly impossible. Today, Castle Pinckney sits where it has always sat - visible from the ferries heading to Fort Sumter, recognizable as a low shape among the harbor's islands - gradually being reclaimed by salt marsh and vine. A fort that never fired a hostile shot, slowly surrendering to the only opponent it could not outlast: nature itself.