Richard Caswell was North Carolina's first state governor under the new constitution, and the fort named for him was completed in 1836 at a cost of $473,402. The design was pentagonal, with brick walls and large earthworks. It mounted more than 61 gun emplacements, all aimed at the mouth of the Cape Fear River. Wilmington, twenty miles upriver, was then the state's largest city and an important port. Fort Caswell, along with Fort Johnston in Southport and later Fort Fisher across the river, formed the defensive line that protected that port. Frying Pan Shoals extended offshore, making approach difficult even without the cannons. For most of the antebellum era, the fort was a quiet posting.

In 1861, before North Carolina formally seceded, a group calling itself the Cape Fear Minutemen seized Fort Caswell twice. Both times Governor John Willis Ellis ordered them to return it to the lone U.S. Army caretaker stationed there. When the state finally seceded, the Confederate Army turned Caswell and Fisher into what observers called one of the most elaborate defensive systems in the world. Together, they kept Wilmington's port open longer than any other Confederate harbor. Blockade runners slipped through under cover of darkness. The Union devised at least six separate plans to take Fort Caswell. All six were shelved. The fort's guns and the shoals were too much.

Fort Fisher fell on January 15, 1865, after a massive Union assault. Two days later, with the position no longer defensible, the Confederate garrison at Fort Caswell received orders to spike the guns, burn the barracks, and explode the magazines. They did. The roughly 100,000 pounds of powder went up at once, and one wall of the brick fortification came down with it. The blast carried to Fayetteville, a hundred miles upriver. With Caswell and Fisher both gone, the Confederacy lost its last open ocean port. Lee was already losing men and supplies he could not replace, and the closure of Wilmington made the squeeze permanent.

The fort was abandoned after World War I. From 1937 to 1941, someone tried to convert it into a resort. The gun emplacements became swimming pools, fed by two artesian wells producing hot mineral water. It did not work commercially. The Navy purchased the property in 1941 and used it as a patrol and anti-submarine base during World War II, when German U-boats prowled the Atlantic coast just offshore. After the war, the Navy declared the fort surplus. In 1949, the Baptist State Convention of North Carolina bought it for $86,000. The North Carolina Baptist Assembly has operated the site as a Christian retreat ever since.

The Fort Caswell Historic District was added to the National Register of Historic Places in 2013. The listing covers two sites, 43 buildings, and 23 structures, including the original brick walls, the earthworks, and a separate rifle range two miles west in Caswell Beach. The district's boundaries extend half a mile south into the Atlantic and east into the Cape Fear River, recognizing the fort's Civil War blockade-running history and its World War II submarine-hunting role. The natural topography is flat, ten feet above sea level, with a wide sandy beach along the southern boundary. Live oaks and yaupon holly cover most of the grounds. Public access is limited and depends on conditions set by the Assembly's director, which is part of why the place still feels separate from the busy beaches just outside its gates.