Pea Patch Island defeated two fortifications before the third one held. In 1794, French military engineer Pierre Charles L'Enfant surveyed the island - which he called Pip Ash - and declared it ideal for harbor defense. Construction of a star fort began before 1817, designed by Joseph G. Totten and supervised by Captain Samuel Babcock. The marshy ground fought back. Uneven settling and improper pile placement caused massive cracking; in one instance, 43,000 bricks had to be taken down, cleaned, and relaid. Babcock was court-martialed for deviating from Totten's plans, though he was acquitted. A fire destroyed much of the work, and the entire star fort was torn down in 1833. Captain Richard Delafield designed a replacement, a massive polygonal fort, but a decade-long legal battle over the island's ownership halted construction. It took an arbitrator's ruling in 1848 - confirming that Delaware owned the island and had validly transferred it to the federal government - before a third attempt could proceed. The present fort, an irregular pentagon with positions for 169 cannon, was erected between 1848 and 1860 atop nearly 5,000 piles driven through forty feet of compressible mud.

The first prisoners arrived in sealed-off casemates, empty powder magazines, and two small rooms inside the sally port, where Confederate names carved into the brick are still visible. As the war escalated, wooden barracks called the bull pen were built outside the fort for enlisted men and junior officers. The first Confederate general imprisoned there was Brigadier General Johnston Pettigrew; about a dozen generals would follow. Rations told a story of escalating deprivation. In 1864, the War Department cut rations in retaliation for the treatment of Northern soldiers in Southern camps. Private Henry Berkeley described dinner: three hardtack, a piece of meat amounting to about three bites, and a tin cup of bean soup, twice a day. Captain Robert E. Park of the 12th Alabama noted the food arrived on greasy tables. Yet prisoners could buy extras from the sutler, fish in the waterways and the Delaware River, and Fort Delaware's official records show it received more care packages than any other Union POW camp in the country.

On August 20, 1864, six hundred Confederate officers boarded the steamer Crescent at Fort Delaware, bound for Morris Island off the coast of South Carolina. Their purpose: to be placed under fire from Confederate batteries, in retaliation for Union officers held in Charleston who were exposed to their own army's shelling. Captain Leon Jastremski of the 10th Louisiana Infantry wrote that he welcomed the move as "a diversion to the monotonous life led in prison." Reverend Isaac Handy, a political prisoner watching from the sally port, described the departure: "Good-bye! Good-bye! was uttered, time and again, as the files moved on." This group became known as the Immortal Six Hundred. Meanwhile, the garrison defending the island was itself a remarkable patchwork: volunteer units from New York, Pennsylvania, Ohio, Delaware, Maryland, and Massachusetts, plus Ahl's Battery - a unit composed entirely of Confederate prisoners who had taken the oath of allegiance to the Union. In August 1864, some 255 men staffed each guard shift across roughly 85 posts ringing the island.

After the Civil War, Fort Delaware continued as a small artillery garrison. In the 1890s, the Endicott modernization program transformed it into part of a three-fort system with Fort Mott in New Jersey and Fort DuPont in Delaware City, defending the Delaware River with modern 12-inch guns. Battery Torbert, built inside the fort, became one of only two three-story Endicott batteries in the United States. Nature battered the island relentlessly. A hurricane in October 1878 destroyed most buildings on the south side, including the Trinity Chapel built by Confederate prisoners in 1863. A tornado in 1885 demolished the hospital. In 1919, soldiers mothballed the fort with orders to bury everything - and they did, including three pieces of artillery. Only one has been recovered: a 15-inch Rodman gun sold for scrap during World War II. Today Fort Delaware is a state park, reached by ferry from Delaware City. The Army Corps of Engineers built a 3,500-foot seawall in 2005-2006 to protect both the fort and a migratory bird rookery considered the largest north of Florida.