The first permanent settlement on Oak Island took place in the 1830s at the island's eastern tip, where the federal government began building a brick coast artillery fortification called Fort Caswell to guard the mouth of the Cape Fear River. The Cape Fear was, then and now, the only deepwater river port in North Carolina, and Wilmington upstream was the largest city in the state for most of the nineteenth century. The fort guarded it the way Charleston was guarded by Sumter and New Orleans by Pickens and Jackson. Confederate forces held Fort Caswell through most of the Civil War, then destroyed the fortifications themselves in January 1865 as Union forces closed in. The fort was rebuilt afterward, then pressed into service again in World War I as a small-arms training base, and again in World War II as a patrol and communications post. In 1949 the federal government sold it to the North Carolina Baptist Assembly, which has used it as a Christian retreat and conference center ever since. The brick batteries and concrete bunkers are still there, integrated into the campus.

Frying Pan Shoals run twenty-eight miles southeast from the mouth of the Cape Fear River, a long submerged ridge of sand that has been catching ships since colonial times. The combination of Frying Pan, Jay Bird Shoals, and Bald Head Shoals made the approach to Wilmington one of the most dangerous on the Atlantic coast. Lighthouses started going up early. Old Baldy, the oldest standing lighthouse in North Carolina, was completed on Bald Head Island in 1817. The Oak Island Life Saving Station went up in 1889, a half-mile west of Fort Caswell, with a tall lookout tower projecting from its roof. The current Oak Island Lighthouse became operational in 1958, a tall cylindrical tower painted in three distinct horizontal bands. It was among the last major lighthouses built in the United States, deliberately constructed without the spiral keeper's stairs of older lights because by then the lights were automated and a stair-climbing keeper was an obsolete profession. Visitors can still climb to the top by appointment, on a steep ladder system the original engineers designed for periodic maintenance access.

On October 15, 1954, Hurricane Hazel made landfall on the North Carolina-South Carolina line as a Category 4 storm at the height of the autumn high tide. Hazel was one of the most destructive hurricanes ever to strike the United States. On Oak Island - then partly developed with vacation cottages on the western end - the storm washed away about 350 buildings on or near the ocean front. The island had been a quiet, sparsely-built barrier strip up to that point. Hazel essentially reset it. The towns of Long Beach and Yaupon Beach were incorporated in 1955, in the immediate aftermath, partly as governance structures to coordinate rebuilding. They merged in 1999 into the Town of Oak Island. The island has been hit again since - by Bertha and Fran and Bonnie and Floyd in the 1996-1999 windrow of bad seasons, by Hurricane Matthew passing almost directly over the island in October 2016, and by Hurricane Florence in September 2018, which washed away most of the beach sand that had been added by restoration projects earlier in the year. Beach restoration on Oak Island is now a continuous process, paid for partly by the towns of Oak Island and Caswell Beach through dedicated Beach Preservation Trust funds, and partly by federal cost-share programs. The beach keeps coming back. So do the storms.

Oak Island is now divided politically into two towns. The Town of Oak Island covers most of the island, including the western portion that took the worst of Hazel in 1954 and is now densely built with vacation rentals and year-round homes. Caswell Beach, incorporated in 1975, occupies the eastern end around Fort Caswell and the lighthouse. The Oak Island Golf Club, designed by George Cobb and opened in 1962, straddles the line - thirteen holes in Caswell Beach and five holes in Oak Island. Two fishing piers - the Oak Island Pier and the Ocean Crest Pier - jut into the Atlantic, and motor and sailboats fill the slips along the Intracoastal Waterway and the smaller Davis Canal and Montgomery Slough. The Run Oak Island race - marathon, half-marathon, 10K and 5K - happens every February. The Fourth of July fireworks happen on the beach. Christmas brings a parade down Oak Island Drive. The island is, in the official town language, a family beach. What that means in practice is a place with low-rise development, wide quiet sand, and a slower rhythm than the high-rise coast a few miles south.