Captain John Smith, the Jamestown explorer, mapped the island in 1608, and his name stuck. In 1614 Governor Thomas Dale sent twenty men under Lieutenant William Craddock to set up a saltworks here, boiling Atlantic seawater down to crystal for the struggling Virginia colony. The men lived on the mainland at a place called Dale's Gift along Old Plantation Creek, but they kept the boilers running on Smith Island. The saltworks did not last long. What replaced it, for the next three centuries, was a quieter and more profitable use of the land: grazing. Smith Island was one of the Virginia Barrier Islands, low and salt-grassed and edged with sand, a place where cattle could be turned out and largely forgotten until rounding-up time.

John Custis received Smith Island as a colonial grant in 1691, and his descendants held it for the next two and a quarter centuries. Among them was Martha Custis, the young widow of Daniel Parke Custis who later married George Washington; she carried the island into her second marriage as part of her substantial estate. Her great-granddaughter Mary Custis married Robert E. Lee, and that is how Lee came in 1832, a young Army lieutenant stationed at nearby Fort Monroe, to ride out and inspect his wife's family's barrier island. He found four tenant families farming there, each with thirty to forty head of milking cattle, and roughly 150 head of wild cattle and a hundred wild sheep wandering free. The soil of the glades, he reported, was "as rich as possible and covered with fine grass." The ridges held pine and sand. The Custis-Lee family held the island until 1911.

Edward Teach, the pirate Blackbeard, found the area off the Virginia Capes fertile pirating ground in the early eighteenth century, and Smith Island made a convenient stopover. The careening was practical: hauling a sloop onto a soft beach, tipping her over, scraping the barnacles and worms from her exposed hull so she could outrun anything that came after her. A creek and a cove on the island still carry his name today, three centuries after the fact. Blackbeard's 1717 capture of the Betty just off Cape Charles, in partnership with his old mentor Benjamin Hornigold, was a representative score. They took her Madeira wine, took her other valuables, and sank the empty ship. None of it would last him much longer; Blackbeard was killed in a Virginia inlet a year and a few months later.

Cattle grazed Smith Island into the 1920s. In the mid-twentieth century a waterfowl hunting club moved in for the seasonal trade. Three steel skeleton watchtowers went up on the island during World War II, built by the Army for coastal observation; they still stand. The Cape Charles Lighthouse, an octagonal steel skeleton 191 feet tall and the second-tallest in the United States, dominates the island's skyline. It is the third tower at the location, the previous two having migrated into the Atlantic as the barrier island slid westward under the surf. Smith Island has been uninhabited since 1963, when the lighthouse was automated and the keepers walked off. The 1895 head keeper's house burned in a brush fire on July 13, 2000. Since 1995 the island has belonged to The Nature Conservancy, which now manages it as a preserve. The Cape Charles light was discontinued in 2019.