European settlement here began in 1613, when Sir Thomas West, 3rd Baron De La Warr, claimed the land and called it West and Sherley Hundred. Three years later, John Rolfe (Pocahontas's husband, who would not be widowed until her death in 1617) named it among the six European settlements in the colony. Captain Isaac Maddeson commanded twenty-five laborers and farmers. When the Powhatan uprising of March 22, 1622 swept through the river plantations and killed nearly a third of Virginia's colonists, West and Sherley Hundred came through largely intact. A 1623 census counted forty-five men, women, and children on the mainland and another twenty-four on what is now Eppes Island, visible from the front lawn. Tobacco moved downriver to Jamestown and across the Atlantic to England. It would become Shirley's foundation crop and, by extension, the foundation of its labor demands.

Edward Hill arrived in the mid-17th century and patented his way into a sprawling estate. By 1660 he held 2,476 acres on the mainland and the 416-acre island. His son Edward Hill II owned the land through Bacon's Rebellion in 1676. One of them built the first Hill House, eventually torn down for salvage. The current mansion, the Great House, went up nearby and was finished in 1738. In its parlor, on June 18, 1793, Anne Hill Carter married Henry "Light-Horse Harry" Lee, the Revolutionary cavalry officer who had eulogized Washington as "first in war, first in peace." The couple's son, born thirteen years later at Stratford Hall, would lead the Confederate Army of Northern Virginia. Robert E. Lee's mother was born at Shirley, and the ties between Shirley and the Lee family run through everything else that happened here.

Indentured Englishmen worked the early fields, but by the early 1700s, as the indenture flow dried up, Virginia's planters turned to chattel slavery. The 1787 Virginia tax census recorded that Charles Hill Carter held 67 enslaved people over the age of 16 at Shirley and another 67 children, alongside 16 horses and 70 cattle. He held still more enslaved people at his Long Bridge plantation in the same county. Through the 19th century, between 70 and 90 enslaved Africans were forced to work at Shirley each year: plowing the fields, cleaning the house, cooking, doing childcare, doing laundry. The two-story kitchen with quarters above it, the laundry building with quarters above it, and the slave quarters identified in the 1979 and 1980 archaeological excavations by William and Mary teams all testify to the architecture of forced labor. After emancipation in 1865, Hill Carter retired and divided the estate; his son Robert kept what is now Shirley, and his other son William Fitzhugh Carter inherited Upper Shirley, today a vineyard.

Shirley was added to the National Register of Historic Places in 1969 and named a National Historic Landmark in 1970. The Great House still stands, surrounded by Queen Anne–era support buildings: a two-story kitchen, a two-story laundry, a smokehouse, a stable, an ice house, a large storehouse, and a brick dovecote that may be the oldest in North America. The James River slides past below the lawn. After Tuttle Farm in Dover, New Hampshire was sold and rebranded, Shirley took on the title of oldest continuously operating business in the United States, with the Hill family running it as a working farm since 1638. The plantation now interprets the lives of the enslaved alongside the family story, an honest reckoning that the marketing materials of an earlier era preferred to leave out.