The plantation takes its name from the Carter family, one of colonial Virginia's most powerful dynasties. Robert "King" Carter, born in 1663 at Corotoman in Lancaster County, accumulated vast landholdings across the colony. He purchased the tract that had been Wolstenholme Towne, and when his daughter Elizabeth married Nathaniel Burwell, she received the income from the land while Robert retained ownership. Their grandson, Carter Burwell, inherited the property and built the mansion that still stands, completing it in 1755. Carter married Lucy Ludwell Grymes, but the couple lived in their finished home for less than two years before Carter died in 1756. The estate passed to his son Colonel Nathaniel Burwell, who raised corn and wheat on the property until the Burwell family sold it in 1838.

Carter's Grove passed through a series of owners who each left their mark on the property. Edwin Gilliam Booth, a Virginia lawyer and "New South" advocate, purchased the estate in 1879, two years before the centennial celebration of Washington's victory at nearby Yorktown. Booth painted the mansion's interior red, white, and blue and planted a grove of locust trees along the approach, turning the property into a celebration of national reconciliation. In 1907, New York silver mining investor T. Percival Bisland bought the estate and modernized it with window screens, indoor toilets, and central heat, but both he and his wife died by 1910, leaving the property to languish. Pittsburgh industrialist Archibald McCrea rescued the dilapidated mansion in 1928, hiring Richmond architect Duncan Lee to transform it. Lee raised the low hip roof, added dormer windows, and gave the house a roofline echoing nearby Westover Plantation, significantly altering the original appearance.

The most important discovery at Carter's Grove was invisible from the surface. In the 1970s, archaeologists excavated the remains of the circa 1620 Wolstenholme Towne fortified settlement, owned by an investment group of the Virginia Company of London. The settlement was one of the earliest English communities in the New World, built on the James River in what is now James City County. The Indian massacre of 1622 devastated the population, and the survivors abandoned the site. For centuries, the land was simply farmed. The archaeological project that began in 1976 revealed the foundations, artifacts, and traces of daily life from those first desperate years of the Colony of Virginia. Carter's Grove was declared a National Historic Landmark in 1971, and the Wolstenholme Towne site, along with slave quarters from a later period, were partially restored to represent their respective eras.

The Colonial Williamsburg Foundation operated Carter's Grove from 1969, when the Rockefeller Foundation gifted the property, until 2003, when the site closed to the public. The practical challenge was distance: Colonial Williamsburg was miles away, and few visitors made the journey. Hurricane Isabel in 2003 further severed the connection by destroying trees and damaging the bucolic Carter's Grove Country Road that linked the estate to the Historic Area. In 2007, CNET founder Halsey Minor purchased the Georgian mansion and its grounds for $15.3 million, announcing plans for a private residence and thoroughbred horse-breeding program. He never moved in. Minor filed for bankruptcy in 2013, and Colonial Williamsburg submitted the only bid at auction in 2014. Samuel M. Mencoff, a founder of Madison Dearborn Partners, acquired the property later that year. A conservation easement on 400 of the 476 acres ensures that the James River viewshed, the wetlands, and the irreplaceable archaeological sites remain protected.