Long before Kingsley arrived, Fort George Island held layers of human habitation stretching back 12,000 years. The Timucua people, specifically the Saturiwa tribe, were the dominant indigenous group, numbering about 14,000 across 35 chiefdoms. They left behind massive oyster shell middens that later builders would repurpose as construction material. A Spanish mission called San Juan del Puerto gave the nearby St. Johns River its name. The British established the island's first plantation in 1765, when Richard Hazard grew indigo with enslaved Africans. Ownership cycled through Spain and the United States, through a bankrupt Revolutionary War veteran and a failed American insurgent, before Zephaniah Kingsley purchased the property and made it his own for 25 years.

Born in Bristol, England in 1765, Kingsley was a slave trader and shipping magnate who published a defense of slavery in 1828 while simultaneously arguing that race alone should not determine servitude. He maintained a polygamous household with four African women, Anna Jai foremost among them. He displayed his multiracial children proudly at the dinner table. He gave enslaved people padlocks for their cabins, armed them for protection, and trained them as skilled artisans whose sale price ran 50 percent above market rate. He wrote about not interfering in his slaves' family lives and encouraged "dancing, merriment and dress" on Saturday afternoons and Sundays. Historians have debated whether this was genuine humanity or a calculated strategy to prevent runaways. The truth is probably both.

The plantation's most haunting features are the remains of 25 slave cabins arranged in a semicircular arc, a formation unique among antebellum plantations in the United States. Each cabin had a single room, a fireplace, and a sleeping loft, all constructed of tabby concrete, a durable mixture of oyster shells, lime, sand, and water. Historian Daniel Schafer suggests that Anna Jai may have been responsible for this distinctive layout, as West African villages were commonly built in circular patterns with the ruling family at the center. Archaeological excavations in 2006 uncovered an intact sacrificed chicken atop an egg inside one cabin, evidence that enslaved Africans preserved their traditions across the Atlantic. In 2010, a cemetery was discovered containing six graves, including three adults likely born in West Africa.

When the Florida Territorial Council passed laws forbidding interracial marriage and property inheritance by free Blacks, Kingsley sent his wives, children, and some enslaved people to Haiti. He sold the plantation to his nephew in 1839 and started anew in the Caribbean. He wrote longingly of the Haitian valley where Anna presided: "You ought to go, to see how happy the human race can be." After his death in 1843, Anna returned to Florida and won a court battle against white relatives to retain the family's holdings. The family line endured. Kingsley and Anna Jai are the great-grandparents of Mary Kingsley Sammis, who married Abraham Lincoln Lewis, founder of American Beach. Their descendant Johnnetta Betsch Cole became the first Black female president of Spelman College.

The State of Florida acquired the plantation in 1955, and the National Park Service took over in 1991 as part of the Timucuan Ecological and Historic Preserve. The main house, built between 1797 and 1798, still stands with its widow's walk and secret walled-up cellar spaces. The kitchen house, known as "Ma'am Anna House" during Anna Jai's time, has been restored. The tabby slave cabins endure despite vandalism, their roofless walls open to the sky. Each year since 1998, the Kingsley Heritage Celebration brings descendants together on the island. The plantation asks visitors to hold contradictions in their minds: a slave trader who freed his wife, an enslaved woman who managed other slaves, a family that defied racial categories while profiting from the system that created them.