The place began as Anderson Station, named for John Anderson, a director of the South Western Railroad, when the line was extended from Oglethorpe to Americus in 1853. When the U.S. post office was established in November 1855, the government renamed the station Andersonville to distinguish it from the post office in Anderson, South Carolina. It was a hamlet of perhaps a dozen houses, a depot, a blacksmith shop, a couple of general stores, two saloons, a school, and a Methodist church. Ben Dykes, who owned the surrounding land and served as both depot agent and postmaster, was the closest thing the settlement had to a leading citizen. Nothing about the place suggested it would become one of the most notorious locations in American history.

In February 1864, the Confederate government established Camp Sumter at Andersonville to house Union prisoners of war. The stockade was designed for 10,000 men. By August 1864, it held more than 32,000, and an average of 96 prisoners were dying every day. The causes were grimly predictable: overcrowding, contaminated water, nonexistent sanitation, inadequate food, and exposure to Georgia's summer heat without shelter. Of the approximately 45,000 men imprisoned at Camp Sumter over its fourteen months of operation, nearly 13,000 died. After the war, Captain Henry Wirz, the camp's commandant, was tried for war crimes. He was found guilty and hanged on November 10, 1865, one of only three Confederates executed for war crimes and the only Confederate officer. His trial remains controversial; the United Daughters of the Confederacy later erected a monument to Wirz in Andersonville, calling the proceedings unjust.

Before and after the war, Andersonville's economy rested on agriculture. The dark reddish-brown sandy loams of the region supported cotton as the primary commodity crop, and for a century the town changed little. The transformation came in 1968, when Mulcoa, the Mullite Company of America, began large-scale mining of kaolin, bauxitic kaolin, and bauxite in the area. Scrub oak wilderness gave way to a massive mining and refining operation. Today the Andersonville site is the world's largest producer of sintered mullites for refractories, shipping more than 2,000 tons of refined ore each week. The geological deposits that make this possible, beds of kaolinite clay in the Nanafalia formation of Eocene age, have been accumulating for roughly 55 million years, meaning the ground beneath Andersonville held industrial wealth long before anyone thought to look for it.

In 1974, long-time mayor Lewis Easterlin and a group of citizens decided to rebuild Andersonville's Main Street to resemble its Civil War-era appearance. The decision was practical, aimed at drawing tourists to a town with few other economic prospects, but it also reflected a community choosing to face its history head-on rather than look away. The Andersonville National Historic Site, established on the grounds of the former prison camp, now serves as a memorial not only to Camp Sumter's dead but to all American prisoners of war. The national cemetery adjacent to the prison site holds the remains of those 13,000 Union soldiers in rows that stretch across the Georgia landscape. Andersonville today is simultaneously a working village, a mining town, and a place of pilgrimage, its population of 237 tending a memory that belongs to the entire nation.