Jackson purchased the property in 1804 from Robert Hays, who had settled the land in 1780 on territory from which Native Americans had been displaced. Jackson and his wife Rachel moved into a two-story log blockhouse originally built to resist attacks. A lean-to was added to the back, and log outbuildings went up around it -- slave cabins, storerooms, a smokehouse. This cluster became known as the First Hermitage. Rachel chose the site for the permanent mansion: a secluded meadow set back from the rivers. The couple lived in the log cabin until 1821. The house that replaced it was remodeled in 1831 into the Greek Revival structure that stands today, with four large rooms separated by a center hall, a classicizing domed limestone tomb for Rachel in the garden she loved, and a large brick smokehouse at the rear that cured pork by the ton. When Rachel died in 1828, just weeks after Jackson won the presidency, he buried her in that garden and never fully recovered from the loss.

The Hermitage was a working cotton plantation. When Jackson bought it, nine enslaved people came with the land. By the time he died in 1845, that number had grown to 110, with another 51 held at Halcyon plantation in Coahoma County, Mississippi -- 161 people in total at the peak of his holdings. They planted and picked cotton, the plantation's major cash crop. They built the structures, cooked the meals, tended the grounds. Alfred Jackson was born enslaved at the Hermitage around 1812 and worked there in various capacities for the rest of his life. After the Civil War, he stayed on as a tenant farmer and eventually became caretaker and guide when the Ladies' Hermitage Association purchased the estate in 1889. Alfred Jackson died in 1901 and was buried near the tomb of the president and his wife -- the only enslaved person honored with burial in the family plot, though he had spent his entire life in service to the property.

Jackson died at the Hermitage in 1845. His son, Andrew Jackson Jr., inherited crushing debt and began selling off portions of the estate. In 1856, he sold the remaining land, mansion, and outbuildings to the State of Tennessee, retaining a provision that the family could stay as caretakers. Tennessee intended to convert the property into a southern branch of the United States Military Academy, but the outbreak of the Civil War in 1861 ended that plan. On May 5, 1863, Union soldiers from Indiana marched up to the Hermitage. Private Joseph C. Taylor recorded the visit in his diary. The mansion survived the war largely intact. Andrew Jackson III and his family were the last Jacksons to live there, moving out in 1893. The Ladies' Hermitage Association opened it as a museum dedicated to Jackson's life and the antebellum South, restoring the interior to its 1837 appearance and gradually buying back all the land that had been sold -- a process not completed until 2003.

From 1988 to 2005, archaeological teams excavated the Hermitage grounds and uncovered what the curated mansion could not show: the material reality of enslaved life. They found an ice house behind the smokehouse. They identified 13 dwellings used by enslaved workers -- at least two log houses and four brick duplexes. They located the cotton gin and cotton press in a field beyond the First Hermitage. Hundreds of thousands of artifacts emerged from the soil. In 2005, the Hermitage mounted an exhibition on slave life at the plantation, installed in the Visitor Center, focusing on the domestic staff and field laborers who made the plantation function. Then, in 2006, the remains of 61 enslaved people were discovered nearby -- property of Rachel Jackson's nephews, buried in unmarked family groups, their ages ranging from 1 to 45. A memorial built in 2009, designed by Aaron Lee Benson, features seven trees planted in the shape of the Little Dipper -- the constellation enslaved people followed north to freedom.

In 2024, researchers found yet another burial ground on the property, believed to hold the graves of 28 enslaved people. The search had taken more than 20 years. Some historians speculate that Lyncoya Jackson, a Creek orphan Andrew Jackson adopted from a battlefield and raised at the Hermitage, may be buried in the enslaved cemetery -- his final resting place unknown despite his unique place in the household. The Hermitage also holds a Confederate Soldiers Home Cemetery, where more than 480 veterans of the Civil War were buried in circles around a monumental stone after the state opened a residential facility for poor and disabled Confederate soldiers in 1892. Gibson Guitar Corporation once made 200 limited-edition "Old Hickory" guitars from timber blown down on the property, sending the first one to the Smithsonian. The Hermitage is a place where stories keep surfacing -- from the soil, from the archives, from the silence. It asks visitors not simply to admire a president's home, but to reckon with everything and everyone that home contained.