Caswell County sits in the Piedmont Triad along the Virginia border, a 428-square-mile rectangle of rolling forest and farmland whose population today is barely larger than it was in 1950. The land was part of the Granville District until 1777, when the Revolutionary government carved a new county out of northern Orange and named it for Richard Caswell, the first governor of independent North Carolina. The earliest settlers were Scotch-Irish and German families who had come down the Great Wagon Road from Pennsylvania, and English and French Huguenot migrants who had moved inland from the coast on the older Great Trading Path. By 1786 the county was the second-largest in the state by population. Yanceyville, the county seat since 1791, sits at the center of a network of tobacco roads that once ran north into Virginia and south to the river port at New Bern. Now those roads carry mostly retirees and cattle trucks.

Slade's accident in 1856 changed the regional economy. Bright leaf cured to a milder, sweeter smoke than the dark fire-cured tobaccos that had dominated the trade, and demand for it would eventually reshape American agriculture. But the Caswell County boom of the 1830s-1860s was built on the labor of enslaved African Americans, whose numbers had grown from roughly one-third of the county's population in 1800 to fifty-two percent by 1850. They cured the leaves Slade perfected, worked the fields he profited from, and built the brick homes in Milton and Yanceyville that still anchor the county's two historic districts. The Bank of Yanceyville, chartered in 1852, briefly held one of the highest market capitalizations of any bank in North Carolina. Reconstruction broke the system that had built that wealth. The planter class lost its labor force, the tobacco industry consolidated in Durham and Danville, and Caswell County, having been almost completely an agricultural economy, had very little to fall back on.

On a spring day in 1870, a Republican state senator named John W. Stephens went down to the basement of the Caswell County Courthouse in Yanceyville. He was a local Freedmen's Bureau agent and one of the most effective Black-rights organizers in the state. A delegation of Conservative Democrats - several of them members of the Ku Klux Klan - waited for him in a small storeroom. When his body was found, the news traveled quickly to Governor William W. Holden in Raleigh. Holden, a Reconstruction Republican who had endorsed universal male suffrage at the state convention three years earlier, declared insurrection in Caswell and Alamance counties, called in a militia under former Union colonel George W. Kirk, and arrested dozens of suspected Klansmen. The episode became known as the Kirk-Holden War. It ended with Holden's impeachment by a Conservative-controlled legislature that December, the first removal of a sitting governor in American history. Stephens's killers were never tried in life; their confessions only emerged after their deaths. The courthouse still stands, restored, on the Yanceyville square.

About twenty miles north of Yanceyville, in the small town of Milton on the Dan River, a free Black cabinetmaker named Thomas Day ran the largest furniture shop in North Carolina from the 1830s until the Civil War. He bought and used the Union Tavern as his workshop, employed both free Black and white apprentices, and built ornate Greek Revival pieces for the planter families whose system kept his own people in bondage. His furniture survives in museums from Raleigh to New York. The Union Tavern - now called the Thomas Day House - was severely damaged by fire in 1989 and has been under restoration ever since. Day's contradictions are the county's contradictions: bright leaf tobacco and enslaved labor; a Klan murder in the courthouse basement and a Reconstruction senator named Wilson Carey who served three terms in the state legislature despite it; a county that named a junior high school for the African American principal Nicholas Longworth Dillard in 1969 after federal courts finally forced it to integrate. Donna Edwards, who would become Maryland's first Black congresswoman in 2008, was born here in Yanceyville. The county still claims her.