Before the war, the building on Tobacco Row at the James River waterfront belonged to Captain Luther Libby and his son George, who ran a ship's chandlery and grocery business from its three stories. In late 1861, the Confederate government leased the structure as a hospital and prison. By 1862, the flood of Union prisoners from the Seven Days Battles, in which nearly 16,000 Union soldiers were killed, wounded, or captured in a single week, forced the Confederacy to reserve the building exclusively for Union officers. The prison contained eight low-ceilinged rooms, each 103 by 42 feet. The second and third floors held the prisoners. Windows were barred but open to the elements, leaving men exposed to Virginia's summer heat and winter cold alike. By 1863, a thousand prisoners were crammed into these spaces, sleeping side by side across every square inch of floor.

The Daily Richmond Enquirer described the scene in 1864 with unsparing clarity: prisoners 'huddled up and jammed into every nook and corner; at the bathing troughs, around the cooking stoves, everywhere there is a wrangling, jostling crowd; at night the floor of every room they occupy in the building is covered, every square inch of it, by uneasy slumberers, lying side by side, and heel to head, as tightly packed as if the prison were a huge, improbable box of nocturnal sardines.' Union surgeons released from Libby reported that over ten percent of prisoners were classified as sick, yet rations consisted of nothing but corn bread and sweet potatoes. Meat had been eliminated entirely. Patients prostrated with diarrhea, dysentery, and fever received food unsuitable for the healthy, let alone the dying. Requests from Union officers for provisions were delayed by prison commandants or simply ignored as the Confederate government funneled its dwindling resources into active combat lines.

In the summer of 1863, amid the suffering, prisoners created something remarkable: a newsletter called The Libby Chronicle, edited by Louis Beaudry and read aloud every Friday morning. Its tone was irreverent, defiant, and darkly funny. Issue number two featured a poem called 'Castle Thunder' cataloguing the menu with mathematical precision: 'We have eighteen kinds of food, though 'twill stagger your belief, Because we have bread, beef and soup, then bread, soup and beef.' The editors reported that 'the uproarious laughter caused by this facetious article has done more good in Libby than cartloads of Confederate medicine.' When some prisoners directed their anger at President Lincoln, blaming him for their prolonged captivity, The Chronicle's editors pushed back, calling such men 'spoiled children' lacking 'the manly courage and intelligence which should characterize the actions of the American soldier.' The newsletter was an act of defiance: proof that captivity could imprison bodies but not the human impulse to create, to mock, and to endure.

In February 1864, 109 Union officers attempted what the press would call the Libby Prison escape. They had dug three tunnels. The first hit water and was abandoned. The second struck the building's log foundation. The third broke through to a small carriage shed 50 feet away. On the night of February 9th, as soon as darkness fell, the exodus began. Captain Morton Tower of the 13th Massachusetts Infantry passed through the tunnel to the yard as the clocks of Richmond struck midnight. By daybreak he had reached a thicket of woods. Tower and his companion, Colonel Davis of the 4th Maine, eluded recapture and eventually joined 57 other escapees who made it back to Union lines. Of the 109 who entered the tunnel, 59 reached safety, 48 were recaptured, and 2 drowned during their flight.

When Abraham Lincoln toured the fallen Confederate capital in April 1865, he came upon Libby Prison. A crowd of onlookers called out, 'We will tear it down.' Lincoln replied, 'No, leave it as a monument.' The building survived fifteen years as a fertilizer warehouse before Chicago candy manufacturer Charles F. Gunther bought it in 1889, had it disassembled brick by brick, and shipped it to downtown Chicago to serve as a Civil War museum. When the museum's commercial run ended a decade later, the building was dismantled again in 1899 and sold as scrap. In 1907, nails salvaged from the prison were melted down and cast into the Pokahuntas Bell for the Jamestown Exposition. Today, the front door of Libby Prison is displayed in the American Civil War Museum at the former Tredegar Iron Works in Richmond. The waterfront site where the warehouse stood shows no trace of what happened there. Lincoln wanted a monument. History gave the building something stranger: a journey to Chicago, a second demolition, and a scattering of relics across the country.