By August 1862, Richmond's provost-marshal John Winder had converted three tobacco warehouse buildings between 18th and 19th Streets on the northern side of East Cary Street into a Confederate prison. The location was Tobacco Row, Richmond's industrial corridor where the sweet-leaf economy had built fortunes. Now the fortune being traded was human misery. Castle Thunder housed civilian prisoners, captured Union spies, deserters, political prisoners, and anyone charged with treason against the Confederacy. The guards earned a reputation for brutality, though prisoners were sometimes permitted boxes of medicine and supplies from outside. President Jefferson Davis himself reportedly declared that for every Confederate sailor hanged by the Union, he would hang a Union soldier of equal rank, chosen by lot from among the thousands held in Richmond's tobacco warehouses. Many inmates at Castle Thunder were indeed sentenced to death.

Among Castle Thunder's most notable occupants was William Jackson Palmer, a Union officer who would later found Colorado Springs and build the Denver and Rio Grande Railway. In 1862, Palmer was captured scouting behind Confederate lines after the Battle of Antietam. He was wearing civilian clothes and gathering intelligence for General George McClellan. When questioned, he gave his name as W.J. Peters and claimed to be a mine owner on an inspection trip. The Confederates never identified him as a spy, but his story was suspicious enough for detention, and he was sent to Richmond's Castle Thunder. Palmer endured months of confinement before being freed in a prisoner exchange, rejoining his regiment in February 1863. The experience did nothing to slow his ambition. After the war, he became one of the great railroad builders of the American West.

About one hundred women passed through Castle Thunder during the war. The nearby Libby Prison, equally notorious, held only men. That administrative distinction made Castle Thunder the default holding place for women the Confederacy deemed dangerous. Beyond Dr. Walker, another remarkable story belongs to Mollie Bean. Bean had disguised herself as a man to enlist in the Confederate Army and served two years with the 47th North Carolina Infantry, sustaining wounds in action twice before her identity was discovered. After the fall of Richmond in April 1865, Union forces took over Castle Thunder and used it to detain those accused of unruly conduct. Bean's Union captors, discovering her sex, suspected her of being a spy. The prison that had held women accused of treachery against the Confederacy now held a woman accused of treachery against the Union, all within the same tobacco-stained walls.

When Richmond fell to Union forces in April 1865, Castle Thunder simply changed hands. The Union military used the buildings for their own detention purposes, holding those accused of disorder in the chaotic post-war city. After military rule ended, the property was returned to its original owners. For fourteen years, the warehouse complex sat along Cary Street, its brick walls carrying the weight of everything that had happened within them. Then in 1879, fire swept through the buildings and destroyed Castle Thunder in its entirety. Nothing of the original structures survived. Today, the stretch of East Cary Street between 18th and 19th Streets in Richmond's Shockoe Bottom district shows no visible trace of the prison. The tobacco warehouses that once lined this corridor have given way to restaurants, apartments, and offices. Only the historical record preserves what happened here.