Charles Cluskey moved to Savannah in 1829 from New York City, where he had apprenticed under the architectural firm of Town and Davis. He brought a sophisticated eye for blending Greek Revival grandeur with English Regency elegance, and the Sorrel-Weed House became his finest residential commission. The National Trust Guide to Historic Places draws comparisons to Thomas Jefferson's Monticello and William Jay's Owens-Thomas House nearby in Savannah. Cluskey borrowed a device from Jay: dividing the foyer with two columns to separate guest space from private family space. But where Jay placed the columns near the stairway, Cluskey set them closer to the front door, creating a small waiting area where visitors would be greeted before entering the main hall. The central stairway mimics the Regency design of the Owens-Thomas House, ascending to a mid-floor landing from which one can proceed left or right to the second floor.

Francis Sorrel (1793-1870) was a wealthy shipping merchant and esteemed citizen of Savannah. His son, Moxley Sorrel, grew up in this house and went on to become a Brigadier General in the Confederate Army, serving under General James Longstreet. After the war, Moxley Sorrel wrote "Recollections of a Confederate Staff Officer," considered one of the finest postwar accounts of the conflict. Robert E. Lee, who had been friends with Francis Sorrel since the early 1830s, visited the home in late 1861 and early 1862 while commanding the Department of South Carolina, Georgia and Florida. Lee returned to visit the Sorrel family in April 1870, shortly before his death. The house also confronts a darker history: Francis Sorrel owned between three and eleven enslaved individuals per year throughout his life.

In January 1940, the Society for the Preservation of Savannah Landmarks chose the Sorrel-Weed House for its first public exhibit, a loan exhibition of eighteenth and nineteenth century furniture and fine arts. That society later became the Historic Savannah Foundation, making this house the birthplace of Savannah's preservation movement. In 1941, businessman A.J. Cohen Sr. purchased the home, and the Cohen family lived there for over fifty years. His son built a brick addition and opened Lady Jane, an upscale women's clothing store that thrived for decades before closing in 1991. Stephen Bader bought the house in 1996 and removed the Cohen-era addition. In 1999, a Victorian stairway added by the Weed family in the late nineteenth century was demolished, and the original Sorrel-era stairway design was recreated based on architectural analysis by Willie Graham, Curator of Architecture at Colonial Williamsburg.

The Sorrel-Weed House has been investigated by nearly every paranormal television program in America. The Atlantic Paranormal Society (TAPS) came during a special 2005 Halloween episode of Ghost Hunters. Ghost Adventures filmed in 2014. Buzzfeed Unsolved sent Ryan Bergara and Shane Madej. The house appeared on the Travel Channel's "Most Terrifying Places in America" in 2010, HGTV's "If Walls Could Talk" in 2006, and the Paula Deen Network in 2015. Conde Nast listed Savannah as one of America's best Halloween cities, citing the house specifically. Today the Sorrel-Weed House Museum conducts historic tours by day and ghost tours every evening, standing at the corner of Bull and Harris Streets as Savannah's premier haunted destination. Archaeological excavations have debunked some of the spookier legends, but the tours continue undeterred.