The house itself was built around 1740 by a German immigrant named Jacob Ege, who had come from Germany to Philadelphia in 1738 and continued south to the James River settlements with the family of his fiancee, Maria Dorothea Scheerer. He built the stone walls as what tradition calls a 'Home for the Bride.' Dendrochronology shows additional construction in 1754. Jacob died in 1762; his brother George Ege went on to serve as a U.S. Representative from Berks County, Pennsylvania. The house stayed in the family until 1911, surviving the Civil War, Reconstruction, and the rapid commercial development of Shockoe Bottom around it. By the time the Ege descendants sold, the house was already two and a half centuries old by Richmond standards - which is to say, it was the city's deep memory in stone. Preservation Virginia saved it in 1911 and opened it to the public in 1922 as part of the new Poe Shrine.

Poe's connection to Richmond was real, even if he never slept under this particular roof. He grew up here after his actress mother Eliza died and the orphaned boy was taken in by the wealthy Allan family. He worked here as a literary critic and editor at the Southern Literary Messenger, the magazine where he first made his reputation. He lived in several houses within walking distance of the Old Stone House - the museum is only blocks from each of them. Eliza Poe lies in the graveyard of St. John's Church on Church Hill, a few blocks east. The museum collected what survived: his vest, his trunk, his walking stick, a lock of his hair, an 1845 publication of 'The Raven,' one of only twelve known existing copies of his first book, Tamerlane and Other Poems. The North Building explores his mysterious death - over twenty-six published theories exist, with the museum favoring the nineteenth-century practice of cooping, in which voters were drugged or beaten and dragged from polling place to polling place to cast multiple ballots.

Behind the museum is a small courtyard the founders called the Enchanted Garden, laid out around motifs from Poe's writing. The design takes its inspiration from his poem 'To One in Paradise.' There is a fountain. There is a shrine. There is a rock inscribed with a character's name from 'A Tale of the Ragged Mountains.' There is a brick wall that recalls the one described in 'William Wilson.' And there are two resident black cats - Edgar and Pluto - who were found as stray kittens behind the shrine and have lived at the museum ever since, providing a recurring living echo of the cat at the center of one of Poe's most famous stories. The garden is also available for weddings, which is its own quiet commentary on how a writer obsessed with premature burial has become, in his adopted city, a setting for happy endings.

The museum has attracted its own ecosystem of devotees over the past century. The actor Vincent Price, whose film career became inseparable from Poe adaptations, visited in 1975 and posed with the museum's famous stuffed raven. In 2014 his daughter Victoria Price came to the museum and said that Poe had been such a presence in her childhood that she thought of him as her uncle. She returned in 2016 for an International Poe Film Festival that screened movies at Richmond's historic Byrd Theatre, where her father's voice still echoes. The museum sits in three buildings now: the Old Stone House itself, the Elizabeth Arnold Poe Memorial Building with its rare editions and daguerreotypes, and the North Building with its death exhibits. The whole complex is a quiet rebuke to the 1909 council that wouldn't put Poe on Monument Avenue. The city did not honor him; his readers did.