The original 1790 structure was a modest Federal-style dwelling, the kind of restrained, symmetrical building favored by the newly independent republic when it was still figuring out what American architecture should look like. Around 1834, the house was enlarged, gaining the proportions that make it striking today: three full stories of white-painted clapboard, deep porches running across both the front facade and the back, and an upright, almost ship-like presence among the live oaks. By then, New Bern was a prosperous port shipping naval stores, lumber, and tobacco out to the Atlantic, and the Attmore-Oliver House grew with the town.

The house carries the names of two families that owned it across generations. Today, the New Bern Historical Society owns the building and uses it as offices and event space, no longer running the property as a full house museum. The interior still holds period furnishings and family belongings collected during the Society's years operating it as a museum, but visitors now see limited displays, opened for self-guided tours by appointment. It is a working headquarters dressed in 18th- and 19th-century clothes - the kind of place where someone is just as likely to be writing a grant application as polishing a silver tea service.

The house has earned a reputation as one of the most haunted in a town that takes its ghosts seriously. New Bern stories include a Civil War cannonball that allegedly arrived through one wall during the 1862 Union assault on the town and lodged itself somewhere in the structure. Other accounts describe unexplained noises and the sensed presence of former occupants. The Historical Society does not vouch for the paranormal claims, but the stories are part of what draws people to ghost tours that wind through Broad Street on autumn evenings. Whether the house is haunted or simply old enough that 234 years of human living have left an emotional residue, the effect on visitors is much the same.

511 Broad Street sits inside the New Bern Historic District, one of the largest and most intact concentrations of pre-Civil War architecture on the southern Atlantic coast. Federal, Georgian, Greek Revival, Italianate, and Queen Anne houses crowd the surrounding blocks, each with its own family stories and architectural quirks. The Attmore-Oliver House anchors a corner of that ensemble - taller than most of its neighbors, painted in the kind of crisp white that makes the surrounding green seem greener. From the air, it appears as a bright rectangle near the bend of the Trent River, just inland from where the Trent meets the Neuse, where Christoph de Graffenried's Swiss and Palatine settlers founded the town in 1710.