The estate began with a real estate transaction. In 1777 Thomas Shore bought 144 acres on Archer's Hill from the heirs of John Martin and built a two-story clapboard house, finished in 1778, fitting the canonical two-over-two center-hall plan of late colonial Tidewater Virginia. From its windows you could see the falls of the Appomattox and the town of Petersburg across the river. The hill commanded the river crossing. Whoever held it controlled the road south. That fact would matter, repeatedly, over the next century.

In March 1781, Baron von Steuben and his Virginia militia fought a skirmish on the hillside against British troops under Major General William Phillips. A month later, in April, the twenty-three-year-old Lafayette returned to the same ground. Phillips noted the position in dispatches, conceding heavy losses in attempts to take it. Phillips died of typhoid fever on May 13, 1781, buried in an unmarked grave near Blandford Church so the rebels would not desecrate it — among the most senior British officers to die and be interred in North America during the war. Cornwallis took command but judged he lacked the strength to dislodge Lafayette from the Heights, and so chose to march south. That decision drew him eventually to Yorktown, and to surrender. Lafayette would return to Violet Bank in 1824 as an honored guest of the United States on his triumphal final American tour.

The original house caught fire in 1810, supposedly on a Sunday while the family was at church. By then Thomas Shore had been dead a decade, and his much younger widow Jane Grey had remarried, to Henry Haxall of the Haxall milling family in Richmond. The chimney stacks and foundation survived the fire, unusual for the period, when lime mortar typically failed in intense heat. Between 1810 and 1815 a new house rose on the old footprint. Architectural historians generally credit the design to a pupil of Benjamin Latrobe, the British-born architect of the U.S. Capitol who had visited Petersburg and gone to the races with Thomas Shore in 1796. The house's octagonal bays, recessed portico, and Adam-style ceiling moldings could have come from a Latrobe sketchbook. The masonry was stuccoed and scored to look like ashlar, hiding where the new brickwork met the old.

From June through September 1864, during the Siege of Petersburg, Robert E. Lee made Violet Bank his headquarters. The property was held in trust by Thomas Shore's grandson, Thomas Gilliam. From the lawn Lee could watch the trenches across the river. Part of the main house served as a military hospital. The cucumber magnolia is said to have shaded Lee's tent, an image the museum has preserved as fact even if the documentation is thinner than the legend. The war impoverished the Gilliams, as it impoverished most of plantation Virginia, and the family sold the house in 1873. Beneath the surface of any Lee-era story at Violet Bank, an honest reckoning must hold the unnamed enslaved people who had worked the property for the Shores, the Haxalls, and the Gilliams across three generations.

The history that followed is the strangest part. In 1905 the Greater Petersburg Realty Corporation bought the house and turned it into a dairy farm, knocking out walls on the first floor so the structure could be used as a cattle barn. In 1914 the company decided to subdivide the property for housing and tore down the main house, leaving only the wing that survives today. The much-reduced Violet Bank was sold off in 1919, lived in by Alice Pierrepont until 1948, then sold to American Legion Post 284. The City of Colonial Heights acquired it in 1959 and used it first as a Chamber of Commerce office, then as the town's first public library. It was added to the National Register in 1974. Since 1988 it has been a small house museum, a working document of Federal interior decorative arts and one of the more layered single buildings in eastern Virginia.