Long before any English boot touched it, the bluff above the Appomattox falls belonged to the Appamattuck people. Their principal town and its leaders, the weroance Coquonasum and his sister Oppussoquionuske, appear on Captain John Smith's 1607 map. Just downstream, Swift Creek emptied into the Appomattox at a confluence the colonists named Conjurer's Neck because, the story went, a Native healer had once lived there and was thought to cast spells on the waters. English colonists sailed up looking for clear land in 1620 and stayed. The Appamattuck, after the violent reorderings of 1622 and 1644, were forced to relocate downstream near Fort Henry, where modern Petersburg now stands.

The slope's military value announced itself in May 1781. The Marquis de Lafayette, twenty-three years old and commanding Continental troops in Virginia, moved south from Richmond and positioned artillery on the heights overlooking Petersburg, where British forces under Major General William Phillips were dug in. Lafayette shelled the town from a position behind a boxwood hedge on what was then called Archer's, Hector's, or Dunn's Hill, around the surviving Oak Hill house. Phillips died of typhoid during the bombardment and was buried in an unmarked grave so the Americans could not desecrate it. Five months later, Cornwallis surrendered at Yorktown, but the British general had already concluded the Heights could not be taken without overwhelming force he did not have.

Eighty-three years later, the same high ground served a different general. From June through September 1864, Robert E. Lee made his headquarters at Violet Bank, the Federal-style house just upstream of where Lafayette had positioned his guns. From the lawn, Lee could watch the smoke rise from the Petersburg trenches across the river. The cucumber magnolia on the property is said to have shaded his tent. Lee left Violet Bank in September, but the siege ground on through winter and into the spring that ended at Appomattox. Today the house is a small museum in a residential neighborhood of Colonial Heights, surrounded by the postwar town that grew around it.

The bluff did not become Colonial Heights, the place name, until a 1906 real estate subdivision. The town was incorporated in 1926 as part of Chesterfield County, became an independent city in 1948 under Virginia's unusual constitutional system, and grew rapidly through the 1960s. In the 1980s, the Temple Avenue Connector and a new bridge across the Appomattox opened the southeastern edge of town to commercial development, and Southpark Mall rose there in 1989. The shopping center boomed at the same time Petersburg's downtown retail collapsed, and the racial geography of the two cities, white-majority Colonial Heights and Black-majority Petersburg, mapped onto that economic divide. A five-year civil rights boycott of Southpark Mall, led by Curtis Harris and the SCLC, ended only after the mayor of Colonial Heights met with the organizers and agreed to expand Black employment at the mall.

On August 6, 1993, a tornado that had peaked at F4 intensity in Petersburg crossed the Appomattox River and cut across Colonial Heights at F3 strength, with winds up to 175 mph. It collapsed the roof of a Walmart on Temple Avenue, killing three people inside. A fourth died in nearby Prince George. Two hundred and forty-six were injured. Damage at Southpark Mall and across the Tri-Cities ran to fifty million dollars. Fifteen years later, an EF1 tornado tore through almost the same ground and flipped cars in a strip mall less than a tenth of a mile from where the old Walmart had stood. The bluff that has drawn artillery for centuries also draws weather, and the people who live here have learned to read the southern sky.