Homes have stood on Libby Hill since at least 1796. The prominent early residents included Governor George W. Smith - who died in the 1811 Richmond theater fire - and Colonel George Mayo Carrington. Neither of their houses survives. The oldest surviving building is the Gentry-Strokes-Crew house at 2718 North 27th Street, built in 1839 by John Gentry and altered many times since. The Saunders House nearby went up the same year, contemporaneous with the Libby House at the corner of Main and 29th that gave the neighborhood its name. The house at 11 1/2 North 29th Street has what Richmond's preeminent twentieth-century architectural historian, Mary Wingfield Scott, called 'one of the most beautiful iron verandahs in Richmond' - the kind of detail that makes a neighborhood matter to the people who study it. From the hill you can see other landmarks: the Lucky Strike tobacco building (now lofts), the Armitage Manufacturing Company building, Rockett's Landing where the steamships used to dock, and Manchester across the river.

Because Libby Hill commanded the view westward toward the river and the city, and because it sat above the ground where the Confederate Navy Yard had operated during the war, Richmond's monument-builders saw it as a place to remember the Confederacy. A memorial to Robert E. Lee was originally planned for Libby Hill before it was relocated to Monument Avenue across town. With Lee's statue gone elsewhere, the Confederate Soldiers' and Sailors' Monument Association formed in 1889 to put a memorial to all Confederate Soldiers and Sailors on the hill. The chair was David C. Richardson, a future Richmond mayor. They chose a design based on Pompey's Pillar in Alexandria, Egypt - a 13-block granite column, one block per Confederate state, with a bronze figure of a private standing on top. It was unveiled on May 30, 1894, at a total cost of over $30,000, with future Richmond mayor Carlton McCarthy as the orator. The monument stood for 126 years.

The monument came down on July 8, 2020, after a summer of public demand following the murder of George Floyd. The city removed it. What remains in Libby Hill Park is a small landscape of fountains, paths, a park house, the plaque describing the view that named Richmond - and other monuments to ordinary public servants. There is a bench for Sergeant John Henry Taylor, who served in the Richmond Police Department for over 25 years before retiring in 2009; it was placed by the Church Hill Crime Watch to honor what they called his 'above and beyond care and concern for the safety of every person living on Church Hill and the surrounding areas.' There is a memorial to Officer Thomas 'Mongo' McMahon, who served for over 20 years and was shot and killed in the line of duty after a car chase. Two officers who walked these streets. The park kept them while the larger monument went away.

In September 2015 Richmond hosted the UCI Road World Championships, the most prestigious annual event in road cycling. The course wound through the city, and one of its signature features was the steep, winding cobblestone path that climbs through Libby Hill Park. Watch footage of the race and you'll see the leaders rising out of the saddle on that climb, pedaling up past the spot where Byrd allegedly stood, past the ghosts of the Confederate Navy Yard, past the plaques to two policemen, past where a column with a bronze private had recently been, with the river view that named the city falling away behind them. The neighborhood that grew up around this single piece of geography is small enough to walk through in an hour. The view that named the city is still here, looking out over a Richmond that keeps changing while standing on the same bend in the same river.