George Cabell was one of the most prominent physicians in early-nineteenth-century Virginia. He attended Patrick Henry in his final illness in 1799 — the Revolutionary orator was Cabell's patient — and he kept up a long correspondence with Thomas Jefferson, who was both his friend and his neighbor. (Jefferson's Poplar Forest retreat sits about twelve miles west.) Before he built Point of Honor, Cabell bought 856 acres on the south bank of the James that included three islands in the river: Daniel's Island, Treasure Island, and Woodruff Island. He purchased the land from Lewellen Jones, who had bought it from Christopher Lynch, son of the Revolutionary-era Quaker Charles Lynch and nephew of the John Lynch who founded the city. The land had passed through three generations of Lynchburg's founding family before reaching Cabell.

Construction began in 1806 — the same year Jefferson started Poplar Forest — and finished in 1815. The two-story Federal mansion is built of stuccoed brick, irregular in plan: a three-bay center section flanked by two octagonal-ended projections, one on each side, that give the facade an unusual rhythm. Inside, the doorways and parlor mantels carry the carved Federal detailing that mid-Atlantic builders had inherited from English Adamesque pattern books. The house bears a strong resemblance to the Hancock-Wirt-Caskie House in Richmond, suggesting that Cabell was drawing from the same regional pool of design ideas. A porch may have been added around 1830 and removed in the 1977 restoration. Wallpaper and additional stucco were also added in the 1830s renovation.

Cabell died in 1823 after falling from a horse. The house passed through his widow and then his son's family, and in 1828 the Langhornes — a Lynchburg industrial family who owned the Langhorne Mills — bought it. Members of the Langhorne family owned Point of Honor at two different points in the nineteenth century. John S. Langhorne lived here before the Civil War, and his children and grandchildren would become some of the most famous Americans nobody can quite place. His eldest son Chiswell Langhorne grew up wealthy enough to send his daughters into a kind of late-Victorian celebrity. One of those daughters was Mrs. Charles Dana Gibson — the model whose profile her husband drew over and over until she became the Gibson Girl, the defining image of American femininity in the 1890s. Another was Nancy Langhorne, who married Waldorf Astor and in 1919 became the first woman ever seated in the British House of Commons. Their cousin Elizabeth Langhorne Lewis was a prominent Virginia suffragette.

By 1878 the house belonged to L. E. Lichford, a grocery wholesaler. Three generations of the Lichfords lived here while the surrounding neighborhood — Daniel's Hill — filled in around them with Victorian and Colonial Revival houses. In 1928 James R. Gilliam Jr. bought Point of Honor and gave it to the City of Lynchburg, which used part of the property as a recreation center until the Lynchburg Historical Foundation received the deed in 1968. A careful restoration finished in 1977 returned the house to something close to its 1815 appearance, and in 1978 it opened as a museum. National Register listing came in 1970. Visitors today walk the same rooms where Patrick Henry's physician once entertained his neighbors and where a Virginia-born girl named Nancy played as a child before she became Lady Astor.