The neighborhood takes its name from Columbian College, a non-sectarian college founded in 1821 by Baptist minister Luther Rice on what is now the southern edge of Columbia Heights. The college eventually became George Washington University and moved to Foggy Bottom in 1912, but the name stuck. The shape of the modern neighborhood was set by Senator John Sherman, the Ohio Republican whose name attached to the Sherman Antitrust Act, who bought up the high ground north of Boundary Street between 1881 and 1882 and laid it out as a streetcar subdivision. Sherman Avenue, on the eastern edge of Columbia Heights, carries his name. The southern part of the neighborhood, which had been a working pasture, was locally called Cowtown well into the 1890s. The northern part had been part of the village of Mount Pleasant. The L'Enfant Plan never reached up this far. The streets here are part of a later, looser grid.

By 1914 four streetcar lines served Columbia Heights, putting downtown twenty minutes away. The neighborhood became a destination for upper-level federal workers, Supreme Court justices, and high-ranking military officers. Authors Jean Toomer, Ambrose Bierce, and Sinclair Lewis lived here. Chief Justice Melville Fuller and Justice John Marshall Harlan had homes on Clifton Street. The Belmont mansion, between Florida and Clifton, marked the southern entrance. The 1924 Tivoli Theatre, a Renaissance Revival movie palace by Thomas Lamb, anchored 14th Street. Apartment buildings rose alongside the rowhouses: the Olympia, Clifton Terrace, Meridian Manor, the Cavalier. The neighborhood was densifying. The same building boom was happening in dozens of streetcar suburbs around American cities. Columbia Heights had it earlier and harder than most.

In 1949 the District redrew the segregation map. Central High School, on the southern edge of Columbia Heights, did not have enough white students to fill its classrooms; the school was renamed Cardozo and designated as a colored high school. The decision accelerated demographic change. Black families who had been concentrated nearer downtown began buying apartment buildings, then individual homes, in Columbia Heights. By the mid-1950s the neighborhood was a strong Black middle-class enclave, with Howard University immediately east and the U Street corridor immediately south. Marvin Gaye attended Cardozo. The neighborhood produced soldiers, civil servants, teachers, musicians, ministers. For about twenty years, between the end of segregation in housing and the events of April 1968, Columbia Heights and the adjoining U Street were one of the most important centers of Black middle-class life in America.

Word of Martin Luther King's assassination in Memphis reached Washington on the evening of April 4. By the next morning the 14th Street corridor was on fire. The riots that followed devastated the commercial spine of Columbia Heights and U Street. More than nine hundred businesses were damaged or destroyed across the city. Federal troops occupied the neighborhoods for nearly a week. The economic damage outlasted the smoke by decades. Middle-class residents who could leave for the Maryland and Virginia suburbs did so. Storefronts on 14th Street that had been thriving in 1967 were vacant for thirty years. The Tivoli Theatre closed in 1976. Latino immigrants arriving from Central America in the 1980s and 1990s found cheap rents in the apartment buildings and the rowhouses, and a new neighborhood began to form atop the ashes of the old one. The Salvadoran community that emerged here is still one of the largest in the metropolitan area.

In planning the Green Line in the 1970s, Metro deliberately routed its tracks through the riot-damaged corridors as a deliberate act of redevelopment policy. The Columbia Heights station opened in September 1999. The DC USA mall opened on March 5, 2008, with 546,000 square feet of retail and a thousand underground parking spaces. The Tivoli Theatre reopened in 2005 as the home of the GALA Hispanic Theatre, a Spanish-language company that has performed in the District since the 1970s. Columbia Heights was named one of the fastest gentrifying neighborhoods in the United States in 2012. The 2010 census found a population that was 43.5 percent African American, 28.1 percent Hispanic, 22.9 percent white. As of 2018, about 22 percent of the housing stock remained reserved for low-income renters. The streetscape today carries five layers of history at once, and the people walking it know which layers belong to them.