The story begins in the 1910s, when roughly a hundred members of a Harlem community center declared their intention to serve. The 15th Regiment was constituted in the New York Army National Guard in 1913, the first unit in New York State organized on the basis of race, composed primarily of African Americans but also including servicemembers from Puerto Rico, the West Indies, Canada, and several countries in Europe and Africa. By 1917, the regiment had reached its full strength of 2,000 men. They trained in New York and South Carolina because they had no armory of their own, drilling in whatever spaces they could find, from rented lofts to public markets. Reorganized as the 369th in 1918, the unit served with distinction in France, spending 191 days in combat, longer than any other American regiment in the war.

The armory rose in two phases. Tachau and Vought, architects of the massive Kingsbridge Armory in the Bronx, won the commission for the drill shed, which was built between 1921 and 1924 in a Romanesque style. The cornerstone was laid in November 1923, and the 50,000-square-foot drill hall, with its three tiers of balconies seating 6,000 to 7,000, was completed the following year at a cost of $800,000. The administration building came later, designed by John S. Van Wart and Sidney Wein in the Art Deco style and constructed between 1930 and 1933 for approximately one million dollars. Its terracotta parapet is embellished with chevron designs and stylized eagles, and inside, the corridors feature terrazzo tile floors and cornices with chevron motifs. A company meeting room includes parquet floors, walnut-wood paneling, and a fireplace with a carved mantel. The contrast between the medieval drill shed and the sleek administration building is deliberate, two eras of military architecture standing side by side.

The armory's life has been defined by reinvention. During the Great Depression in 1934, it served as a temporary homeless shelter alongside Brooklyn's 14th Regiment Armory. Two years later, it exhibited artwork from 40,000 people employed through the Works Progress Administration. It hosted track and field competitions, tennis matches, a 1964 speech by Elijah Muhammad, and a 1978 soccer demonstration by Pele. When New York's homeless population overwhelmed the city's shelters in the 1980s, the state turned the armory over to the city for a 200-bed men's shelter, even as the 369th Sustainment Brigade continued to operate from the same complex. The Police Athletic League ran a community center out of the drill shed starting in 2006, and the Harlem Children's Zone operated programs there as well. In September 2012, a community-painted mural devoted to the regiment's history went up on the walls. A month later, Hurricane Sandy partially flooded the building.

The New York City Landmarks Preservation Commission designated the armory a city landmark in 1985. It was listed on the National Register of Historic Places in 1994. But landmark status did not ensure upkeep. By the 2000s, the building was decrepit, and plans to renovate it into a museum showcasing 369th Regiment memorabilia stalled when a bid was rejected in 2005. The armory closed to civilians for renovations in 2014, only to be sidelined further by a lead cleanup project in the basement. Despite fears that children using the community programs might have been exposed to lead contamination for over a decade, state officials maintained the contamination was confined to areas that had been off-limits to children. The building's story mirrors the broader tension in historic preservation: the gap between recognizing a building's significance and actually committing the resources to maintain it.