The building was designed in the Greek Revival style — a fashion that was reaching the end of its Western popularity by the 1840s but still carried associations of civic seriousness and permanence that colonial administrators valued. Historians have not definitively established who drew the plans: the candidates are Murdoch Bruce, a Scottish inspector of buildings, and Lieutenant Bernard Collinson of the Royal Engineers, both of whom were active in early Hong Kong. The attribution remains unresolved. What is certain is the result: a single-storey building with a broad verandah and colonnaded frontage suited to the tropical climate, formal enough for official entertaining and domestic enough for a senior officer's residence. It was known as Headquarter House until 1932, when it was renamed Flagstaff House.

For more than a century, Flagstaff House served as the private residence of whoever held the position of Commander of British Forces in Hong Kong — a posting that carried both military weight and social prestige in colonial society. The house occupied land associated with Victoria Barracks, and its elevated position above Queen's Road announced the primacy of the garrison in the colony's early decades. Officers and their families came and went; the building absorbed the routines of military domesticity, formal dinners, and changing political circumstances. When the British garrison's role in Hong Kong diminished in the post-war decades, the house's purpose as a residence became increasingly ceremonial, and the question of what to do with it eventually became urgent.

The decision to convert Flagstaff House into a museum rather than demolish or repurpose it as offices was made carefully. The building was restored as far as possible to its original mid-19th-century appearance, structurally reinforced, and its interior modified for public use. The Flagstaff House Museum of Tea Ware opened in 1984. Its core collection was built around a donation made in 1981 by Dr. Kwee Seong Lo, a local collector and the founder of Vitasoy, whose gift of approximately 600 ceramic and purple clay tea vessels now forms the heart of the museum's holdings. The new gallery was arranged to display ceramics and Chinese seals alongside the tea ware, giving the collection a breadth that the building's modest scale handles quietly and well.

There is a particular historical irony in a tea museum occupying a building that once housed the military commanders of a colony whose founding was inseparable from the tea trade. The First Opium War, which led to Britain's acquisition of Hong Kong in 1842, was driven in significant part by the trade imbalance that British tea purchases from China had created — silver flowing out of Britain, opium flowing in the other direction as a financial corrective. Flagstaff House was built four years after that settlement. The museum does not labour this point; its galleries focus on the aesthetics and craft of tea ware rather than its political history. But the building itself, standing in a park on land that was once a military garrison, carries that history in its bones.