Charles St George Cleverly, Hong Kong's Surveyor General, designed the building and oversaw its construction starting in 1851. It took four years to complete. The first governor to reside there was Sir John Bowring, the fourth governor of the territory. Over the following 142 years, 25 of the 28 governors who served Hong Kong would use Government House as their official home. The 24,000-square-metre plot sits between Upper Albert Road and Lower Albert Road in Mid-Levels, its main entrance facing south toward Victoria Peak. The Gate Lodges at the Upper Albert Road entrance, also built in 1855, are the oldest surviving structures on the grounds. For decades those lodges housed the Government House Guards — various British Army units, including the 1st Battalion of the Queen's Royal Surrey Regiment in 1962–63, who stood watch over the governor.

In 1941, Japanese forces occupied Hong Kong. Government House became the residence of the Japanese Military Governor. Three years into the occupation, in 1944, architect Seichi Fujimura remodeled the building — adding a tower and changing the roofline to give the structure a hybrid Japanese-neoclassical appearance, a visual signal of the new order. After the surrender, British authorities removed many of the Japanese fittings, but the tower was left standing. Whether this was expedience, an oversight, or a deliberate choice to preserve the historical record, the result is a building that carries its contradictions on its exterior. The exterior plasterwork is Art Deco; the roofline is something else entirely. Government House was declared a monument in 1995 under the Antiquities and Monuments Ordinance.

The grounds offer a more peaceful layer of the building's history. Azaleas were first planted in 1919, and they bloom every spring in multiple colors — the garden opens twice a year to the public, with at least one opening timed to the spring bloom. A rare Brownea grandiceps, known as the Rose of Venezuela, grows on the grounds: seven metres tall, with a crown spread of nine metres, it was brought from South America and is listed in the Register of Old and Valuable Trees of the Leisure and Cultural Services Department. A kidney-shaped swimming pool, built between 1971 and 1982, sits in the garden. A fish pond was added later. Each successive resident has left small marks. When Donald Tsang, the second Chief Executive, moved in after extensive renovations in 2006, HK$300,000 was specifically allocated to a new pond for his koi collection.

The transfer of sovereignty in 1997 brought the last governor, Chris Patten, out of Government House for the final time. Tung Chee Hwa, the first Chief Executive under the new order, chose not to live there — relocating instead to Grenville House. Rumors at the time mentioned warnings about surveillance devices installed throughout the building during the colonial era. Whether or not that story is true, the symbolism was clear: Tung was signaling a deliberate distance from British legacy. Donald Tsang eventually moved in, in January 2006. Today Government House continues its official function — banquets, honours ceremonies, formal receptions. The Hong Kong Police, inheriting the job from the British Army, now guard the gate.