The hill's colonial history began not with a residence but with a medical experiment. In December 1859, Governor Hercules Robinson ordered a path cut from what is now Robinson Road to the summit of Victoria Peak. By the spring of 1862, a military sanatorium had opened on the plateau below the flagstaff, constructed solidly and sited for the fresher air of higher altitude. Seventeen patients were sent there. None of them improved — in a year when Hong Kong's overall health was poor, the Peak offered no particular advantage. The military abandoned the site. It sat unused, occasionally visited by picnickers, until 1867, when Granville and Matilda Sharp (the couple after whom Matilda Hospital is later named) took a lease on the empty buildings. They had long argued that the Peak's elevation made it a healthier alternative to the lower districts. They were right about that, even if the sanatorium had failed to prove it.

In 1867, Governor MacDonnell purchased the site from the War Department and converted the former sanatorium into the first Mountain Lodge — a bungalow complex with three structures. The main lodge faced toward Pok Fu Lam on one side and a lawn on the other; two smaller buildings flanked the lawn with their backs to the hillside, styled after European cabins. MacDonnell had taken temperature measurements and found that the Peak ran 14 degrees Fahrenheit cooler than Central District in summer. That difference mattered enormously before air conditioning existed. The lodge was severely damaged by a typhoon the following year. Governor Kennedy ordered refurbishment and expansion in 1873; another typhoon destroyed that work in 1874. The pattern — build, storm, rebuild — would define the property for two decades. A third major typhoon struck in 1892, after which Governor Robinson commissioned the Director of Public Works, Francis Cooper, to undertake a proper restoration. Cooper's assessment concluded that the lodge was too deteriorated to save. It was demolished in 1898.

A second Mountain Lodge — a two-storey Renaissance Revival building — replaced the original and remained standing until 1946. The cast of people who passed through it included most of Hong Kong's governors of the late colonial period. Sir Cecil Clementi had the building refurnished in 1925 and added a small safe. Thomas Southorn, who served as Colonial Secretary, lived there with his wife at various points between 1925 and 1936. By 1932, voices within the government were calling for the Lodge to be abandoned; a house in Fanling was proposed as the governor's alternative retreat. By 1938, a plan existed to abandon both Government House and Mountain Lodge in favour of a single new residence at Magazine Gap. Then the Second World War arrived, and the plan was set aside.

Mountain Lodge was demolished in 1946, a year after the war's end. The Gate Lodge — the small building that once housed the keeper of the property — survived and remains a declared monument today. In 1969, the Urban Council transformed the site into Victoria Peak Garden, a public park built on the lodge's foundations. The pavilion at the centre of the garden sits directly on the masonry platform that once supported the second Mountain Lodge. Three marking stones inscribed "GOVERNORS RESIDENCE" were discovered in 1978; one stands at the northeast corner of the former grounds, another was placed in a flowerbed near Government House in Mid-Levels in 1980. During pre-construction checks for a Peak improvement project in January 2007, archaeologists found original wall fragments, roof tiles, and granite steps from the second lodge still embedded in the ground. The summit has changed entirely, but its layers have not entirely disappeared.