The barracks occupied a substantial tract of land bounded by Cotton Tree Drive, Kennedy Road, and Queensway. Together with Murray Barracks, Wellington Barracks, and Admiralty Dock, they formed a connected British military zone at the heart of Central — land set aside for empire at the moment of colony's founding. The barracks were named for Queen Victoria, the monarch at the time of construction, which was the era's habit. They housed various units over the decades, including, around 1868, Indian troops of the garrison in a set of blocks built specifically for them. Blocks A and B later served as headquarters buildings; a bamboo bridge once connected them. Block D became the Headquarters of the Brigade of Gurkhas in the 1970s.

In December 1941, the Japanese invasion of Hong Kong ended British control of the colony. The Victoria Barracks, like the rest of Hong Kong, fell under Japanese occupation and were used by Japanese forces until the end of World War II in 1945. The Japanese left behind structures that persisted on the site well into the late 1970s. After the war, the barracks underwent major restoration under British administration. Part of the land was returned to the Hong Kong Government in 1967; most of the rest followed in 1979. In that same year, people seeking to regularize their status — immigrants applying for Hong Kong identity cards — lined up at the Victoria Barracks office for registration. The military compound, for a moment, became a place of bureaucratic hope.

In March 1977, the Governor of Hong Kong appointed the Victoria Barracks Planning Committee to advise on the site's future. By September that year, the committee had published its report. The decisions that followed reshaped the Admiralty landscape permanently. The northern section of the barracks was converted into Hong Kong Park — opened in 1991, now home to the Edward Youde Aviary, the Flagstaff House Museum of Tea Ware, and the Cotton Tree Drive Marriage Registry, housed in Rawlinson House, a surviving barracks building. The southern part of the former barracks is now Pacific Place, one of Hong Kong's largest mixed-use developments, along with the High Court and the Queensway Government Offices.

Not everything was demolished. Several structures within the former barracks were assessed and graded as Grade I historic buildings. The Former Explosives Magazine complex — two magazines and a laboratory building separated by earth mounds called traverses, built as buffers against accidental detonation — still stands and is now part of the Asia Society Hong Kong Centre. Cassels Block, dating from the early twentieth century and originally housing married British officers, survived and remains. Freyberg Block, built in 1931 as quarters for the Head of the Intelligence Services, did not. The blocks built for Indian troops, which had later housed garrison headquarters, were also lost. On the lawn outside Flagstaff House, a stone from the Royal Navy Telegraph that once stood within the barracks grounds sits quietly. In Hong Kong Park, a statue of an unnamed World War I British soldier still marks the site's military past, accompanied by a memorial plaque to the defenders of Hong Kong in December 1941, including CSM John Robert Osborn, who was awarded the Victoria Cross during the Battle of Hong Kong.

Hong Kong Park does not announce its history. There are no plaques at every turn. The aviary is charming, the greenhouse elegant, the paths pleasant. But the ground beneath it all — beneath the fountains and the tai chi courts and the children playing — held barracks for 130 years. Japanese structures occupied it. Men from the Punjab and Gurkha highlands served here. Intelligence officers worked in buildings that are now gone. What remains is a park and a mall and a courthouse, which is to say: the ordinary present, built on an extraordinary past. The Victoria Barracks are the kind of place that rewards knowing. The knowing doesn't change the park. But it changes you as you walk through it.