The prison rose from granite quarried in the colony's earliest years, its thick walls and arched windows reflecting the Victorian conviction that authority should be visible and formidable. Completion in 1842 made it contemporaneous with the founding of Hong Kong itself — less a later addition to the colony than part of its original skeleton. Through the following decades it expanded steadily. The D Hall complex, built in 1895 in Neo-Georgian style, gave the compound its imposing central character. B Hall, C Hall, and E Hall followed in 1914, and by that point the prison occupied a substantial block bordered by Hollywood Road to the north, Old Bailey Street to the west, Arbuthnot Road to the east, and Chancery Lane to the south. Nine buildings in total — three stories, two stories, one — housing cells, dormitories, a hospital, kitchens, exercise yards, a laundry. It was a complete world, sealed off from the busy streets just outside its gates.

In 1931, a Vietnamese nationalist named Nguyen Tat Thanh — later known to the world as Ho Chi Minh — was arrested in Hong Kong at the request of French colonial authorities and imprisoned here. He remained until 1933. At the time he was a Comintern agent, still years away from leading a country or a revolution. What he made of those two years between granite walls in a British colony is not well documented, but the fact itself is striking: the man who would go on to shape the history of Southeast Asia spent formative time in Hong Kong's oldest jail. It is the kind of historical coincidence that a place like Victoria Prison accumulates without particularly trying.

The Japanese occupation during World War II left the prison damaged, its buildings battered by bombing. It reopened in 1946 after restoration and resumed its original function, but Hong Kong itself was changing in ways that transformed what the prison was used for. When Hong Kong became a declared port of first asylum for Vietnamese refugees in the late 1970s, Victoria Prison took on a new role as a transit and repatriation centre — a holding place for people displaced by conflict rather than criminals. This was not the last shift in its purpose, but it may have been the most poignant. The prison was formally declared a monument on 8 September 1995, together with the adjacent former Central Police Station and former Central Magistracy. It was officially decommissioned on 12 March 2006.

When the government began planning the site's cultural redevelopment in the early 2000s, one building became unexpectedly contentious. F Hall — completed in 1913, rebuilt in 1931 as a two-storey structure with a weaving area, severely damaged in World War II, rebuilt again in 1948 as a government printing workshop, later converted to an office — was not on the Antiquities Advisory Board's list of historical buildings. The Central and Western District Council pushed back. F Hall, they argued, represented the entrance to the interior of the compound; without it, the heritage site lost its integrity and the history of prisons in Hong Kong would be diminished. The debate played out through formal motions and government replies across 2004. It was a small argument about a modest building, but it captured something real: when you restore a heritage site, every decision about what to keep and what to tear down is also a decision about which history gets told.

After an eight-year restoration that cost approximately US$485 million, Victoria Prison reopened in 2018 as part of the Tai Kwun Centre for Heritage and Arts. The name means "big station" in Cantonese, and the compound now encompasses the former Central Police Station and former Central Magistracy alongside the prison — three declared monuments united under a single cultural identity. The cells are open to visitors. The exercise yards host performances. Galleries occupy what were once administrative blocks. Local artists displayed work in the cells as early as the 2007 open days, just a year after closure — a preview, perhaps, of what the space could become. The granite walls that once sealed people in now draw people from across the city in. The function has reversed; the weight of the place has not.