The building predates Ho Tung's ownership by decades. It was constructed before 1894 — the exact date is uncertain — and originally owned by a woman named Dona Carolina Cunha, whose history is now largely lost to the record. What survives is the structure itself: a two-story colonial mansion in St. Augustine's Square, in the São Lourenço neighborhood, its architecture a product of the Portuguese colonial style that characterized Macau's more affluent residential districts. Ho Tung purchased it in 1918, drawn by the combination of a pleasant square and its proximity to Macau's established social and commercial circles. He used it as a retreat from the pressures of Hong Kong commercial life — a quieter house on an unhurried colonial square, about an hour by ferry from his main residence. That arrangement shifted decisively when the war arrived.

Portuguese neutrality was a peculiar condition in wartime Asia. Macau continued operating as a semi-normal city while Hong Kong, Shanghai, and most of coastal China fell under Japanese occupation. The territory became crowded with refugees: Chinese families, Europeans, and prominent figures from Hong Kong who calculated, correctly, that the Japanese would not challenge Portuguese sovereignty over such a small and strategically marginal territory. Ho Tung arrived and stayed. He was not idle during those years — by several accounts he remained engaged in business affairs and maintained his extensive correspondence — but the war reduced his circumstances considerably. The Macau mansion, once a weekend retreat, became his primary home for four years. That period gave the building a significance beyond mere architecture: it is a place where a remarkable life was sheltered from history's worst convulsions.

Ho Tung died in 1956 at the age of 93, one of the wealthiest and most socially prominent figures Hong Kong had produced. His will specified that the Macau mansion should become a public library. The government accepted the bequest and the Sir Robert Ho Tung Library opened to the public in 1958. It became part of the Macao Public Library system, with the old building housing the head office of the library network on the ground floor and the Macao ISBN Agency on the second. In 2005, a modern addition was constructed in the back garden, expanding the collection and making the library the largest public library in Macau. The juxtaposition is visible from the street: pre-1894 colonial villa in front, contemporary glass-and-concrete extension behind. Both serve readers. Neither would exist in this form without the singular life that connected them.

St. Augustine's Square, where the library stands, is one of the quieter corners of Macau's Historic Centre. The tourists who crowd Senado Square and the Ruins of St. Paul's tend to thin out by the time they reach this neighborhood. The library occupies its site with the undemonstrative authority of a building that has outlasted multiple purposes and still found new ones. Its garden — now partly given over to the modern extension — has the quality of a place where time moves differently than it does on the casino strip a kilometer away. That is perhaps fitting for a library, and especially fitting for one that carries the name of a man who used it as a refuge. The books that now fill the rooms of his old retreat offer a different kind of shelter than the one he needed in 1941 — but shelter nonetheless.