The Windermere mansion at Ho Sai Wing's new estate was not a single building but a compound. There was the main mansion itself, colonnaded and substantial. Across the plot, a detached single-storey building stood separately. In front, a sports ground. Each summer, Ho Sai Wing and his family would come from the city to spend the season there. Robert Hotung's wife, Margaret Mak — also known as Maclean, her name rendered in Chinese as 麥秀英 — visited three times between 1923 and 1944. Robert Hotung himself, despite his son's evident attachment to the place, never came. The mansion eventually prompted others to settle nearby, and a village called Ho Tung Lau Village grew up around the estate — borrowing the 'Ho Tung' name from the family, a small act of place-making that would outlast the family's occupation of the land.

In 1949, the political landscape of Asia shifted violently. The People's Republic of China came to power on the mainland, and the British administration in Hong Kong braced for the implications. The Royal Air Force established a permanent airstrip and airbase at Sha Tin Hoi, named Sha Tin Airfield, and the government requisitioned the Ho Tung Lau estate. Wah Kiu College, which had been using the campus, was forced to leave; its last academic year at Ho Tung Lau ended in the summer of 1949. The mansion and its grounds became Arcullis Camp, a military installation that operated until 1962. The colonnaded holiday house that Ho Sai Wing had imagined as a retreat became, for thirteen years, an army facility on the edge of a Cold War border.

After the military's departure, life around Ho Tung Lau reorganised itself around transport. In 1955, a ferry service began connecting Ma On Shan — then still a small settlement around an iron ore mining operation — to Ho Tung Lau, running from 6 am to 6:30 pm at half-hour intervals, with a fare of 50 cents per crossing. The ferry was intended to link the villages on both shores and make Ma On Shan accessible for workers and visitors. According to maps from 1970, the ferry pier was not located at the original Windermere mansion site but northeast of it, at what had once been the officer's mess of Sha Tin Airfield. This geographical drift — the ferry attaching its name to the northeastern building rather than the southern mansion — gradually shifted how people used the name 'Ho Tung Lau,' associating it with the pier and its surroundings rather than with the estate. A bus service also came during this period, serving Ho Tung Lau Village. In the late 1960s, St. Andrew College briefly used the old mansion as a campus; when they left, the building was gradually abandoned.

The physical landscape of Ho Tung Lau today would be unrecognisable to Ho Sai Wing. Where the Windermere mansion and its sports ground stood, or approximately where they stood, three private housing estates have been developed: Royal Ascot, The Palazzo, and Jubilee Garden. The old shore of Sha Tin Hoi — Tide Cove — where Ho Sai Wing purchased his plot of reclaimed land in 1923 is now thoroughly landlocked, swallowed by successive rounds of reclamation that turned the inlet into the flat terrain of modern Sha Tin. Fo Tan MTR station stands next to the Ho Tung Lau Maintenance Centre, which is where the name 'Ho Tung Lau' now most commonly appears on maps and transit signage. A holiday estate that became a village that became a military camp that became a ferry terminal that became a railway maintenance depot: it is a compressed history of how Hong Kong's New Territories absorbed the twentieth century.