Lyemun Barracks took its name from the fishing village of Lei Yue Mun below. The British military decided in 1885 to build a permanent installation here, and in 1889 transferred the land to the War Department. The position was strategically clear: the site commands the eastern approach to Victoria Harbour, the narrow passage through which any fleet approaching from the east would have to pass. The barracks complex divided into three portions — the central area of main barracks (now the holiday village), a western ridge fort, and a headland fortification that became the Hong Kong Museum of Coastal Defence. By the 1930s, however, advances in aviation and long-range artillery had reduced the site's strategic value. The heavy guns guarding the strait were less relevant than they had once been.

The Japanese assault on Hong Kong began on 8 December 1941, the day after Pearl Harbor. By 19 December, the battle had reached Lyemun. At 7:45 PM, a detachment of Japanese soldiers, accompanied by local collaborators in a lorry, broke through the perimeter fence at the Sai Wan Antiaircraft Battery with bombs and killed the British sentries. About 29 British soldiers were locked inside an ammunition magazine. Around midnight, the Japanese ordered them out and bayoneted them. Two Chinese-British soldiers survived by hiding under the bodies of their fallen comrades. They lay still for four days before escaping. A counterattack by C Company of the Royal Rifles of Canada failed to retake the position — the Japanese held the high ground and were well dug in. The barracks fell that night and did not return to British hands during the occupation.

After the war, the barracks housed the Depot and Record Office of the Hong Kong Military Service Corps from 1948 to 1986. When the British military's presence in Hong Kong contracted, the buildings were handed over to the government in 1987. The southern portion of the complex was converted into the Lei Yue Mun Park and Holiday Village — the only government-run holiday camp in Hong Kong's urban districts. It accommodates up to 282 people in four family hostels and two group hostels, with balconies, bedrooms, and bathrooms. The buildings, constructed in European colonial style with white walls and light blue window frames, were adapted for their new purposes: the old Soldiers' Quarters became the main indoor recreational centre, the Officers' Mess became a group hostel, and the Sergeants' Mess became an arts and crafts centre.

The married quarters built for soldiers and their families were given literary names that have stayed with the buildings. Block 30, built in 1936, is the Masefield Block, named after the English poet and writer John Masefield. Block 31, built in 1907, bears the name of William Shakespeare. Block 32, completed in 1909, is Tennyson. Block 33 is Milton. Block 34, also from 1936, is Wordsworth. Those names — inscribed on the exterior walls — are perhaps the most quietly poignant detail of the whole complex: English Romantic poetry etched into a military barracks on the edge of Victoria Harbour, surviving two world wars and a colonial handover to end up on the walls of a government holiday camp. The whole compound was graded as a Grade I historic structure in December 2009.

The holiday village has found itself pressed into service during Hong Kong's public health emergencies. During the 2003 SARS outbreak, when the Amoy Gardens housing estate became a major cluster and over 200 residents were infected, the government ordered the unaffected residents of Block E into isolation at Lei Yue Mun Park and the Lady MacLehose Holiday Village in Sai Kung. Residents returned home after Hong Kong was removed from the WHO's list of affected areas in late June 2003. In January 2020, the site again served as a quarantine location when the first COVID-19 cases emerged in the territory. The barracks that once guarded the harbour entrance had become, in the twenty-first century, a place of refuge for a different kind of threat entirely.