The hill's history as a named place began in 1873, when the Royal Navy purchased the Seaman's Hospital in Wan Chai, using money obtained from selling the hospital ship HMS Melville. The renamed Royal Naval Hospital stood on the hill, which was promptly named for the China Station's Commander-in-Chief, Vice-Admiral Charles Shadwell — becoming Mount Shadwell. A neighbouring hill got its own designation later, around 1931, when an Infectious Diseases Hospital was built there. This second hill was named Mount Parish, after Commodore John E. Parish, the Naval Officer commanding in Hong Kong from 1873 to 1876. At the foot of the hill, a granite pillar still stands beside Queen's Road East, inscribed with a '7', an anchor, and the year '1905' — a boundary stone of what were once Royal Navy lands.

Before the war began, the Hong Kong Government bored a network of air raid precaution tunnels beneath Mount Parish. The system stretched 1.8 kilometres through the hillside, with 13 portals numbered 71 to 83 opening onto the slopes beside Stubbs Road, Queen's Road East, and Kennedy Road. Two ventilation shafts circulated the air through tunnels divided across three levels, connected by sloping passages. Like 28 other ARP tunnel networks across Hong Kong, the purpose was to give ordinary people somewhere to shelter when the bombing began. When Maltby warned on Christmas Eve that Japanese forces might infiltrate through those same passages, the tunnels that had been built to protect civilians had become a tactical problem. British gunners at one point fired an 18-pounder at the tunnel exits, trying to drive Japanese soldiers out into the open.

Mount Parish fell to Japanese forces on Christmas Day 1941. The buildings on both hills were damaged during the fighting. After the war, the Ruttonjee Sanatorium took over the former Naval Hospital site on Mount Shadwell. On Mount Parish itself, the Infectious Diseases Hospital was demolished in the early 1950s to make room for a new campus of Wah Yan College, Hong Kong. The school opened in 1955, designed by Professor Gordon Brown of the University of Hong Kong, covering 20,000 square metres of hilltop with classrooms, laboratories, a hall, and a chapel. Governor Sir Alexander Grantham performed the official opening on 27 September 1955. By 1990, the name Mount Parish had been dropped entirely from the map — the college had become the hill's identity.

The ARP tunnels did not sit empty after the war. In the 1960s and 1970s, a total of 55 cubic metres of radioactive waste was disposed of inside them. Safety concerns mounted in subsequent decades. A 1991 government report recommended transfer to a dedicated facility, but the process moved slowly. On 19 January 2001, a man was discovered inside the tunnels and examined by health physicists; no radioactivity was found on his body or clothing. The government eventually built a new radioactive waste storage facility on Siu A Chau island, and in 2005 the waste stored under Mount Parish was transferred there. The new facility opened officially on 24 June 2006. The three accessible portals — numbers 72, 80, and 81 — remain open today to government staff. Most of the others were filled in after the war. The hill keeps its secrets, but by now it has given most of them up.