The island was ceded to Britain in 1860 alongside Kowloon, as part of the Convention of Peking. The British came first as quarrymen — the island's granite was useful, and the English name reflects that early industrial purpose. By the 1850s it had already acquired a prison, a bleak use of its isolation. The Qing dynasty name, Ngong Shuen Chau, had different resonances entirely, but the British designation stuck and the quarrying defined the island's early colonial identity. Isolation, as with so many islands in colonial settings, made it convenient for purposes the mainland preferred to keep at a distance — punishment, storage, interception. A Royal Navy Radio Interception and Direction-finding Station was established there in 1935, and from 1935 to 1939 the base served as the main radio interception unit for the Far East Combined Bureau, operating across Victoria Harbour from the naval dockyard.

The Japanese Imperial Army captured Stonecutters Island on 11 December 1941, following heavy shelling — three days after the attack on Pearl Harbor and the beginning of the Battle of Hong Kong. Merchant ships in the island's docks were scuttled, demolitions were carried out at Kowloon Naval Yard, and the island's radio installations were repurposed for Japanese military communications and for extending the broadcast range of NHK's Overseas Bureau. The occupation brought one of the stranger details of the island's wartime history: the Japanese established a snake farm there, using the island's isolation to keep the venture away from the city. The snakes — banded kraits, cobras, and others — were milked for venom, which was then processed into antidotes for soldiers bitten on active duty elsewhere in the Pacific theatre. The island's natural fauna had already included significant snake populations; the farm was, in a grim way, making use of an existing resource.

After the war, Stonecutters Island became home to various British Army units, an explosives sub-depot, and a community of personnel who commuted to the mainland by Royal Navy ferry — the T-Boats, which ran between the island, HMS Tamar in Hong Kong, and the Star Ferry terminal in Kowloon. The island also had a NAAFI shop, a restaurant, a swimming pool, and chalet-style bungalows for rest and recuperation. What makes the postwar period most vivid, though, is a detail about the island's policing. The Army Department Police — the ADP — were Sikh officers, chosen in part because their religious observance meant they neither smoked nor drank alcohol. They were a consistent presence throughout the British era, and it was, as one account notes, common to see their blue pagris — turbans — drying in the sun outside their barracks. In the early 1980s, two ADP officers were Indian national hockey players, and the officers were often seen playing barefoot on the field.

In the 1970s and 1980s, the island was also a forward operating base for a Royal Navy Hovercraft unit — Naval Party 1009 — which operated two SRN6 Mk6 hovercraft to assist the Hong Kong government with anti-illegal immigration operations along the coast. The unit was disbanded in 1985. A Jardine Matheson and DuPont explosives factory operated on the island during this period as well, manufacturing water gel and other commercial blasting explosives for the construction industry. Then came the 1990s and the West Kowloon Reclamation: the island was connected to the Kowloon peninsula to free up land for the road and railway network serving the new airport at Chek Lap Kok. Sovereignty transferred to the People's Republic of China in 1997, and the naval base is now operated by the People's Liberation Army Navy. In 2001, a large sewage treatment facility opened on the island; it has since reduced E. coli levels in nearby waters by 99 percent, allowing coral to return to Victoria Harbour. The island that was absorbed into the city ended up cleaning the water around it.