The site chosen for the camp — by the Japanese authorities, in consultation with Hong Kong's Director of Medical Services and Colonial Secretary — was St. Stephen's College, a secondary school on the peninsula, and the grounds of the adjacent Stanley Prison, which the Japanese reserved for their own prisoners. Prior to the occupation, the school had classrooms, bungalows for teachers, an assembly hall, and science laboratories. More than twenty internees were assigned to each teacher's bungalow, which had been built for a single family. In the science laboratories, internees lived in spaces partitioned with sacking and old blankets. The total internee population reached about 2,800, of whom an estimated 2,325 to 2,514 were British, alongside Americans, Dutch, Canadians, and others. When the first arrivals reached the camp in early January 1942, they found no cooking facilities, no furniture, inadequate toilet facilities without running water, and rooms filling quickly with strangers.

The Japanese had made no plans for the civilian population they now held. The internees were left, in effect, to govern themselves. Committees formed for housing, food, and medical care. Every morning, rice congee was served at eight o'clock; rice with stew followed at eleven and five. The diet was designed to sustain life, not to maintain health. What supplemented it came from friends and relatives in the city who mailed food packages, from Red Cross aid that arrived irregularly, from vegetables grown in gardens scratched out of the prison grounds, and from a camp canteen and, when necessary, the black market. The most common sicknesses were malaria, malnutrition, beriberi, and pellagra — diseases of deficiency, of the body consuming itself when it cannot get what it needs. Soap and disinfectant were in chronic short supply. That no major epidemic broke out was attributed by historian G. B. Endacott to the volunteer auxiliary nurses and the improvised medical system the internees built for themselves. There was a hospital — Tweed Bay, occupying a building that had housed Indian warders before the war. There were also musicals, plays, variety shows, language classes, and lectures on photography and yachting. People found ways to make the time endurable.

Records show 121 internees died in the camp. The majority were over fifty, and most died of illness. There were accidents too: two people died from falls, one child drowned, and on 16 January 1945 an American aircraft accidentally bombed Bungalow 5 during a US Navy raid on Hong Kong harbour, killing 14 internees. Those 14 were buried in Stanley Military Cemetery on the hillside above the beach. Seven internees were executed by the Japanese authorities after a radio set — used to smuggle messages in and out of camp — was discovered. The remaining internees were forced to watch the torture of those arrested. Three Chinese policemen who had brought cigarettes and tobacco to the internees were executed by decapitation. Despite the daunting obstacles — Japanese-occupied territory, a language barrier, the difficulty of finding food — there were three major escape attempts. Two in March 1942 succeeded: one group of eight reached Macau by boat; two others made it through the New Territories into mainland China. A third group of four was caught within a few miles of camp and released back after several weeks' imprisonment.

The Americans left first. On 29 June 1942, 377 Allied internees — primarily Americans, alongside Canadians and others — boarded the Asama Maru after a negotiated exchange of nationals; they reached New York City on 25 August. Their departure had been preceded by improved food and permitted contact with Chinese friends outside the camp — a manipulation designed partly to generate favourable propaganda, which a Japanese news agency then published as American journalists calling the camp "probably the most comfortable in the world." Some 73 Canadian internees were repatriated in September 1943 aboard the Teia Maru, which sailed to Goa for an exchange at sea. The British internees waited until the war ended, their repatriation blocked by a diplomatic impasse over Japanese pearl fishermen held by Australia. When Japanese forces surrendered in August 1945, the survivors walked out after forty-four months. The camp's human cost was eventually partially acknowledged: US law authorised payments of US$60 per adult per month of internment; British internees who had resided in the UK received £48.50 each — a sum that many survivors regarded as inadequate acknowledgment. St. Stephen's Chapel was built on the college grounds in 1950, with a memorial window above its west door. The window was a donation, placed to remember the people who had lived and died within these walls.