Fantome was gazetted as an Aboriginal reserve in 1925, and from 1928 it held a lock hospital, an institution for the forced treatment of Aboriginal people with venereal disease. The word treatment understates the reality. These were people removed under the Queensland protection regime, sent across great distances, and confined on this island until they were judged clear. The hospital operated until 1945. By the late 1930s Fantome had also become a clearing station, where Aboriginal people bound for Palm Island were examined first, their bodies inspected before they were allowed to land. The administration prized secrecy above almost everything. Records were thin, names went unrecorded, and the policy was managed so quietly that even the spread of disease was hard for the authorities to track. That silence makes the human scale of what happened here difficult to recover, and all the more important to try to honour.

In 1939 a lazaret, a leprosarium, opened at the island's northern end, and the following year Indigenous patients held at Peel Island near Brisbane were transferred to Fantome under police escort, in conditions of deliberate secrecy. From then until 1973 the lazaret isolated Aboriginal, Torres Strait Islander and South Sea Islander people diagnosed with leprosy, far from family and home. From 1940 Catholic nuns of the Order of Our Lady Help of Christians provided daily care; from December 1944 the Franciscan Missionaries of Mary took over that role. The policy they served was harsher than anything they could soften: a system of racial segregation that treated isolation as the natural response to Aboriginal illness, while non-Indigenous patients elsewhere were treated under very different conditions.

One life lends the island a face. In 1945, a seven-year-old South Sea Islander boy named Joe Eggmolesse was diagnosed with leprosy and taken from his family under police escort, carried more than a thousand kilometres by rail and sea to Fantome, where he remained for ten years. He was a child, and the island was his childhood. Decades later, a feature documentary, Fantome Island, made by Sean Gilligan and released in 2011, followed Eggmolesse back as a seventy-three-year-old for a Remembrance Day on the island that had held him. The film won the State Library of Queensland's John Oxley Memory Award and Best Australian Documentary at the 2012 Human Rights Arts and Film Festival, carrying these stories to people who had never been told them.

When the institutions closed, the island was abandoned and its buildings eventually burned, but the cemeteries remained. The lazaret stood on the low coastal ground at the northern end, the lock hospital a few kilometres inland, and around both the dead were buried where they fell. The island is the site of some two hundred graves in all, with around 120 still marked at the lazaret cemetery and others unmarked, the resting places of people who died in isolation and were never carried home. The sites are heritage-listed now, recognised at last for what they hold. But the sea is taking them. As of 2020, tidal erosion threatens the graves, the danger sharpened by the low ground, the loose soil, encroaching vegetation and a warming climate. People from Great Palm still cross to Fantome to fish and to remember. The island asks only that the dead be allowed to keep their names against the water.