Under the Aboriginals Protection and Restriction of the Sale of Opium Act 1897, the Queensland government held sweeping power over the lives of Aboriginal people, including where they could live and whether they could be detained. From 1928, Fantome Island held a lock hospital for Aboriginal, Torres Strait Islander and South Sea Islander people with sexually transmitted infections. From 1939, it also held a lazaret, a leprosarium, for those diagnosed with Hansen's disease. White patients with the same illnesses were dealt with under ordinary health law and housed elsewhere. The non-European patients were sent here, segregated by race and by disease, and the choice was never theirs to make. The architect of the lazaret plan, the senior health official Raphael Cilento, argued partly on grounds of cost, though he dressed the policy in the language of returning northern patients closer to their own country.

Twenty-two kilometres of sea separated Fantome Island from the mainland near Ingham, and that water did the work of a prison wall. Patients lived in small huts, little more than twelve by fifteen feet, sweltering in the heat and offering scant shelter when the tropical rain came. Complaints about poor food and inferior medical care ran through the decades. From 1940, Catholic nuns, first from the Order of Our Lady Help of Christians and later the Franciscan Missionaries of Mary, lived on the island and nursed the sick, a thread of care running through a system built on confinement. Visits from family were rare and tightly rationed. People grew up here, grew old here, and many simply ended here, far from the communities and relatives the policy claimed to honour.

There is a cemetery on Fantome Island with around 120 marked graves, and the true number of dead is harder to know. In the lock hospital alone, dozens died in single years. Some patients transferred from the earlier Peel Island lazaret were dead within five years of arriving. For the people of nearby Palm Island, this is not distant history but family history; many have relatives buried in that ground. The arrival of sulphone drugs in the 1940s finally began to cure leprosy and slow the dying, but the isolation continued for decades more, long after medicine had made it unnecessary. To read the names is to be reminded that these were not cases or inmates but mothers, fathers and children, taken from home and kept apart until the end.

The lazaret closed on 3 August 1973. Within weeks the Health Department burned the complex, and over the following months most of what remained was destroyed; the great water tanks were dismantled the next year. Today the island is uninhabited, and only ruins and that quiet cemetery survive among the trees. Survivors have carried the story forward. Joe Eggmolesse lived to tell his, and a 2011 documentary, Fantome Island, follows a survivor returning to the island as an elder for an act of remembrance, laying bare the racist policies that put her and so many others there. The film won a Queensland history award and a national human rights award. The Manbarra people, also known as the Wulgurukaba, are the traditional custodians of these islands, and Fantome was returned to the care of the Palm Island community in 1986. The site was added to the Queensland Heritage Register in 2012, not as a monument to those who ran it, but as a place that insists the suffering here be neither hidden nor forgotten.