The island had a longer history than its grim title suggests. Long before any chaplain set foot on it, the Pydairrerme people, a band of the Oyster Bay tribe, came here to gather shellfish, prising abalone and mussels from the rocks and sheltering under an overhanging cliff. They left behind a midden, a deep accumulation of discarded shells mixed with the charcoal and ash of old campfires, and that midden still survives, protected today as an Aboriginal relic. When European surveyors first charted these waters in 1827 they called the place Opossum Island, after Captain John Welsh's sloop sheltering nearby. The renaming to Isle of the Dead in 1833 marked a hard turn in the island's story: from a place where people came to be fed and sustained, to a place where they were brought to be buried.

The arithmetic of the cemetery tells you everything about the colony's hierarchy. Roughly a thousand people lie here, though the true figure is unknowable, lost with destroyed records and incomplete registers. The Wesleyan mission kept a burial book from 1833 to 1843; only a Church of England register survives for part of the years after. What records remain show that over half the convicts buried were labourers, with shoemakers, carpenters and sawyers making up much of the rest, ordinary working men who died of dysentery, fever, accident, and sometimes murder or their own hand. Free people were buried apart, on the northern corner, beneath the roughly 180 headstones recorded in the late 1970s. It is estimated that fewer than one in ten of all the island's graves was ever formally marked. A tour guide later recalled small metal numbers pressed into the convict graves; even those went missing in the 1920s, taking the last trace of identity with them.

Two men are known to have lived on the island as its gravediggers, both of them convicts, both of them keeping company with the buried. The first was John Barron, an Irish convict who worked the island for more than a decade until he was pardoned in 1874. The second, Mark Jeffrey, an Englishman, actually volunteered for the job. He lived on the isle from Monday to Saturday and rowed back to Port Arthur only for Sunday church, digging graves until the penal settlement closed in April 1877. Jeffrey could not read or write, but late in life, ailing and poor in a Launceston depot, he narrated his story to someone who could. It was published as A Burglar's Life, a convict's own account of the dark days of transportation, told from the inside by a man who had spent years burying the people the system used up. Among those he may have buried was Henry Savery, a forger and Australia's first novelist, laid here in 1842.

For all its sorrow, the island holds one of the most quietly important objects in the science of the sea. On 1 July 1841, a broad arrow benchmark was cut into a vertical face of rock to record the mean level of the ocean, struck under the direction of Thomas Lempriere, who had been keeping meticulous tide and weather observations after the Royal Navy's surveyor drowned in 1837. The visiting Antarctic explorer James Clark Ross helped see it placed. That single chiselled line is now believed to be one of the oldest sea-level benchmarks in the world, and among the very few early measurements surviving anywhere in the Southern Hemisphere. More than a century and a half later, it gave climate scientists a precious fixed point against which to measure how far the seas have risen. In 2005 the benchmark joined the Australian National Heritage List for its exceptional value to climate research, and in 2010 the whole island became part of a UNESCO World Heritage listing of Australian Convict Sites. The dead were forgotten by design. The mark beside them turned out to remember everything.