Castle Down holds 66 cairns dating to the Bronze Age. Most are flat platform cairns - low stone piles built up over a generation or two and then left as monuments rather than active burial places. Just south of the SSSI boundary, an Iron Age field system covers another 3 hectares: low stone walls, hut circles 5 to 7 metres across set 5 to 20 metres apart, and middens up to 11 metres long, 10 metres wide and a metre high. The middens are kitchen-rubbish heaps from the Iron Age - bones, shells, broken pottery, the residue of meals eaten on this windswept plateau two and a half thousand years ago. The field system was reused in the nineteenth century when Tresco farmers came back to the same workable ground. The same place, the same low walls of granite, repurposed across two millennia of human absence and presence.

Two artillery forts sit on the western side of Castle Down. The older - King Charles's Castle - was built between 1548 and 1551 in the reign of Edward VI, perched too high above New Grimsby harbour for its guns to fire effectively downward. For a brief period it was the principal stronghold in the islands, but by the 1590s it had been superseded by the much better-sited Star Castle on St Mary's. The name King Charles's came later, from Royalist occupation during the English Civil War. In June 1651 the Parliamentary admiral Robert Blake took the islands from the Royalists, and Cromwell's Castle was built the following year on the rocks below, on the foundation of a sixteenth-century blockhouse. It was updated around 1740 with a platform for cannon on the seaward side. The two castles still stand on the same headland - one a sixteenth-century failure, the other a seventeenth-century replacement, both English Heritage sites today.

In 1652 the Parliamentary Survey of Scilly recorded a row of shallow pits and spoil heaps on Castle Down, following the line of a tin lode. The pits ran 1.8 to 2.4 metres deep with some shafts reaching 7.3 metres, and at the western end of the workings there was an adit - a horizontal access tunnel cut into the side of the hill. The mining had started in the 1640s and was abandoned by 1652. The surveyors recorded that it was of no value. Cornwall and the Scillies share the same Hercynian granite geology, the same mineral-bearing veins that produced enormous tin wealth on the mainland, but on Tresco the lode was simply too thin or too erratic to repay the effort. The spoil heaps remain, low ridges in the heather where the gorse grows thinner over disturbed ground.

On the north coast, where the cliffs drop to the Atlantic, a fissure in the granite leads down into a cave called Piper's Hole. The cave consists of a 20-metre boulder-filled passage that opens into an underground pool. Reaching it requires a scramble down the cliff and then a careful traverse over wet stones in the dark. Victorian tourists could ride deeper into the cave: a punt was kept at the inner pool during the nineteenth century, and visitors paid to be poled to the back of the chamber by lantern-light. In 1993 the cavers Peter and Myrtle Ashmore investigated the cave's troglophile fauna and found a springtail - Onychiurus argus - that was new to Britain. The species had been recorded in caves in Belgium, France and Spain, but never before in the British Isles. Piper's Hole is a reminder that the most local-feeling places can still hold things no one has ever catalogued.

The vegetation on Castle Down is ankle-high. Thin podzolic soils over coarse-grained Hercynian granite, constant salt-laden wind, and almost no shelter combine to prune everything down to a compressed carpet of plants. Western gorse and bell heather dominate, with common bird's-foot trefoil, English stonecrop, heath bedstraw, lousewort and tormentil mixed in. Forty-five species of lichen have been recorded on the down, including unusual Heterodermia communities. The Isles of Scilly are the only European sites for one species, Heterodermia propagulifera - the rest of its range lies in tropical and subtropical zones. The southern edge of the Late Devensian ice sheet reached the northern islands of Scilly about 18,000 years before present, and you can still see glacial outwash gravels with erratic pebbles on the northern part of the downs - rocks the ice sheet dropped and walked away from.