For most of the Middle Ages, this was the port where pilgrims from Ireland, the Isle of Man and the wider Christian world stepped onto Galloway soil to walk inland to St Ninian's shrine at Whithorn. They landed on Isle Head, where the roofless 13th-century Saint Ninian's Chapel still stands, repaired and partly rebuilt in 1898 by the 3rd Marquess of Bute. The chapel was probably erected to replace an even earlier 12th-century one whose foundations were uncovered during excavation. An outer enclosure, best seen in oblique sunlight, may date from the Celtic period. The signposted Pilgrim's Way, kept up by the Whithorn Pilgrimage Trust, still threads its 25-mile route from Glenluce down through the Machars to this point. People have been arriving here for spiritual reasons for at least sixteen centuries.

Above the harbour, the Isle Castle dates from 1674. A stone over the original door bears the initials of Patrick Houston of Drummaston and his wife Margaret Gordon. The building is small, nearly square, with angle turrets carried on three plain corbels that vanish on the inside. The Captain's Garden extends into the harbour, decorated with simulated man-o'-war gun ports. Out on the headland above all this stands The Cairn, a square white tower that has guided sailors here for centuries. During the Second World War, the cairn doubled as a tracking station for anti-aircraft gunnery practice, where radio-controlled 'Queen Bee' aircraft flew up and down in front of the guns as live targets. The concrete pad for the control hut is still there, now occupied by a memorial stone, shaped like an anchor, to the crew of the Solway Harvester.

The Isle's harbour was a working port long before it was a holy one. Pliny the Elder, writing before his death in AD 79, listed an island called vectis off the British coast that has been identified by some scholars with the Isle of Whithorn; Ptolemy may have meant Whithorn with his name Lucopibia. From the 18th century the Isle was both a haunt for smugglers and a busy legitimate harbour. Here you could board the steam packet Countess of Galloway and sail to Liverpool, where you might find passage on an emigrant ship bound for the New World or Australia. By the end of the 20th century the trading vessels had given way almost entirely to yachts and dinghies, and the harbour itself dries three hours either side of high water, leaving boats sitting on the mud.

There is one more thing the Isle is famous for, and it has nothing to do with saints. Various scenes from the 1973 horror film The Wicker Man were shot in and around the village. The climax, in which Edward Woodward's policeman is led to his fate, was filmed on the clifftops at Burrow Head just to the south, and at St Ninian's Cave. The co-owner of the Isle of Whithorn castle, Elizabeth McAdam Laughland, appeared in some of the scenes, along with other local residents. The film is now a permanent part of the village's mythology, a much stranger pilgrimage than the medieval one but, in its own way, no less devoted. Visitors still come from around the world to walk Burrow Head, and the cliffs receive them politely.