The recorded religious history of Caldey runs back to the sixth century, when a saint named Pyr is described as abbot of the monastery around the year 500 in the Life of St Samson. The Welsh name Ynys Bŷr, the Island of Pyr, comes directly from him. The English name comes from later visitors. The Vikings, raiding along this coast from the ninth century onward, called it Kald ey, the cold island, and the name stuck in Anglicised form through medieval documents that record it as "Caldea" by the early 1100s and "Caldey" by 1291. After the Norman Conquest, Robert fitz Martin, Lord of Cemais, gave the island to his mother Geva. In the twelfth century Tironensian monks from St Dogmaels Abbey, sixty miles north on the Pembrokeshire coast, established Caldey Priory here, and it remained a working religious house until Henry VIII dissolved it in 1536.

The story goes back further than any of the monks. Three caves on the island, Nanna's Cave, Potter's Cave, and Ogof-yr-Ychen, the Ox Cave, have yielded human remains stretching back to the Upper Palaeolithic. In Ogof-yr-Ychen the bones of people who lived between 7,590 and 5,710 BC were excavated, and stable isotope analysis showed they had eaten almost entirely from the sea. They were Mesolithic foragers working the shoreline of an island that already, nine thousand years ago, sustained a tiny human population. Potter's Cave, discovered by a Caldey monk named James Van Nedervelde in 1950, contained three human skeletons embedded in stalagmite. Two were Mesolithic. One was Romano-British. The bones are now kept either at the abbey or at the Tenby Museum and Art Gallery, where you can see them on display.

The current Caldey Abbey is a strange and beautiful building, an Arts and Crafts experiment finished in 1910 by the architect John Coates Carter for an Anglican Benedictine community. Carter gave it white roughcast walls, red-tiled roofs, a tapering church tower with primitive crenellations, and five side-windows lighting the nave. Pevsner called it the most complete example of Arts and Crafts ecclesiastical architecture in Britain. The Anglican monks fell into financial trouble by 1925 and sold the property in 1929 to a community of Belgian Cistercians from Chimay, who had been displaced by the First World War and were looking for a permanent home. The Belgians converted to the Roman Catholic Church and made Caldey one of only two Trappist monasteries in the United Kingdom. They have been here ever since.

The island's history also contains decades of serious harm. Multiple cases of child sexual abuse linked to Caldey have come to light since 2011. A monk named Thaddeus Kotik was found in a 2017 civil case to have abused six girls on the island between 1972 and 1987. Two other men with abbey connections, John Shannon and John Cronin, have been convicted of child sex offences. A fugitive on child-images charges was found living at the abbey in 2011 after seven years in residence. In 2024 the abbey commissioned an independent review led by social worker and former assistant police and crime commissioner Jan Pickles OBE, which was published in December 2024. The abbot, Father Jan Rossey, apologised publicly for the abuse and the cover-up. The community now works with the Catholic Safeguarding Standards Agency, has a safeguarding officer, and has adopted the no-touch policy for visitors that Pickles recommended. The history is honestly recorded on the abbey's own website. The girls and the women they became are at the centre of that record.

Near the abbey stands the Caldey Stone, a sixth- or seventh-century inscribed pillar bearing both an Ogham inscription and a Latin one, one of the rare bilingual stones from the Celtic Christian period in Wales. It is among the most significant early medieval monuments in the country. Beyond it the island works. About forty permanent residents live alongside the twenty Cistercian monks. Red squirrels were reintroduced in 2016 after a rat eradication programme cleared the way. Rare-breed sheep and cattle graze the meadows. The boats from Tenby Harbour, 2.5 miles north, run all spring and summer, depositing visitors who buy the perfume and the chocolate and walk to the lighthouse on the south cliffs. Caldey is an island that has been continuously inhabited for nine thousand years, continuously prayed on for fifteen hundred, and continuously reckoning with what its institutions have done for at least the past decade. All of those things are true at the same time.