Henry I died in 1135, and the years of political chaos that followed — known to historians as the Anarchy — left the northern English counties exposed. David I, King of Scots, sent raiders under William Fitz Duncan to harry Cumbria and Northumberland. Calder Abbey, only four years old, was one of the houses they reached. The Scots despoiled the wooden buildings and drove the monks out. The community fled back to Furness for sanctuary, but Abbot Gerold refused to resign his abbacy to the Furness leadership, and the dispute pushed them out again. They began a wandering life that reads like a slow medieval pilgrimage: first to Hood near Thirsk in Yorkshire, then to Old Byland near Rievaulx Abbey, then to Stocking, and finally to a permanent home where they built the great Byland Abbey — one of the most beautiful Cistercian foundations in England. After Gerold travelled to Savigny in France to plead his case, his community was released from Furness's jurisdiction in 1142. Calder, by then, had been silent for four years.

A new attempt at colonisation came in 1142, under Abbot Hardred, again from Furness — but this time with the protection of William Fitz Duncan, whose Scottish raiders had wrecked the first community. Politics in the borderlands moved like that. The Savigniac order was amalgamated into the Cistercian order in 1148, and Calder, by extension, became Cistercian as well. The monks built a stone church by 1180, of which the west door is the main surviving piece, then rebuilt most of the structure in the early English style in 1220 under Thomas de Multon of Egremont. Calder was never wealthy. By 1535 its annual net income was just £50 — a small house by monastic standards, in a quiet valley, doing its quiet work for four centuries.

Scottish raids came again in 1216 and 1332. By 1381 the community was down to four monks and three lay brothers. At the Dissolution in 1535–36, the only recorded relic in the monastery's possession was a girdle — a belt — claimed to have belonged to the Virgin Mary. King Henry VIII's commissioners filed an unfavourable report against the abbey and its community, though their motives were often suspect even at the time. The abbey was surrendered on 4 February 1536 by the last abbot, Richard Ponsonby, and his nine remaining monks. They walked out, and Henry gave the buildings to Sir Thomas Leigh, who pulled off the roof, sold what he could, and reduced the church to a ruin. The girdle of the Virgin Mary disappeared into whatever destination such things go after the Dissolution. So did most of the abbey's records.

Parts of the south and east claustral ranges were incorporated into Calder Abbey House, an early-nineteenth-century private residence built atop the medieval foundations. The rest of the abbey remains as a picturesque ruin — its 13th-century cruciform church, with aisleless presbytery, transepts holding two eastern chapels each, and an aisled nave of five bays, standing roofless beside the River Calder. The oldest pieces, the west door and parts of the transepts, date to around 1175. Just upstream, where the River Calder meets Friar Gill, sits Matty Benn's bridge — a medieval packhorse bridge spanning the river that is still in use and open to the public. The abbey itself is on private land. New trees obscure much of the view. In 1934, on the 800th anniversary of the founding, over 2,000 people attended a mass held in the ruined church, led by Dr Thomas Wulstan Pearson, OSB, the Roman Catholic Bishop of Lancaster. The community that David's raiders broke in 1138 had been remembered, for one afternoon, in the place where it had first tried to take root.