Fergus, Lord of Galloway, founded the priory in the middle of the 12th century, during the reign of King David I of Scotland, initially for a community of Augustinian canons regular. Around 1175 the Augustinians were replaced by Premonstratensian canons regular, the white-robed order colloquially known in Britain as the White Canons. The Premonstratensians had already established themselves at Soulseat Abbey before 1161. At Whithorn they took over the cathedral chapter of the Diocese of Galloway, which Fergus had revived after the old line of bishops died out in the 8th or 9th centuries. For centuries the prior stood next in rank to the bishop himself, and the community held the right to elect that bishop, although the Archbishop of York occasionally overrode them in favour of secular clergy.

By the early 16th century Whithorn Priory was opulent. Its income at the time of the Scottish Reformation was estimated at over £1,000, and in 1516, possibly because the money attracted exactly the wrong kind of attention, it was placed under the rule of a commendatory prior, an arrangement that handed monastic revenues to a powerful outsider. The Reformation broke everything open. The last Catholic prior, Malcolm Fleming, died in 1568, but not before being committed to prison in 1563 for the crime of saying Mass. He was a man caught between an old faith and a new law, and the law won. The whole property of the priory was vested in the Crown by the annexation act of 1587, and granted in 1606 by King James VI to the Bishop of Galloway when he re-established Episcopalianism.

By the Revolution of 1688, the Galloway see was the richest in the kingdom after St Andrews and Glasgow. The priory church, which doubled as the cathedral, had a long aisle-less nave, a choir of roughly the same length, and a lady chapel beyond. Thomas Sydserf, Bishop of Galloway from 1635 to 1638, undertook an ambitious remodelling of the nave. In 1684 the nave and western tower were still standing. They are not anymore. What remains today is a roofless nave and the extensive vaulted crypts under the eastern end of the church. The 3rd Marquess of Bute paid for a major excavation and restoration in 1886-7, conducted by William Galloway. The complex is now a scheduled monument.

The priory's burial register reads like a slow tour of medieval Galloway power. Walter of Whithorn, Henry of Holyrood, Simon de Wedale, all medieval bishops and priors with roots in this place. Archibald Douglas, the 5th Earl of Angus, was buried here, though his heart was carried south to Douglas in Lanarkshire, a medieval gesture of divided allegiance that Scottish nobility liked to make. Walking the roofless nave today, knowing whose ghosts are listed in the stonework, gives the site a different weight than any visitor card can manage. The pilgrims who walked here for fifteen hundred years walked for a reason: this was, by the standards of medieval Scotland, the closest you could get to the country's first saint.