Robert was one of the most powerful magnates of Norman England. Created Earl of Gloucester by his father, he became the chief military prop of his half-sister Matilda in her long civil war against King Stephen - the period English historians used to call simply 'The Anarchy'. The Bristol he based himself in was a fortified port, growing rich on Atlantic trade and slave-trading raids into Ireland. The priory he founded outside the castle walls in 1129 was Benedictine, dependent on Tewkesbury Abbey. Robert endowed it in 1137 with the right to hold an annual fair - the St James's Fair, which would become one of the great commercial gatherings of the West Country for the next seven centuries.

On 9 January 1540, Henry VIII's commissioners surrendered St James' Priory to the crown as part of the Dissolution of the Monasteries. Most monastic buildings in England were stripped of their lead, their stone carted off, and their walls left to fall. St James' very nearly went the same way. The cloister and conventual buildings were indeed demolished. But the nave survived - because of an agreement signed back in 1374 between the Abbot of Tewkesbury and the parishioners of St James', which had transferred the nave to the local people for their use as a parish church. When Henry's men came for the monastery, the nave was no longer monastic property. They left it standing. In 1543 the priory site and the right to hold the fair were sold to a London merchant-tailor. The fair survived under various owners for centuries more.

Robert himself was buried here in 1147. His wife, Mabel FitzRobert, Countess of Gloucester, joined him. So did Eleanor, the Fair Maid of Brittany - one of the saddest figures of medieval royal history. Eleanor was the niece of King John and, by strict primogeniture, had a stronger claim to the English throne than he did. John kept her imprisoned for the rest of her life, moving her between castles, ensuring she never married, never had children, never became a rival. She died in 1241 at Bristol Castle, where she had spent her final years, and was brought across the river to be buried at St James'. Her name appears on the priory's burial register as a quiet, awful coda to the politics of the Plantagenet succession.

By the early 2000s, St James' was in trouble. The fabric was deteriorating. The building sat on Historic England's Buildings at Risk Register, classified as in very bad condition. A consortium led by the St James Priory charity raised the cash to save it: a £3.2 million grant from the Heritage Lottery Fund, matched by £1.2 million raised by the charity itself. Conservation work began in November 2009. Twenty-one months later, on 25 July 2011 - the feast of St James - the church reopened. By 2014 it had come off the Heritage at Risk Register entirely. During the work, archaeologists from Bristol and Region Archaeological Services found a small block of Bath stone, carved with hour lines and medieval Arabic numerals. It looks like a fifteenth-century scientific sundial, and may be the earliest of its kind in Britain.

Until 1996, St James' was an Anglican parish church. That year, the Little Brothers of Nazareth re-established it as a Catholic place of worship within the Diocese of Clifton, and set up the St James Priory Project - a service that helps vulnerable people, particularly those with histories of substance dependency and mental illness, find their way back. During the restoration came another discovery that made headlines around the world: a medieval statue in the church, long thought to be modestly draped, turned out to have originally been topless. Earlier centuries of churchwardens had quietly covered her up. The 2011 conservators uncovered the original. A 900-year-old building still keeps surprises.