Norman expansion into south Wales began in the late 1060s, soldiers pushing westwards out of recently conquered England and building castles wherever they paused. Bernard de Neufmarché conquered Brecknock in 1091 and assigned the manor of Hay to Philip Walwyn, one of his followers. Walwyn built the first castle here outside the main settlement, on a mound near St Mary's church called Hay Tump. Sometime in the late 11th or early 12th century, a new fortification was built inside the town itself, two hundred metres higher up the hill: an earth ringwork with a stone gate tower. This was the beginning of Hay Castle. The English lordship around it was called Hay Anglicana, a name that distinguished it from the Welsh-speaking territory just over the river Wye. The town grew, walls were built, and by 1200 the castle had been substantially rebuilt in stone by the de Braose family.

The de Braose castle had a turbulent first century. During the First Barons' War in 1215, Reginald de Braose joined the alliance against King John, and the king himself attacked the castle and took it. In 1231 the Welsh prince Llywelyn ab Iorwerth (Llywelyn the Great, grandfather of the last Llywelyn) burnt both town and castle. Henry III rebuilt it in 1233. During the Second Barons' War in 1263, Prince Edward (the future Edward I) captured it; the following year Simon de Montfort and Llywelyn ap Gruffudd took it back and burnt it again. By the time the antiquary John Leland visited in the 1530s, the town was "wonderfully decaied" and the castle, he wrote, had once "bene right stately" but was in decline. Hay had been on the front line of Anglo-Welsh wars for two hundred years, and the architecture showed every battle.

In the 17th century someone - either Howell Gwynne in the early years or James Boyle of Hereford in 1660 - built a new house against the old medieval keep. The Jacobean mansion was two storeys tall with three counting the facade and seven dormer gables in a Dutch style, with a large staircase and formal gardens outside the keep. It was an oddity: a comfortable country house with a 12th-century military structure attached to it like a stone tumour. The house was divided up among tenants in 1702 and passed eventually into the Wellington family. Until 1812, the basement of the medieval keep was used to supplement the town gaol. In 1809 the industrialist Sir Joseph Bailey leased the castle and bought it in 1844, adding terraced gardens and a walled kitchen garden. The Jacobean mansion was used as a vicarage from 1825, including by Archdeacon William Bevan.

Around 1961 the castle was acquired by Richard Booth, a wildly eccentric Oxford graduate who had decided to open a bookshop in Hay-on-Wye. He chose the town because property was cheap, transport was easy enough by road, and the surrounding farmland and chapels were full of unwanted libraries. By the late 1960s, Booth had opened multiple bookshops in Hay and was working on a strategy of mass acquisition that involved buying entire university and seminary libraries by the lorryload. The castle became one of his shops; it also became a venue for parties and a holiday cottage. The 1977 declaration of independence was theatre but it worked. Hay-on-Wye became known as the town of books, the Hay Festival started in 1988, and by the early 21st century the town hosted around two dozen specialist booksellers. The castle, meanwhile, was rotting. A fire in 1977 destroyed the western half of the interior. Booth lived elsewhere on the site while the building decayed.

In 2011 the castle was sold for around two million pounds to the Hay Castle Trust. Rick Mather Architects were appointed to design the restoration. Grants from the National Lottery Heritage Fund of over five million pounds, with additional money from the Welsh Government and private donors, funded an eleven-year project to restore the Jacobean mansion as a centre for arts, literature, and learning. The fragments of the medieval keep were stabilised. The wooden door on the left side of the gateway, which probably dates from around 1300, was conserved in place. The right door, from the early 17th century, was kept too. On 26 May 2022, more than nine hundred years after the first stone gate tower was built, Hay Castle opened to the public for the first time in its history. The keep now has a viewing platform at the top, looking out over the Wye and the bookshops below.