The castle sits on an isolated ridge above the Western Cleddau river, which makes it the natural defensive site in this stretch of southwest Wales. There may have been an Iron Age hill fort here long before the Normans, although no physical evidence survives at the present location. Pembrokeshire was unusual among medieval Welsh borderlands in that its conquerors were not exclusively Norman. After the Norman conquest of West Wales in 1093-94, Henry I encouraged Flemish settlers to colonize the area, and the original Haverfordwest castle is credited not to a Norman baron but to Tancred the Fleming, husband of Gwladus (the aunt of the chronicler Gerald of Wales). The first stone castle replaced the timber structures around 1200, and the rectangular northeast tower that became the keep dates from this rebuilding. The castle as visible today is largely a redesign of about 1290, when much of the curtain wall and the round corner towers were raised.

Welsh princes did not let the new castle settle in peace. Gruffydd ap Rhys, Prince of Deheubarth, attacked it (unsuccessfully) in 1135-36, the first of several Welsh attempts to dislodge what was now an English stronghold deep in the Marches. In 1173 Henry II of England himself stopped at Haverfordwest on his way home from Ireland, the castle's first royal visit. In 1188 Gerald of Wales arrived with Archbishop Baldwin of Canterbury, the two of them on the famous preaching tour around Wales recruiting volunteers for the Third Crusade that Richard the Lion Heart would lead to the Holy Land. In 1220, Llywelyn the Great, the prince who would briefly unite most of Wales under his command, threatened Haverfordwest and burned the town. The castle on its ridge held. By the 14th century it had also been garrisoned by William Marshal, regent of England, who controlled much of Pembrokeshire.

By the 16th century the castle was dilapidated, the gunpowder revolution having made tall stone walls less useful than thick earthworks. The English Civil War (1642-1651) jolted it back into service. The Royalists were holding Haverfordwest in 1644 when the cattle-induced retreat occurred. Parliamentarian forces took the town, and Royalist recaptured-and-lost it again before the southwest collapsed after Colby Moor (1645). Cromwell's 1648 demolition order was carried out partially: enough was knocked down to make the castle militarily useless, enough remained for later civic projects to reuse. From the 18th century onwards the surviving walls and inner buildings were converted into a county prison. Visitors to the museum today can see the prison's original cell door, leg irons, and the great gate lock that once kept inmates inside the very same stones that had once kept Welsh attackers out.

The medieval castle's original entrance was on the west side, guarded by a gatehouse of which nothing remains. Two round towers stood at the northwest and southwest corners; a square tower with an additional projecting turret occupied the southeast. The southwest and southeast towers rose three storeys, and the southeast tower had a basement and a postern (a small rear gate) that defenders could use to counter-attack a besieging force. The great hall ran along the southern range with high windows and (the records say) scaling ladders. After the prison closed, the building served as a police station and council offices before becoming the Haverfordwest Town Museum, now operated by the Pembrokeshire Coast National Park Authority. An extensive archaeological excavation in January 2008 added to what we know about the medieval fabric. Wiston Castle sits six miles to the northeast, Pembroke Castle twelve miles to the south: between them, these three ridges held the Norman, Flemish, and English grip on Pembrokeshire for several centuries.