Henry de Beaumont, Earl of Warwick, conquered Gower for the Normans in the early years of the twelfth century and parcelled out manors to consolidate his hold. Pennard, perched on a limestone spur above the mouth of the Pennard Pill stream and the white sand of Three Cliffs Bay, became one of his demesne holdings. The first fortification was a timber ringwork, an oval enclosure 34 by 28 metres with a ditch and rampart, holding a timber hall in the centre. To the north and west, cliffs dropped sharply to the stream and the dunes below. To the east a church and a small settlement grew up, and a rabbit warren was established in the nearby dunes for fresh meat. It was a modest castle for a modest manor, useful as a base from which to hold a corner of Gower.

Sometime in the early thirteenth century a stone hall replaced the timber building on the same spot, built from red-purple sandstone with white limestone detailing. Around the turn of the fourteenth century, while the castle was in the hands of William de Braose and his son of the same name, the timber defences were replaced with a stone curtain wall about eight metres tall, with battlements and arrow loops. A square mural tower went up on the western spur, a circular tower on the north-west corner, and at the entrance the Braoses built a gatehouse with two half-circular towers. The design was self-conscious, the towers echoing those at Caerphilly Castle a few miles inland, but the actual defences were thin. The gatehouse had a single portcullis and only a few arrow loops. It looked impressive without being especially formidable, which was probably the point.

The Braoses may have built Pennard in stone as a replacement for their castle at Penmaen, which they had just been forced to abandon to encroaching dunes. The decision did not buy them much time. Within a generation or two the same dunes that had swallowed Penmaen began to overrun Pennard. The church of St Mary's filled with sand and was abandoned. The settlement around the castle thinned out and disappeared. A 1650 survey called the fortification desolate and ruinous, surrounded by sand. By 1879 large cracks had appeared in the southern tower of the gatehouse and it partially collapsed. The Pennard Golf Club, which by then owned the site, met with the Royal Institution and the Cambrian Archaeological Association in 1922 to discuss saving the rest. The estimated cost of proper repair was too high. The gatehouse was patched with concrete instead, in 1923 and 1924, the kind of compromise that buys decades but rarely centuries.

The remaining southern wall collapsed at the start of 1960. An archaeological survey followed in 1960 and 1961, and urgent masonry repairs in 1963 were paid for jointly by the Ministry of Public Building and Works, the Gower Society, the golf club, and a public appeal launched by local newspapers. The ruins today include the gatehouse, still reaching up to its battlements on the east side thanks to its unusually strong mortar; the north and east curtain walls, now about a metre thick and averaging five metres in height; and the remains of the square mural tower on the western spur. They are protected as a Grade II* listed building and scheduled ancient monument. The Pennard Golf Course wraps around the ruins, so visiting now means walking carefully along fairways toward the cliff edge, where Three Cliffs Bay opens below and the sand still creeps up the dunes.