Sir Rice Mansell, who died in 1559, started the buildings sometime between 1520 and 1538. The Gateway and the South Range went up first, the entrance plaque carved with the coats of arms of the Mansell family and of the Penrice and Scurlage families to which Sir Rice was related, every shield insisting on the lineage that justified the scale of what was rising behind it. The site already had medieval history. A 1306 charter from Swansea mentions tenants of ancient knight's fees at Oxwich, suggesting fortifications had stood here before, and a ruined tower to the north-east of the present castle may be a fragment of the earlier work. Sir Rice swept most of that away. The house he built was a statement, and his son Sir Edward Mansell, between 1560 and 1580, made it a louder one.

The East Range that Edward Mansell added carried a great Hall, and together with the six-storey South-East Tower it offered enough room to house a substantial household and a small army of guests. This was the architecture of the Elizabethan prodigy house, the kind of building designed to entertain a monarch should one ever drop in. Oxwich became one of the leading examples of the form in Wales, with kitchens in the South Range, a stone dovecote to the north-east supplying meat year-round, and accommodation extensive enough to embarrass a county sheriff. It is possible the East Range bankrupted Sir Edward. The Mansell finances were already strained, the project was enormous, and by 1632 the family had leased the castle out and moved on. The Great House was less than fifty years from completion to absence.

Parts of the building collapsed in the eighteenth century. After that the South Range was patched up and rebuilt as a farmhouse, which it remained, more or less, into the twentieth century. By 1949 demolition was being seriously considered. The state took ownership instead, and Oxwich passed into the care of what is now Cadw, the Welsh heritage agency. The Hall is roofless. The East Range is missing most of its upper floors. The South-East Tower still stands to its full six storeys, with its window mullions intact and a clear view across Oxwich Bay to the dunes and salt marsh beyond. The grand house is gone but the shape of grandness survives.

In 1968, during routine maintenance work at the castle, a man named Cyril Grove found a gold ring brooch in the ground. It is two inches in diameter and dates to between 1320 and 1340, decorated with two rubies and three cameos, and it is one of the finest pieces of medieval jewellery ever found in Britain. The cameos are believed to have been added later; sapphires may have once filled their settings. Provenance is unknown. The brooch may have been part of the treasure of Edward II, or a Mansell family heirloom that someone hid and never returned to collect. It is on display today at St Fagans National Museum of History. The brooch is the kind of find a maintenance worker dreams about, dug out of a piece of ground where people had been walking, ploughing, and digging for six hundred years.