By the end of the thirteenth century the de Braose family had made Oystermouth, not Swansea Castle, their principal residence. They could afford to. A high curtain wall went up around a courtyard. Three-storey residential blocks with fireplaces and garderobes on each floor were added inside. A chapel block rose with a traceried window facing east. Basements were dug for storage. The castle was both a fortress and a comfortable home, the kind of place a baron could entertain a king without embarrassment. Edward I did briefly visit in December 1284. The Welsh fort that supposedly stood here in earlier centuries, called Caer Tawy in medieval literature, left no trace. What we see is what the de Braoses built when they finally had enough peace and enough money to build it in stone.

The most distinctive feature of the castle today is not its curtain wall or its gatehouse, but the second-floor chapel attributed to Aline de Breos, daughter of the last de Braose lord. She inherited the lordship in the early fourteenth century and improved the chapel into one of the finest in any castle in south Wales. Its tracery windows survive. During recent conservation work, traces of a fourteenth-century wall painting were found inside, dating back over seven hundred years. The faint surviving figures appear to be angels, framed within a double-arched canopy. One wing of feathers is still visible. A circular head of yellow hair with a nimbus around it is still there. Most of the rest is gone, scraped away by weather and time, but the fragment is enough to suggest the chapel was painted from end to end. Aline married John de Mowbray, taking the lordship to the Mowbrays, and from them it passed to the Herberts and finally to the Dukes of Beaufort.

A 1650 survey of Gower described Oystermouth as an old decayed castle of no use, but of a very pleasant situation. The line is almost touching in its honesty. The military function had vanished. The residence had been abandoned. What was left was a picturesque ruin on a hill above the bay, exactly the sort of thing that eighteenth-century artists would paint over and over. George Grant Francis restored the structure in the 1840s while the castle still belonged to the Duke of Beaufort. In 1927 the Duke gave the castle to Swansea Corporation, and ever since it has been a municipal heritage site, free to wander, with views down across the Mumbles village and out over Swansea Bay toward the open Bristol Channel.

In 2009 the Welsh government announced £19 million for heritage sites across the country, and Oystermouth received about £1 million of it. The castle closed in 2010 for refurbishment and reopened in July 2011 with new visitor facilities, an education space, improved access, and a thirty-foot-high glass viewing platform and bridge leading to Aline's Chapel. The contrast between the new glass walkway and the seven-hundred-year-old stone is deliberate, intended to let visitors see the chapel painting and the upper rooms without further damaging the original surfaces. It is the kind of restoration choice that would have horrified Victorian preservationists. It works.