Henry de Beaumont, the Earl of Warwick who held most of Gower as his personal fief, founded Swansea Castle in 1107 as the caput, the administrative head, of his new lordship. The earliest castle was a timber motte and ringwork on the east side of what is now Swansea city centre, with the River Tawe flowing immediately behind it where the Strand now runs. The motte was 52 metres across, second in size only to Cardiff Castle among the Norman ringworks of south Wales. In 1116 the Welsh attacked the castle and destroyed its outer defences. In 1192 the Welsh prince Rhys ap Gruffydd, known as the Lord Rhys, besieged the castle for ten weeks; the garrison nearly starved before the siege broke. These are the kinds of facts that shape a town. Swansea grew up around a castle that had to be fought for repeatedly, and the city's place in Welsh history rests partly on the fact that it survived.

What you can see of the castle today is two sides of the rectangular south-east corner of what was once the outer bailey of the rebuilt thirteenth- and fourteenth-century castle. The south face ends in a tall garderobe tower and is capped with an elegant arcaded parapet walk, a row of small pointed arches running along the wall-head. The design is unusual for a castle and resembles structures at the Bishop of St David's palaces at Lamphey and St David's. Alina de Mowbray, daughter of the last de Braose lord of Gower, ruled the lordship from her marriage in 1306 until her death in 1331. Her son John de Mowbray probably commissioned the arcaded walk shortly after taking over, when the castle was already losing its military relevance and acquiring instead the kind of decorative flourishes a comfortable residence deserves.

By 1650 a surveyor described Swansea Castle as a decayed Building. The owners had been absentee for two centuries, the military function was long gone, and the structure was being repurposed by whoever needed walls and floors. In the 1670s the castle's square tower became a bottle factory. In 1700 a town hall was built inside the castle courtyard. By the mid 1700s the Great Hall had been converted into Swansea's workhouse, an institution that would have housed the destitute under conditions that contemporary readers find difficult to read about. The same stone walls that had repelled the Lord Rhys in 1192 were now confining men, women, and children whose only crime was poverty. The workhouse function continued into the nineteenth century. The castle was, by then, less a monument than a piece of urban infrastructure that happened to be very old.

Between 1909 and 1913 the large motte and much of the castle interior were demolished to make room for a new newspaper office. The decision would horrify a modern conservation officer; in early-twentieth-century Swansea it was a routine piece of urban renewal. The new office building housed the South Wales Daily Post, and in the very early 1930s a young Dylan Thomas, not yet famous, walked into work there each day to file reviews and short reports. He was not a particularly good employee. He drank too much, slept at his desk, and was let go after about a year. The newspaper offices stayed until 1976. Then they came down.

Through most of the twentieth century the castle was fenced off and opened to the public only on rare occasions, most recently for tours on St David's Day in 2012. In the early 2010s a project funded by the European Regional Development Fund and the Welsh Government opened the castle on a more permanent basis, with a stone-paved courtyard, information panels, and demountable stairs giving access to the upper floors. The plan was to host markets, public tours, and events tying the castle into a broader Swansea heritage trail. The remaining south-east corner still stands at the edge of Castle Square in the centre of the modern city, two sides of an old wall with the arcaded parapet still visible at its top. It is the city's oldest surviving building, and on a Saturday afternoon it competes for attention with the shopping centre across the street, which is exactly what an old castle in a working city centre is supposed to do.