The present farmhouse is 16th century with an earlier medieval core - possibly originally a tower house, an architectural style more common in Pembrokeshire than here. The walls are local sandstone from a quarry on the manor's own land, Garreg Llwyd. A line of corbels runs along the south-facing walls, a feature particular to old Carmarthenshire buildings. There were originally seven square chimneys, two unusually set diagonally in the east wall - which makes it odd that the 1672 hearth tax records the house as having only two fireplaces. The discrepancy is almost certainly fraud: rather than pay tax of two shillings per hearth, the owners stripped or hid most of them. The same trick was tried with windows after 1697, when a window tax of three shillings per opening was introduced. Many of Court Farm's stone and wooden mullion windows were simply blocked up with masonry. The tax was repealed in 1851, but several of the west wall windows have remained sealed shut ever since.

The first recorded owners came in 1128, when Maurice de Londres granted the Manor of Pembrey to Sir John Butler of Dunraven Castle. The Butler name has an odd origin. The family's founder, Arnold, was actually the butler - the household officer - to William de Londres, lord of Ogmore Castle. When the inhabitants of Glamorgan attacked Ogmore while William was away campaigning at Kidwelly, Arnold the butler successfully defended the castle, and was rewarded with Dunraven and a new identity: Arnold le Boteler, anglicised in later generations to Butler. As manorial tenants of the Lord of Kidwelly, the Butlers were obliged in time of war to provide five archers "according to ancient custom." Their wood at Pembrey was called Coed Marchog, Knight's Wood, and it lies on the east side of the old Pembrey-to-Pinged road below Garreg Lwyd. The Butlers held Court Farm for centuries until the line ended with Ann Butler, who married Sir Richard Vaughan of Bredwardine and carried the manor out of the family.

Next to the house stands an unusual large barn with an embattled parapet on its south elevation - a defensive military feature on what should be an agricultural building. Whether it was actually defensive nobody can now say. It may have been part of a more extensive curtain wall, or it may have been decorative crenellation in an era when defensive theatre still mattered. Pembrey in the medieval period sat on a frontier: Norman-held lands to the east, the Welsh kingdom of Deheubarth to the north-west pressing back against Kidwelly and the coast. Local landowners would have been useful, occasionally, with a body of armed retainers and a strong building to keep them in. The barn's parapet is one of those architectural details that quietly remembers a period when farming and soldiering were not entirely separate trades.

Court Farm was abandoned around 1948. One room still kept its Jacobean panelling when the door closed for the last time; that panelling is presumably long gone. In 1984 an application to demolish the building was lodged, and refused the following year on the strength of a council feasibility study. Locally a campaign formed around the schoolteacher John Evans, who wrote letters to everyone he could think of - the Lord Chancellor, both archbishops, the Welsh Office, members of parliament, academic societies, the Prince of Wales (who wrote back, in support). Cadw funded half the cost of the study. The Reverend W. Roberts of Burry Port submitted an essay to the Western Mail Civic Pride Competition in November 1985 arguing for preservation; he ended with the line, "the ancient St Illtud's Church remains; the notorious wreck-strewn and beautiful Cefn Sidan Sands remain; the third part of the trinity must also remain." The house remains. Whether it will be restored, and to what use, has been argued now for more than forty years.

A fresh feasibility study was commissioned in 2003 by the architectural firm Davies Sutton, who had successfully restored Sker House, a building of similar character on the Glamorgan coast. Their structural engineers, Mann Williams of Cardiff, inspected the walls in February 2003 and reported back in language that managed to be both technical and quietly hopeful: "the main walls remain reasonably plumb and stable when considering the extensive period of neglect. Replacement of floors and roof structures will reinstate the necessary support to the walls and provide a significant improvement to stability." In other words: the bones are good. Whatever the building eventually becomes - museum, dwelling, tourist office, all have been mooted - the case for keeping it has been made on architectural, historical and stubbornly local grounds for two generations. Sometimes the best evidence of a building's worth is the length of the argument it provokes.