Long before James Butler chose this spot, Gowran was already old. The kings of Ossory - the Mac Giolla Phádraig, ancestors of the Fitzpatricks - kept a residence here, and were often called the kings of Gowran in the annals. The local placenames still carry the residue: Rathvaun, Rathcash, Rathcusack, Rathgarvan - rath meaning ringfort, the small circular earthworks where Iron Age and early medieval Irish lived, farmed, kept livestock and buried their dead. Larger sites were called duns; the village of Dungarvan, in the same parish, is one. Just inside the door of nearby St Mary's Church stands a Christianised ogham stone from the third or fourth century, with a Christian cross carved over its pagan inscription, the layered evidence of a place that was sacred long before the Normans arrived in 1169.

After the Norman invasion, Theobald FitzWalter received a grant of 44,000 acres - the Manor of Gowran - and was named first Chief Butler of Ireland. His descendants, the Butler family, held the land for almost five hundred years. They built more than one castle in the area: Ballysean Castle near the centre of Gowran, Neigham Castle four kilometres out, Paulstown Castle on the road towards Carlow. The Butlers were powerful enough to marry royalty - James Butler, the second Earl, was called the Noble Earl for being a great-grandson of Edward I of England through his mother. In 1501 Margaret FitzGerald, Countess of Ormond, rebuilt the Butler castle here and decorated St Mary's Church, including some of the surviving sculptural work in the nave. By then Gowran was a walled town, with markets, parliamentary representation, and a Magdalen hospital built outside the walls in 1578 for, as the records gently put it, the relief of poor leprous people.

The town of Gowran, under the command of Colonel Robert Hammond - who happened to be Oliver Cromwell's cousin - surrendered to the Parliamentary army on 21 March 1650 after a brief siege. The garrison were offered quarter and accepted; they handed their officers over to Cromwell, expecting the officers to be allowed honourable surrender too. Cromwell ordered all but one of the officers shot by firing squad. A Catholic priest captured in the castle was hanged. The castle itself, attacked during the siege, was badly damaged - not destroyed, but enough that no one repaired it. For the rest of the century the building stood ruinous, slowly losing stones to local masons, while the Butlers were exiled and their lands granted to Cromwellian officers and settlers.

In 1713 Henry Agar - whose family had come from Yorkshire around 1650 in the aftermath of Cromwell - built a new castle close to the ruined Butler one, recycling its stones. The Agars would hold Gowran for the next three centuries. They became Viscounts Clifden, sat in the Irish House of Lords, and produced the Honourable Liliah Agar-Ellis, who married Lord Annaly in 1885 and became known locally as Lady Annaly. She presented silver cups to the village cricket club, hosted race meetings at what is now Gowran Park - the racecourse, opened in 1914, is still one of Ireland's best-known venues for National Hunt racing, and Arkle himself won here in 1964. The current castle - the third on the site - was built between 1816 and 1819 by Henry Agar, second Viscount Clifden, to designs by the Kilkenny architect William Robertson, who also designed most of the present Kilkenny Castle.

In 1957 the Irish Land Commission sold the castle and 68 acres of surrounding land to James and Mary Moran. The Morans lived in it for four decades, the longest continuous residence by any family since the Agars. In 1998 their heirs sold the castle to an Isle of Man company controlled by a Northern Irish developer. Plans for a 156-house development on the demesne were refused by Kilkenny County Council in 1999; a 2004 plan for a hotel, apartments and holiday lets was refused too. Then on 16 May 2010 the castle, by then unoccupied for eleven years and deteriorating, caught fire. The Kilkenny Fire Brigade reached it quickly; the building was badly damaged but not lost. NAMA, the Irish state asset agency, took over the castle's portfolio in 2011 after the Irish economic collapse, and in 2013 the property was sold again. Restoration began that year and was completed in 2014. The third castle is back.

Today Gowran Castle stands as a substantial cut-limestone country house with three storeys, side bays, and the broad orderly proportions of late Georgian-Regency design. The 4.5-mile demesne wall that the Agars built around it in the eighteenth and nineteenth centuries is still standing and now forms part of the boundary of Gowran Park racecourse. The castle is private but the surrounding landscape - the racecourse, the medieval church, the streets of Gowran with their old town wall fragments - is publicly accessible, and the town's small triangular plan still records where the Norman foundation went down on the older Irish settlement, eight centuries ago.