The house exists because Michael Cox, Anglican Archbishop of Cashel, decided in 1767 that he needed a country seat to match his rank. He commissioned Davis Ducart, an Italian-born architect working in Ireland, to design something in the Palladian style. Ducart delivered a main block of three storeys over a basement with seven bays, extended by graceful wings, and is said to have drawn inspiration from the 1703 Buckingham House in London. Construction ran from 1767 to 1771, and to distinguish his creation from the larger Castletown House in County Kildare, the place became known as Castletown Cox. Ducart is also credited with designing the church at nearby Whitechurch, which Cox built in 1766 with help from a £200 grant from the Board of First Fruits.

When Michael Cox died in 1779, the house passed to his only son Richard, and from Richard to a grandson, also named Michael, born in 1768. The Cox family connection ran out by inheritance in 1833, when Catherine Cox married William Villiers-Stuart and the property changed hands within the family. From at least 1909 until 1921, the Earl and Countess of Dunraven made Castletown their home, but in October 1921 they auctioned off a selection of contents with Battersby and Company. In 1926, when Colonel Wyndham-Quin inherited the Dunraven title, he sold the house outright to the Blacque family and moved to the family seat at Adare Manor. The cycle of ownership and contents sales would continue: Christie's ran another major auction in 1991 for the new owner, George Magan, who became Baron Magan of Castletown.

Lord Magan undertook an extensive restoration of the house in the early 21st century, the kind of work that returns Georgian interiors to museum standard. Then in 2017 the house and 513 acres went on the market, and Country Life ran the headline that became the property's tagline. In 2019 the estate sold for approximately 20 million euros to Kelcy Warren, the Texas billionaire who built Energy Transfer Partners into a pipeline empire. In a postscript almost too neat to be real, Lord Magan was evicted from Castletown Cox that same year for failure to pay 100,000 euros per annum in rent to the trust into which he had placed the estate. The man who saved the house could no longer afford to live in it.

The formal gardens were designed by Mollie Cecil, the Dowager Marchioness of Salisbury, who was born a Wyndham-Quin and brought her famous horticultural eye back to her family's old house. Around the demesne, twin gate lodges from around 1825 are listed as being of special artistic and architectural interest, as are the farmyard, the farm manager's house, and the ruins of an ice house. From the air, the property reads as a long pale rectangle set in green Kilkenny pasture, the wings reaching out either side of the central block, the parkland stretching in every direction toward the Suir Valley. It is the kind of view that explains why every owner, from Cox to Warren, has felt the pull.