After the de Lacys, the lands passed to the Knights Templar, the warrior monks of the Crusades who maintained estates across Western Europe to fund their wars in the Holy Land. The Templars held Clontarf until their order was suppressed by Pope Clement V in 1312. The estate then passed through various hands, including the Knights Hospitaller who absorbed many Templar properties, before being dissolved by Henry VIII in the 1540s along with every other Irish monastic foundation. The last prior of Clontarf, John Rawson, struck a soft landing for himself: in 1541 he surrendered the castle and its lands to the Crown and was created Viscount Clontarf in return. Sixty years later, in 1600, Queen Elizabeth I granted the estate to Sir Geoffrey Fenton, her secretary of state for Ireland, and the property passed by marriage into the King family. George King supported the Irish rebellion of 1641 and was duly stripped of his lands when the Cromwellian forces consolidated their grip on the country.

On 14 August 1649, two days after Oliver Cromwell landed at Ringsend with his New Model Army, the Clontarf estate was given to Captain John Blackwell. Blackwell soon sold his interest to John Vernon, Quartermaster General of Cromwell's army. Vernon held the estate when the restoration of Charles II in 1660 might have unwound many such Cromwellian grants - but he held on, passed Clontarf to his son Edward in 1660, and his descendants would hold the castle for nearly three centuries. The last of the direct male line, Edward Kingston Vernon, succeeded in 1913 on his father's death, lived in the castle for only six months, and then let it to John George Oulton and his wife Mona, who eventually bought it outright in 1933. The Vernons had held Clontarf longer than the Tudors held the English throne, longer than the Romanovs held Russia, longer than the United States has existed as a country.

George Frideric Handel stayed in Dublin for nine months in 1741 and 1742, during which time he composed Messiah and gave its world premiere at the Music Hall on Fishamble Street on 13 April 1742. He was a frequent visitor to Clontarf Castle during this period. The lady of the house at that time was Dorothy Vernon, who had grown up in Hanover and was, in the careful Victorian phrasing of one source, 'particularly intimate' with the composer. Handel wrote a piece called Forest Music for her, said to combine German and Irish melodies; the score is now lost. The neighbouring shoreline of Dollymount is traditionally said to take its name from the same Dorothy Vernon. Almost a century later, J. M. W. Turner painted the castle for his patron Walter Fawkes, who had married Maria Sophia Vernon. Turner had never actually visited Ireland, but he worked from one of Maria Sophia's own sketches. The finished watercolour was mislabelled 'Caltarf Castle' and its true subject was only conclusively identified in 1998. It depicts the building that preceded the present 1837 structure, and it is generally regarded as Turner's only Irish landscape.

After the death of J. G. Oulton in 1952, his son Desmond sold the castle to pay death duties. It stood empty until 1957 and changed hands twice in the 1960s before Gerry and Carmel Houlihan bought it in 1972 and reopened it as a cabaret venue - one of the most popular in the country through the 1970s and 1980s. Tom O'Connor, Maureen Potter, and the accordionist Dermot O'Brien all recorded live albums at Clontarf Castle. The Derry-born singer Dana Rosemary Scallon was crowned Queen of Cabaret there in 1968. Two years later she would win the Eurovision Song Contest for Ireland with 'All Kinds of Everything', becoming the first Irish winner and one of only seven non-Irish-speaking acts ever to win Eurovision with a song in English. Phil Lynott of Thin Lizzy memorialised the building in 'The Friendly Ranger at Clontarf Castle,' the opening track of the band's 1971 self-titled debut album. The cabaret years ended in 1998 when the building was reopened as a 111-room four-star hotel.

The 1837 building by William Vitruvius Morrison is what visitors see today: the older castle was found unsafe and was demolished by the Vernons when Morrison's new one was complete. The current structure is in the Gothic Revival idiom that nineteenth-century Anglo-Irish landlords favoured when expressing their attachment to medieval romance. As the hotel, it has been substantially enlarged with modern wings on either side. Most of the former estate is long since gone to housing - the village of Clontarf has absorbed the parkland in waves through the twentieth century. What remains is a modest curtilage with an ornamental gatehouse, much of it now car parking. Cyril Connolly, the English literary critic whose Enemies of Promise was one of the most influential books of twentieth-century literary criticism, was a grandson of the Vernons. His childhood memories of the castle in the early 1900s open his memoir. The castle he describes - shabby, half-empty, full of inherited furniture nobody knew what to do with - is the one the last of the Vernons sold in 1933.