The Madryn family held this estate for about two hundred years, on the south flank of the Llyn Peninsula between Pwllheli and the bulk of Carn Fadryn. Their slow decline began in the late sixteenth century with the schemes of Robert Dudley, the Earl of Leicester and Elizabeth I's favourite, whose financial pressure pushed the family toward debt. The English Civil War finished what Dudley started. Colonel Thomas Madryn sided with Oliver Cromwell against Charles I, served as MP for Caernarfonshire under the Protectorate, and found himself ruined when the monarchy was restored under Charles II. When the last male Madryn died, the estate passed during the joint reign of William III and Mary II to Owen Hughes, the wealthy Beaumaris attorney associated with the Wynn family of Gwydir Castle.

Hughes had no direct male heir either. The estate passed by marriage to the Parry family, who hyphenated themselves to Jones-Parry and held Madryn for another two centuries. The most colourful of them was Sir Love Jones-Parry, 1st Baronet (1832-1891), son of Lieutenant-General Sir Love Jones-Parry of the British Army. In 1862, with the journalist Lewis Jones, he sailed to Patagonia in southern Argentina to scout for land where Welsh-speaking emigrants might settle without the linguistic pressure of either English landlords or American assimilation. The expedition led directly to the 1865 founding of Y Wladfa, the Welsh colony of Patagonia in the Chubut valley, which still has Welsh-speaking communities today. Sir Love also rebuilt the family seat at Madryn in the mid-nineteenth century in a Welsh Gothic style: turrets, battlements, pointed windows, the full Victorian medieval fantasy.

Madryn's grounds hosted the Eisteddfod of 1875, an event that drew thousands of visitors from across Wales to the lawns and gardens around the castle. Photographs from the occasion show the white marquees, the procession, the orderly crowds in their summer best. The estate descended after Sir Love's death to his nephew William Corbet Yale, a Deputy Lieutenant of Caernarfonshire, whose own death in 1909 ended the direct line. In 1910 the castle was sold at auction along with its art collection: van Dyck, Holbein, Hoppner, and Gainsborough canvases that had hung in its rooms for generations were dispersed to dealers and collectors across Britain and beyond.

Through the twentieth century the building drifted through various uses, partly residential, partly institutional, never quite finding a steady purpose. A fire in the 1960s gutted the main block. In 1968 what was left was pulled down, and a caravan and holiday park was built within the estate grounds. The Tudor Gatehouse, a fragment of the older castle that pre-dated the Victorian rebuild, survives as a Grade II listed building under Cadw, an isolated stone arch beside a country lane. It is the last visible piece of two hundred years of Madryn family history, and another two centuries of Hughes and Parry stewardship, and the launching point of a Welsh colony eight thousand miles away in Patagonia where the language is still being spoken.