Hayton Castle began as a fourteenth- or fifteenth-century tower house, the type built across the border country in response to Scottish raids and English counter-raids. It was extended in the sixteenth and seventeenth centuries, and gradually domesticated from fortress to family home. The current building is a rectangle running 29 metres north to south and 16 metres east to west. Walls thick enough to stop a battering ram, a barrel vault in the basement, and a single two-metre-thick wall through the centre of the building all point to the original castle's bones underneath later additions. The windows tell their own history: fifteenth-century, Elizabethan, and Georgian openings sit in the same elevation, marking each owner who decided the place needed more light. It was officially designated as a Grade I listed building on 11 April 1967.

The manor of Hayton was originally granted by Alan, second Lord of Allerdale, to his huntsman Seliff, whose descendants took the name de Hayton. The line passed through several families and reached the Musgrave family through a marriage in the fifteenth century. From Nicholas Musgrave, who died in 1506, the castle descended through fourteen generations for roughly three hundred years. Names recur: Sir Edward Musgrave, MP, who died in 1641. Sir Edward Musgrave, the 1st Baronet of Hayton Castle, made a baronet of Nova Scotia in 1638 and the same man who later gave his horse to the king. Through Richard, Richard, Richard, the title passed until Sir Richard Musgrave, the 5th Baronet, took the name Hylton from his uncle's will and died without children in 1755. The title moved on; the manor stayed.

In the early nineteenth century the manor of Hayton separated from the baronetcy and passed to Mrs Jolliffe, a younger daughter and eventually sole heiress of Sir Richard Musgrave. The Jolliffes never lived at Hayton. They preferred to lease it as a tenant farm, which is how the castle entered a quieter chapter. A list of nineteenth- and early twentieth-century tenant farmers reads like a parish register: the Reverend Isaac Robinson in 1816, John Blackburn and his son in 1828, Nancy Blackstock the widow of John in 1882, Thomas Biglands, remembered locally as a sportsman, until 1903. That year the castle was bought outright by David Mitchell, recently returned from three years of service in the British Army during the Anglo-Boer War in South Africa. His family still owns it today.

Hayton Castle is one of several Cumbrian houses claimed to have sheltered Mary, Queen of Scots in the days after she fled across the Solway Firth from Scotland in 1568. The legend has her staying here on the night of 17 May. On leaving, she is said to have given her host a miniature painting of herself, set into the centre of a plain gold ring, which passed down through generations as a family treasure. The legend also has her spending the same night at Cockermouth, a few miles away. Both cannot be true, and history has never quite decided which is. The Musgrave coat of arms above the seventeenth-century fireplace was put up by Anne Musgrave, daughter of Sir Richard. On the outside north wall, an older shield of the Martindale or Mulcaster family survives. In the garden, a sundial dated 1725 carries the initials of Sir Richard Musgrave and his wife Anne, marking a small and patient claim that they were here.