The castle was originally called Benduff - in Irish, 'the black peak.' It was built around 1470, most likely by Catherine Fitzgerald, daughter of Thomas, 7th Earl of Desmond, who married Finghin MacCarthy Reagh, head of the MacCarthy Reagh dynasty of Carbery. Catherine's father had been Viceroy of Ireland under Edward IV - one of the most powerful men in Ireland - and was executed at Drogheda in 1466, having allegedly fallen victim to a feud with the queen, Elizabeth Woodville. Catherine herself, the Annals of the Four Masters note, died in 1506. She comes down through legend as 'The Black Lady,' a half-remembered figure tied to the dark stone walls of Benduff. The castle was built in the standard Norman style: square keep, walls eleven feet thick, loopholes for arrows and light, three internal arches, and originally about seventy feet tall. The sheltered valley site was unusual - most feudal strongholds sat on high ground - but isolation here was protection.

The lands of Knocknamadogue, on which Benduff stood, had belonged to Florence McCarthy, a Catholic rebel whose estates were forfeited to Oliver Cromwell's forces during the Cromwellian conquest of Ireland in the 1650s. The surrounding lands had belonged to the O'Donovans. The English government granted them, along with the castle, to Captain William Morris - a Cromwellian soldier born around 1620. Morris took over 1,500 acres in around 1660, renaming the castle Castlesalem, the name surviving partly in deeds and partly as Castle Sally or Mount Salem in local usage. The grant was formalised in 1678 under the Act of Settlement. The confiscation was part of one of the largest forced transfers of land in European history - Catholic Irish landowners stripped of property to settle English soldiers and adventurers. Morris was a beneficiary of that displacement.

Then, in 1656, William Morris became a Quaker. The conversion of a Cromwellian soldier to the Religious Society of Friends - pacifist, anti-clerical, opposed to tithes and oaths - was remarkable. He became an active member of the emerging Friends community in Ireland, hosting meetings at Castlesalem and standing against the established church. He wrote a treatise against the payment of tithes in 1663 or 1664. The Archdeacon of Ross promptly sued him. The Archdeacon then died suddenly under circumstances that, according to John Rutty writing in 1753, so affected his fellow clergy that they did not sue the Morrises again for years. George Fox, the Quaker founder, visited Ireland in 1669; Paul Morris, William's brother, guided Fox unharmed through Cork city even though warrants were out against him. The Morrises were people of influence, and they used it for a faith their neighbours considered seditious.

William Penn first met William Morris through Quaker circles. By early 1670, when Penn was twenty-five and travelling in Ireland on legal business for his father, the friendship had become close enough that Penn stayed repeatedly at Castlesalem. He described the place in his Irish Journal as 'an old house joined to a castle of still greater antiquity, standing on a rock rising in the centre of a small romantic vale surrounded by steep and lofty hills.' He noted the great trees - yews and beeches - and a rookery improbably nested in a grove of laurels. The conversations among Quakers at the castle in those months would have ranged over persecution in England, the future of religious dissent, and the practical question of how to build a community where Friends could worship without harassment. Twelve years later, Penn would receive a charter from Charles II for the colony of Pennsylvania, founded as a refuge for Quakers and a 'holy experiment' in religious tolerance.

The castle was sold in 1853 at the encumbered estates court - a forum for the forced sale of bankrupt Irish estates after the Famine - for £1,350, bought by a Dr. Fitzgibbon. By 1870 it was in the hands of Mrs Eliza Fitzgibbon. In 1895, Patrick Daly bought the castle and the land from the Fitzgibbons, and the Daly family has farmed it ever since - three generations and counting. Michael and Margaret Daly and their family have partially restored the castle tower; the slated roof that William Morris put on it after taking off the original seventy-foot height is still in place. The castle now serves bed and breakfast guests and offers tours. The cattle still graze the fields. The rookery is still there. A National Monument by any reasonable measure, kept alive by a family that has stewarded it longer than the Morrises ever did.