Kilclief is the oldest tower house in the Lecale peninsula, the long flat block of County Down between Strangford Lough and the Irish Sea where late medieval Anglo-Norman lords concentrated thickly enough to leave more castles per square mile than almost anywhere in Ireland. The building style at Kilclief, with its four floors and its tall parapet and its paired projecting turrets joined at the top by a high arch, is something close to a template for what came later. Look at Jordan's Castle in Ardglass and you will see the same idea. Look at Audley's Castle a few miles up the coast on Strangford Lough and you will see the same idea simplified. Kilclief is what the late medieval Lecale lordship looked like at the moment the form was being worked out. The construction dates of 1412 to 1441 line up with Sely's rise to the bishopric and place him squarely at the centre of the project.

The castle is four storeys tall. The ground floor is vaulted in stone, the masons setting their barrel vault on wicker centring in the same fashion as at Jordan's Castle a few miles down the coast, the imprint of the woven willow still visible in the mortar on the underside of the curve. Two projecting turrets rise the full height of the tower at the seaward corners. One, to the south-east, contains the spiral stair that links the floors. The other, to the north-east, carries a stack of garderobes, the medieval indoor latrines, each one opening off a different floor and dropping through a shaft inside the wall. At roof level the two turrets are joined by a high arch, a machicolation, with a drop hole through which defenders could pour or push unpleasant things on anyone trying to break in at the door below. The battlements above are stepped in the Irish manner. On the second floor a thirteenth-century coffin-lid from a nearby church has been reused as the lintel above the fireplace. It is the kind of detail that tells you the builders did not have unlimited stone and that even sacred things, in a frontier lordship, became raw material when needed.

The historical record on John Sely is thin enough that what survives is mostly the disgrace. He was Bishop of Down for fourteen years. He built the castle, or had it built, on land that belonged to the see. He lived there openly with Lettice Whailey Savage, a married woman who shuttled between Kilclief and another property she kept at the equally evocative-sounding Smithing-Upon-Down, and who, the medieval chroniclers note, was an avid collector of rare ceramics. The 1443 ejection is the moment the story breaks the surface. What happened to Sely afterwards is not well-recorded. Lettice is mentioned only by way of his disgrace, never in her own right, the way medieval women so often are. Whatever the texture of their lives at Kilclief, the building they shared survived the scandal in better shape than the bishop did. It was garrisoned for the Crown in 1601 by Nicholas FitzSymon and ten warders during the Nine Years' War, used as a working defence of a still-strategic stretch of Strangford coast, and continued to stand in good repair into the modern era.

Kilclief Castle is now in state care, managed by the Department for Communities' Historic Environment Division, and guided tours are offered in July and August. The tower stands on the road between Strangford village and Ardglass, two and a half miles south of Strangford, with the lough opening out to the north and the Irish Sea pushing against the shore to the east. The two-light window in the east wall is a modern reconstruction based on a single surviving fragment found in the rubble, but the rest of the upper-floor windows are essentially as Sely left them. From the parapet, on a clear day, the Mournes lift above the southern horizon, the long line of Strangford Lough runs north, and the coast of the Isle of Man is faintly visible across the water to the east. The view that the bishop's mistress kept her eye on five hundred years ago has not changed in any of the ways that matter.