Robert de Say, nicknamed Picot, was an early Norman baron who took the land from the Anglo-Saxon thegn Edric the Wild after the Conquest. Roger de Montgomery, Earl of Shrewsbury, granted Picot twenty-seven manors of which Clun was the largest, and from these holdings the Marcher Lordship of Clun was assembled. Picot threw up a motte-and-bailey castle on a defensible bend of the river. The Saxon village stood on the south bank around the church of St George; the new Norman town grew on the north bank below the castle. The two halves were eventually joined by a 14th-century pack horse bridge that still carries traffic today and still attracts the old Shropshire saying: whoever crosses Clun Bridge comes back sharper than he went.

What made Clun strategically significant was the castle-guard arrangement Henry I established around 1100. Knights from a chain of fiefs east of Clun owed forty days of military service a year to defend the castle, with their fiefs linked to it by the old Roman road running along the river. Henry II invested heavily in stone walls and towers between 1160 and 1164. Twenty-five Welsh settlements in the area also owed feudal military service to the castle. By the 13th century Clun was the centre of an honour, a Marcher barony governed by its own law, where the lord had powers the king reserved everywhere else. The great keep that survives today, eighty feet tall and four storeys, was built into the side of the motte rather than on top of it, an unusual arrangement chosen to spread the load on the earth mound. Five tall round-arched windows ventilated each of the three upper storeys. The arrowslits are mostly decorative. This was a castle built for comfort as much as defence, designed for a family that increasingly preferred to live in style.

The Fitzalans inherited Clun through Isabella de Say's marriage to William Fitzalan, Lord of Oswestry, around 1155. They became the dominant Marcher dynasty in this corner of the frontier. Llywelyn the Great pressed against the Marches in the 1230s and Clun was garrisoned by royal troops to keep it secure. After Edward I broke Welsh power in the 1280s, the military function of Clun receded. The Fitzalans, who also held the much grander Arundel Castle in Sussex, turned Clun into a hunting lodge with pleasure gardens. The castle saw a brief military revival during the Glyndwr Rising of 1400-1415, when Thomas Fitzalan refortified it against the Welsh rebels. By the time the antiquarian John Leland visited in the 16th century, the place was already falling into ruin. The English Civil War passed it by; there was nothing left worth garrisoning. Parliament slighted it anyway in 1646, on general principle.

Sir Walter Scott is said to have stayed at the Buffalo Inn in Clun while writing parts of his 1825 novel The Betrothed, basing his fictional castle Garde Doloreuse on the ruin above the town. The town's most famous literary visitor was the poet A.E. Housman, whose A Shropshire Lad gave the local villages a couplet that has followed them ever since: 'Clunton and Clunbury, Clungunford and Clun, are the quietest places under the sun.' Research by the Campaign for the Protection of Rural England in 2009 essentially confirmed the verdict, ranking Clun among the most tranquil locations in England. From the air the keep still rises sharp against the river bend, the three smaller mounds around the motte still visible as humps in the grass, the Victorian sketch's romantic ruin remarkably unchanged. The Duke of Norfolk still holds the title Baron Clun. The lord no longer hangs criminals.