Long before any earl took the title, this site held a Roman fort, with a garrison of perhaps 300 men, operating between 80 AD and 170 AD. Archaeological excavations to the northwest of the castle uncovered the foundations. After Rome withdrew, the location stayed strategic: it sat at the entry to the upper Clyde Valley from the south, guarding the Mennock Pass that funneled travellers between England and central Scotland. Records show Sveinn as Lord of Crawford before 1100, his name suggesting Norse origins. His son Thor, Lord of Tranent and Sheriff of Edinburghshire, succeeded him. Crawford Castle was in existence by 1175 and was probably first built as an earthwork and timber stronghold by Thor or his father Sveinn. The Barony of Crawford was at that time the largest and most influential barony in southern Scotland.

The Lindsay family inherited the barony when William Lindsay married Thor's younger daughter, granddaughter of Sveinn, around 1154. William Lindsay probably built the stone castle by 1175. He is recorded as Lord of Crawford by the late 1180s. The Clan Carmichael of Meadowflatt acted as hereditary constables of the castle from an early date, retaining the post under successive owners. This was a long-tenured family relationship of a kind common in medieval Scotland, where loyalty bound generations as firmly as marriage and land. The Lindsays held Crawford for centuries, accumulating the prestige that culminated in David Lindsay's earldom in 1398 and the famous duel that earned it. The castle had become Lindsay Tower in popular speech, the family's stone embodiment in the Clyde Valley.

At the accession of James IV in 1488, the barony was transferred to Archibald Douglas, 5th Earl of Angus, as reward for supporting the young prince's rebellion against his father James III. The Earls of Angus held Crawford until 1528 when their estates were forfeited by the young James V. The king then used Crawford as a hunting lodge until his own death in 1542. James V's mistress, Elizabeth Carmichael, was the daughter of the hereditary constable, and she became the mother of John Stewart, Commendator of Coldingham. In July 1541 James V and Mary of Guise came to Crawford bringing tapestry from the royal collection to furnish their lodging at the castle. George Carmichael, son of the Captain of Crawford, presented three ounces of Scottish gold to the queen. Royal visits to a remote castle leave traces like these: a hanging, a small gift, a king and queen sleeping above a hunting park.

After 1542 the barony returned to the Earls of Angus. The keepership of the Carmichaels of Meadowflatt finally ended in 1595. In 1633 the 11th Earl was created Marquess of Douglas, and the castle was probably rebuilt around this time. Much of what survives today dates from this 17th-century refurbishment. The castle later passed to the Duke of Hamilton, then to Sir George Colebrooke in the 18th century. After a period of use as a farmhouse, the building was abandoned at the end of the 18th century. Much of its stone was carried away to build the present Crawford Castle Farm. Four stone tablets bearing coats of arms, one with the date 1648, are built into the west and south walls of Castle Crawford House nearby. Old castles in Scotland often end this way: not as ruins of romance but as quarries of practical reuse.

The earliest earthworks at Crawford comprise a motte around 5 metres high with a surrounding ditch and a bailey of about 45 by 33 metres to the southwest. On the motte stand the remains of a curtain wall surrounding an enclosure about 20 metres square. Round towers may have stood at the corners, though only outlines remain. A range of buildings on the southwest side was tower-like, three storeys plus an attic, with a vaulted basement and projecting chimney-breast. A second range was added later in the 17th century with larger windows. The arched recess in the east wall suggests a single-storey projecting building. Crawford Castle is a Scheduled Ancient Monument, listed by the Royal Commission as a castle or motte. The Lindsay name lives on in the family that still bears it. The stones, scattered into farmsteads, remember the rest.