The site had been fortified before, but the present castle dates from after 1602, when raiders from the Clan Mackenzie burned Strome Castle on Loch Carron. The MacDonells of Glengarry, a powerful branch of Clan Donald, responded by hardening their seat on the Raven's Rock. The completed building was already old-fashioned by Scottish standards of the late 17th century. It was a tall, narrow tower house rather than a true fortification, with five storeys in the main block and six in the tower wing, six small bartizans at the corners, but no defensive parapets. The thick walls would not have stood serious artillery. The interior, behind those plain elevations, would have been rich, with painted ceilings, tapestries on whitewashed walls, a large chimney piece in the great hall.

In 1654, during the wars of the three kingdoms, troops under General Monck attacked the castle on Oliver Cromwell's orders and burned it down. The MacDonells repaired it. By 1688 the family was holding the castle for King James VII of Scotland against the new monarchs William and Mary. It surrendered to government forces in 1692. The Jacobites retook it during the 1715 rising and lost it again in 1716. They held it once more during the 1745 rising, when Bonnie Prince Charlie himself stopped at Invergarry. He rested at the castle shortly after raising his standard at Glenfinnan, and again, broken, on the way north after Culloden in 1746. Lord MacDonell sat on the Prince's Council. The MacDonells were as committed to the Stuart cause as any clan in the Highlands.

After Culloden, the Duke of Cumberland set about systematically dismantling Highland power. His troops reached Invergarry and tried to blow the castle up. The thick walls would not entirely yield. So the engineers settled for what military historians call slighting: removing the southeast wall and demolishing the northwest extension that contained the staircase. The castle was never rebuilt. The MacDonells moved to a new house, the future Invergarry House, somewhere nearby. By 1760, when the cleric Richard Pococke passed through on his Scottish tour, the new dwelling was complete. The old castle was left to the weather and the ravens.

We know how the castle looked because of an unlikely document. In 1714, with another Jacobite rising looming, a French engineer named Brigadier General Lewis Petit des Etans was sent north by the Board of Ordnance to survey buildings of military significance. He compiled at least eighteen drawings, signed off in his hand, of Scottish castles. One of them was Invergarry. The plan at first floor level shows the staircase in the northwest wing. The elevations show the building tall and narrow, the bartizans neat at the corners, the cap-house above. The drawing is held by the National Library of Scotland, and it is the most precise picture we have of the castle in its full form, a generation before Cumberland's powder arrived.

Around 2000 the north staircase, the last upstanding internal element, collapsed. Historic Scotland organised a programme of consolidation in 2007, inserting stabilising beams to keep the remaining walls upright. The work is described in detail in Renewed Life for Scottish Castles. The ruin is now a scheduled monument, owned since 1960 by the Invergarry Castle Preservation Trust. The neighbouring Invergarry House became the Glengarry Castle Hotel in 1960, set in the same grounds. In 1957 the Glengarry News published an open call to all McDonalds, MacDonalds and MacDonells worldwide to donate a pound to save the castle. The walls that Cumberland's men could not bring down are now held up by beams and a small trust, with Loch Oich beneath and the canal carrying narrowboats past in summer.