The castle is first mentioned in a document around 1200 as part of the line of peel towers that watched the Tweed Valley for raiders. It was associated with the Frasers and may have been named for Oliver Fraser, who gave lands to Newbattle Abbey and whose name is preserved in the abbey register. The Frasers exerted power from Oliver Castle for generations as hereditary Sheriffs of Tweeddale, an office that Bernard and Gilbert Fraser held in turn. The family produced figures of national importance. William Fraser rose to become Bishop of St Andrews and died in exile in France during the Wars of Scottish Independence. His brother Sir Simon Fraser of Oliver and Neidpath, a Knight Banneret, did the opposite: he stayed and fought.

Sir Simon Fraser is one of those medieval figures whose story reads like a tragedy of conscience. He swore loyalty to Edward I, then broke with him to fight for Scotland, then submitted again, then broke away again. His most famous victory came at Roslin in 1303, when a Scottish force defeated three English columns in a single day. Three years later he was captured by Edward's men and dragged south. In September 1306 he was hanged, drawn, and quartered in London. The estates passed to his daughter Joan, who married Sir Patrick Fleming of Biggar. For nearly two centuries Oliver Castle belonged to the Flemings, until 1524, when Thomas Tweedie murdered John, 2nd Lord Fleming. The Flemings held on to the estate for a while longer through John's son Malcolm, but eventually Oliver Castle passed to the Tweedies, the same family whose member had committed the murder.

The Scottish Marches were notoriously unruly. Disputes ran on cattle and sheep, on long-remembered grudges, on whose grandfather had stolen what from whom. In an episode entirely characteristic of the period, Thomas Porteus of nearby Hawkshaw was arraigned on 16 February 1489 for the theft of seventy-four lambs from the lands of Oliver Castle, then belonging to William and Lawrence Tweedie. The exact count of seventy-four lambs is the kind of detail that makes the legal record feel suddenly alive. Someone had counted, someone had complained, the matter had reached a court. Multiply that across hundreds of years and dozens of families and the everyday texture of Border life comes into focus: not the grand battles of Bannockburn and Flodden, but seventy-four missing lambs and a neighbor who knew where they had gone.

By the seventeenth or eighteenth century the Tweedies had built a new house near the old tower, reusing stones from the medieval ruin. That house in turn was replaced around 1780 by the present Oliver House, begun by Thomas Tweedie of Oliver. The old laird's residence beside it bears the date 1734, with the initials of James Tweedie and his wife Margaret Ewart, and a still older date of 1649 was once inscribed on its walls. Oliver House remains occupied and lived in. The hillfort site to the northeast is a scheduled monument. Visitors to the upper Tweed can see Oliver House from the road and walk to the knoll where Sir Simon Fraser's ancestors built their tower. There is little to look at and a great deal to imagine.