The recorded history of Isel begins in the reign of Henry II, when Alan, the son of Waltheof, granted the demesnes of Isel, Redmain, and Blindcrake to a Norman lord named Randulph d'Engayne. His granddaughter Ada married Simon de Morville, lord of Burgh by Sands. After several rounds of inheritance through coheirs and second marriages, the manor passed to Margaret, daughter of William Moulton. In 1315, Margaret married Sir William de Leigh, and Isel entered the Leigh family for the next two and a half centuries. William died in 1354 and was buried beside his horse in the Isel churchyard. The Leighs held Isel until 1572, when Thomas Leigh, the last of the name, gave the estate to his second wife Maud Redmain. Her third husband, Wilfred Lawson, conveyed it to himself.

The Lawsons traced their descent to John Lawson of Fawkegrave in Yorkshire in the early thirteenth century. The Sir Wilfrid Lawson who acquired Isel through Maud Redmain was born in 1545 and died in 1632 at the age of eighty-seven. He had been educated at Trinity College, Cambridge in 1562 and at Gray's Inn in 1564, served as High Sheriff of Cumberland four times, sat in parliament for Cumberland four times, was knighted in 1604, and helped govern the Anglo-Scottish border. He died childless, and Isel passed to his great-nephew Sir Wilfrid Lawson, who became the 1st Baronet of Isell in 1688. The baronetcy ultimately ran for ten generations before extinction with the 10th Baronet in 1806. At Isel itself, though, the direct line faltered earlier: the 4th and 5th Baronets both died in infancy, ending the Isel branch of the family in 1743. After that, the residency becomes sketchy, with branches of the family probably continuing to occupy the house until the Wybergh family arrived in 1806.

The most striking feature of Isel Hall is the pele tower, the oldest part of the building together with the Great Hall. The exact date of construction is not recorded. What is known is that in 1387, an army of Scots under the banners of the dukes of Douglas and Fife raided and captured Cockermouth Castle and laid waste to the surrounding countryside. The present tower most likely dates to a time shortly after that raid, built by owners who had just seen how vulnerable the valley was. A Carnarvon arched doorway in the basement points to construction in the early fifteenth century. The tower rises thirteen metres on a rectangular footprint of about thirteen by 7.75 metres, with walls two metres thick and a vaulted basement divided by a cross wall. The masonry is freestone rubble with red sandstone dressings to the sixteenth-century windows. Above the basement, three floors are reached by short flights of stairs and passages.

When Sir Hilton Lawson, the 4th Baronet of Brayton, died in the twentieth century, Isel Hall was bought privately by Margaret Austen-Leigh of Fareham in Hampshire, a first cousin of the previous owner. In 1941, Margaret married Richard Austen-Leigh, the great nephew of Jane Austen and a publisher who had revised the edited works of his great aunt. He died in 1961. During her years at Isel, Margaret bred Shetland ponies and large poodles. She had no children. When she died in 1986, she left the hall to her friend and distant relative Mary Burkett OBE. Mary had recently retired as director of the Abbot Hall Art Gallery and Museum in Kendal, where she had won the first Best Museum of the Year award and served on national arts committees. With help from English Heritage, she carried out substantial improvements to the pele tower, the terrace, and the sunken garden. She also compiled a memoir of a former parlour maid at Isel Hall, Miss May Moore. Mary Burkett died in late 2014. The hall passed to its current owners.

The facade of the hall has a curious ornament that exists in only one other English country house. The line of the eaves is broken at regular intervals by an open arch rib superimposed on top of the wall. The feet of the arches sink just below the eaves and rise above the slates with a terminal at the crown. Smaller, unpierced versions appear between them. The only known parallel is at Hardwick Hall in Derbyshire, a comparison that would have flattered any Cumbrian builder. In 1960, three days of auction at Isel disposed of much of the interior. The marquee on the terrace lawn held 1,071 lots over thirty pages of catalogue, including 109 of silver, 264 of furniture, and nearly 300 of linen alone. The highest price paid was £300 for a Hepplewhite mahogany break-front bookcase, a sober Cumbrian sum for a sober Cumbrian house.