William Lowther, 1st Earl of Lonsdale, built the castle between 1806 and 1814 as his architect Robert Smirke's first major commission. Smirke went on to design the British Museum. William inherited from a notoriously disreputable cousin, Sir James Lowther, who had owed William Wordsworth's father a large debt and refused to pay it for years, leaving the orphaned Wordsworth siblings in poverty. When William inherited in 1802 he immediately repaid the debt with interest, befriended the poet, and assisted him financially for the rest of his life. Wordsworth stayed often at Lowther, writing many of his published letters from there. Robert Southey came too, and J.M.W. Turner painted Lowther Castle - Evening, which now hangs in the Bowes Museum. In 1839 the American politician Daniel Webster visited with his wife, both keeping diaries describing the castle bell rung at their arrival through the arched stone gateway.

Hugh Cecil Lowther, the 5th Earl of Lonsdale, inherited in 1882 at the age of 25 with no preparation for managing a great estate. He had left Eton at twelve to play sport. He spent the next half-century spending what his ancestors had built. He kept yellow-liveried footmen, a chamberlain, a master of music in charge of 24 musicians who travelled house to house, his entire household moving by special train. Kaiser Wilhelm II came to Lowther for grouse shooting in 1895 and again in 1902, the imperial flag flying over the house. The kings of Italy and Portugal followed. The Kaiser gave Lonsdale the Prussian Order of the Crown first class, and a Mercedes. Hugh's passion for cars made him the first President of the Automobile Association. He raised the Lonsdale Battalion of the Border Regiment for the First World War, a pals battalion of Cumberland men. They were almost wiped out on the Somme.

Hugh outlived his money. In 1921 he sold Whitehaven Castle, in 1926 Barleythorpe Hall in Rutland, the same year the West Cumberland coal mines closed and the Lowther income collapsed. In 1935 he simply left Lowther Castle because he could no longer afford to live there, moving to smaller accommodation nearby. He died in 1944 aged 87, having outlived his wife Grace by three years. She had lost her only baby to a hunting accident in her twenties and spent the rest of her life as a partial invalid. The 6th Earl inherited a vastly diminished estate and held a great auction sale of contents in 1947. Then came September 1942, when the tenants were summoned to a meeting in Penrith and told their land was being requisitioned for war work. A column of Matilda tanks rolled into the grounds. Lowther, like nearby Brougham Hall, became a site for the secret Canal Defence Light project. The army damaged the grounds and buildings, and by the time the war ended the castle had been empty for years.

James Lowther, returning from the Second World War, called the castle a place that exemplified gross imperial decadence during a period of abject poverty. He tried to give it away to three local authorities. All three refused. He could not afford to open it to the public and could not afford the death duties. So in 1957 he ordered the roof removed, the upper masonry brought down, and the gardens turned into farmland. The forecourt held the pig pens. The Japanese Garden, created in 1904 by local gardener Thomas Richard Hayes with imported Japanese trees and curved bridges, vanished under spruce. For decades the empty shell of the castle sat at the centre of the estate, weather coming in through what had been the great rooms, gardens reverting to wilderness.

In 2000 the Lowther Estate and English Heritage commissioned a conservation plan, and in 2005 a partnership with the Royal Horticultural Society began to regenerate the site. The castle and 130 acres of grounds transferred to a charitable trust in 2007. After an initial nine million pound consolidation, the ruin and some gardens opened to the public on 22 April 2011, the first time visitors had been admitted since 1938. Designer Dan Pearson now oversees the gardens, working from plans by Patrick James and Dominic Cole. The hand-built children's playground uses eleven miles of sustainably sourced timber. The 2018 Cumbria Tourism Awards named Lowther the Large Visitor Attraction of the Year. The roofless walls remain roofless, deliberately so, the ruin presented as ruin rather than restored to imitate what it once was. The wider estate is undergoing rewilding, and red kites circle above what used to be the family seat of an earl who once owned every yellow car in Cumberland.