Running along the top of the central facade between the two towers, in carved stone, is a partial quotation from the Roman poet Lucretius: Suave mari magno turbantibus aequora ventis e terra magnum alterius spectare laborem. Pleasant it is, when over a great sea the winds trouble the waters, to gaze from shore upon another's great tribulation. It is the opening of Book II of De rerum natura, and it expresses the philosophical pleasure of recognising one's own safety while watching others struggle. Stuart de Rothesay had crossed Europe in wartime, had served as ambassador in Vienna, St Petersburg, Madrid and Paris, had seen empires rise and fall. The view from his oriel window opens south across landscaped gardens to the chalk pinnacles of The Needles and the Isle of Wight. The sea below the cliffs has been the boundary between safety and trouble for as long as people have lived here.

The architect was William Donthorne, a founder member of the Royal Institute of British Architects. The castle is built on an L-shaped plan, oriented on a south-east axis so that the oriel window looks straight out to The Needles. But the stones were where the strangeness lay. Stuart de Rothesay had acquired large quantities of carved medieval stonework from two ruined Norman sites: the Benedictine Abbey of St Peter at Jumieges, and the Grand' Maison of Radeval at les Andelys. Both had fallen into disrepair after the French Revolution. The stones were shipped across the Channel and worked into the new English castle. A sixteenth-century oriel window and an ancient stained glass window were also incorporated. Highcliffe is, in part, a French abbey rebuilt as an English fantasy on the Dorset coast. The castle was completed in 1835. Charles became Ambassador to Russia in 1841, and died at Highcliffe in 1845, buried at St Mark's Church a short walk from the gate.

When Charles's widow Lady Elizabeth died in 1867, the castle passed to her younger daughter Louisa, who had married Henry, Marquis of Waterford in 1842. Louisa, born in Paris in 1818, had become an accomplished painter; her work is still valued today. After her husband's death in 1859 she lived in Ford, Northumberland, where she painted Biblical scenes on the walls of a school hall that survives as Lady Waterford Hall. She inherited Highcliffe in 1867 and divided her time between Northumberland in winter and Dorset in summer. The painter Augustus Hare visited and recorded a typical day: late breakfast in the room looking across the sunlit sea to the Isle of Wight through magnolia leaves, reading aloud in the porch-room, painting in the library, walks on the sands. In 1880 the Prince and Princess of Wales, the future Edward VII and Queen Alexandra, made an unexpected visit, landing on the beach with their two sons and three little princesses and drawing fishing nets on the shore until evening. Louisa died in 1891.

The next owner was a distant cousin, Edward Stuart Wortley, who had distinguished himself in the First Boer War. In 1907 Kaiser Wilhelm II stayed at Highcliffe for three weeks to recover his health, and the visit was widely publicised. The Kaiser presented two stained glass windows to his host, installed in one of the rooms before he left. The following year he invited Stuart Wortley as guest of honour to the German Army manoeuvres in Alsace. Between 1916 and 1922 the castle was leased to the American retail entrepreneur Harry Gordon Selfridge, who had opened his great Oxford Street store in 1909. Selfridge installed modern bathrooms, steam central heating and a new kitchen at Highcliffe, even as a tenant. His wife Rosalie founded the Mrs Gordon Selfridge Convalescent Camp for American Soldiers in the castle grounds during the war. Rosalie died in the 1918 influenza pandemic. Harry's mother Lois died in 1924. He gave up the lease in 1922 but asked, when his time came, to be buried at St Mark's Churchyard at Highcliffe next to his wife and mother. He is.

Violet Stuart Wortley sold Highcliffe in 1949. The buyer turned it into a children's convalescent home. Unfounded allegations led to a court case that closed the home; the castle was sold on, partly for housing development, then to the Claretian Missionaries who used the great hall as a chapel and trained student priests. They left in 1966. A fire damaged the Great Hall in 1967, and another the following year removed the main staircase. The castle slipped into ruin, exposed to weather and vandals. National concern was raised by English Heritage, the Ancient Monuments Society, the Victorian Society, SAVE Britain's Heritage. Christchurch Council compulsorily purchased the building in 1977 for £65,000 and opened the grounds to the public to mark the Silver Jubilee. Restoration began in earnest in 1994. The Heritage Lottery Fund agreed to £2.6 million in 1998, the East Tower was rebuilt, the Wintergarden was repaired, and the castle reopened. Phase Six in 2008 opened the state dining room and the East Tower; the Penleaze Wing followed in 2018 after eighteen months of further work. Highcliffe is now Grade I listed, open to the public, and described as the most important remaining example of the Romantic and Picturesque style in England.