In 1641, Humphrey Weld purchased the castle from Howard's heir, beginning a family connection that has lasted into the twenty-first century. The Welds were Catholic at a time when that identity carried real danger. After the English Civil War, during which Roundheads seized the castle and used it as a garrison, Humphrey Weld regained the property but struggled with mounting debts. Successive generations married strategically and managed carefully to keep the estate alive. When Thomas Weld inherited in the 1770s, he refurbished the interiors in Adam style and enriched the library. He also entertained George III, a relationship that bore extraordinary fruit.

In 1786, Thomas Weld built a Roman Catholic chapel dedicated to St. Mary in the castle grounds, the first Catholic chapel erected in England since the Reformation. There was a condition: King George III required that it not look like a church from the outside. The architect John Tasker obliged, designing it in the form of a Greek mausoleum. The chapel's significance extends beyond architecture. On 15 August 1790, John Carroll was consecrated there as the first Catholic bishop in the United States, the Bishop of Baltimore, by Bishop Charles Walmesley. A building designed to be invisible became the birthplace of American Catholic episcopal authority.

Following the French Revolution, surviving members of the French royal family were invited to use Lulworth as a residence in exile. Charles X of France and his family stayed briefly after the July Revolution of 1830, on their way to Edinburgh. The castle had become a place where the dispossessed found shelter. Then, on 29 August 1929, fire gutted the building. The Adam-style interiors that Thomas Weld had so carefully crafted were destroyed. The family built a new residence nearby and left the castle as a roofless shell, open to the Dorset sky, for nearly half a century.

In the 1970s, restoration began with help from English Heritage. The work took until 1998 and included a new roof and stabilized walls, though no internal floors or walls were reconstructed. The result is something between a ruin and a building: a restored exterior enclosing hollow, cathedral-like spaces. The Weld family still owns the estate, which now hosts visitors, medieval-themed events, and since 2017 the Bestival music festival, whose main stage sits in the castle's foreground. Part of the surrounding Lulworth Estate serves as a Ministry of Defence firing range, adding an unexpected layer to the landscape. In 1986, the Baltimore-born organ builder William Drake restored the 1780 Seede organ in the chapel, closing a transatlantic circle that began when John Carroll crossed the ocean for his consecration two centuries earlier.