Walworth changed hands more often than most castles. The Hansards held it from 1150 until Elizabeth Hansard, an infant heiress, married Sir Francis Ayscough in 1539. The Ayscoughs held it briefly before selling to Thomas Jenison, who rebuilt the place around 1600. Jenison's widow Elizabeth née Birch was the one who entertained King James in 1603. In 1679 Francis Jennison sold up and emigrated to Europe. The castle passed to the wine merchant Matthew Stephenson in 1759 for £16,000, then to Newcastle merchant John Harrison in 1775, then by marriage to the Aylmer family in 1819. The Aylmers held it until 1868, when a tragedy on the railway changed everything.

On 20 August 1868, John Harrison Aylmer, his wife, and his eldest son were travelling on the Irish Mail express from London to Holyhead when it collided with stray wagons loaded with paraffin near Abergele in North Wales. The fire killed 33 people. Among them were three Aylmers of Walworth. Their younger sons Vivian and Edmund, aged 12 and 9, inherited the castle. Vivian grew into the kind of Victorian aristocrat his era produced in quantity. He became High Sheriff of Durham and a big game hunter. In 1885 he crossed the Horn of Africa, exploring Somaliland. He died in 1931 and was buried, with his brother, at St Philip's Church in Caerdeon, North Wales, far from the inheritance the railway disaster had given them.

During the Second World War, Walworth Castle housed prisoners of war. Two hundred German and Italian officers lived here under the command of Major Rollin Holmes. Their huts stood to the east of the castle, and some of those buildings may still be identifiable on the site. In 1950 Durham County Council bought the castle and turned it into a girls' boarding school. The transformation from prison camp to schoolroom took only five years. For three decades the corridors echoed with adolescent footsteps where soldiers had once been confined. Then in 1981 the council sold the building again, and the new owners renovated it as a hotel. It joined the Best Western group and has welcomed paying guests ever since.

The castle is built from partially rendered limestone rubble, roofed in Welsh slate, with a Tudor manor house design from around 1600. The interior was reworked in 1740, gaining Palladian plasterwork and Rococo details. In 1864 the main staircase was rebuilt. The medieval south-west tower of the previous castle still stands incorporated into the building. The parkland south of the castle was originally enclosed demesne land, and ridge-and-furrow patterns from medieval farming are still visible in the fields. Archaeologists have identified a possible enclosure and a U-shaped earthwork at the southern edge. There may have been a lost settlement here called North Farm, perhaps two settlements, perhaps one that shifted location over time. The 1870 gate lodge imitates a medieval battlement, a Victorian fantasy of how a real castle gate ought to look.

The castle has accumulated ghost stories the way old buildings do, and the hotel has occasionally leaned into them at Halloween. In 2002 the restaurant earned three-star classifications from both the RAC and the AA. In 2007 the castle was lit pink for Valentine's Day. In 2008 it sat on the route of the Quaker Triathlon, organised by the local Rotary Club. The Darlington Education Village held its formal ball here in 2009. Today the building is still a polling station during elections, the same rooms that once held German prisoners and royal travellers now used for the most mundane civic duty. A medal belonging to Thomas Jennison, the man who built the place around 1600, was unearthed during road work in 1937 or 1938. It showed a bridge and his name.