Penrith Castle was built piecemeal between 1399 and 1470 as a defence against Scottish raids. The exact builders are tangled in conflicting evidence. William Strickland, who became Bishop of Carlisle, was once credited with the original work, but a tenurial reconstruction suggests he never held the castle site at all, and his pele tower was probably elsewhere in Penrith. The most likely builder is Richard Neville, Earl of Salisbury, with earlier groundwork possibly laid by Ralph Neville, 1st Earl of Westmorland, who held the lordship of Penrith from 1397. The castle and lordship became the most important Neville offices in the fifteenth century and a key source of local patronage. Professor A.J. Pollard has estimated the Nevilles' Penrith estates were worth approximately 350 pounds, a substantial sum in mid-15th-century terms.

Salisbury was killed at the Battle of Wakefield in 1460. His son Richard, Earl of Warwick, the Kingmaker, inherited the castle but died at the Battle of Barnet in 1471 without leaving a male heir. The castle reverted to the Crown. King Edward IV granted it later that same year to his younger brother Richard, Duke of Gloucester, the future Richard III. Gloucester used Penrith as a base for managing northern affairs, was appointed Sheriff of Cumberland five consecutive years, and was described in records of 1478 as of Penrith Castle. Tradition holds that he stayed at the nearby Dockray Hall, once the Gloucester Arms, during building work on the castle itself. When Gloucester seized the throne as Richard III in 1483 and then died at Bosworth in 1485, the castle returned to royal possession. It would never again play such a central role in English history.

After the Tudors, Penrith Castle and the town remained Crown estate for two centuries. In 1696 William III gave it to his Dutch friend Hans Willem Bentinck, 1st Earl of Portland, along with most of the other Crown property in Cumberland. The Earls and Dukes of Portland sold it to the Dukes of Devonshire in 1787. They in turn sold it to the Lancaster and Carlisle Railway Company, which built Penrith railway station on land that had once been the castle's outworks. The station and the castle now sit directly opposite each other, separated by a road. In the 1920s the ruins passed to Penrith Urban District Council, which laid out the grounds as a public park and built housing nearby. The castle's afterlife as municipal greenspace is improbable but suits it surprisingly well.

Penrith Castle today is maintained by English Heritage and listed Grade I on the National Heritage List for England. The ruins, according to one nineteenth-century description, are remarkable more for their extent than their magnificence. Projecting corbels on the eastern front once supported an open corridor. Large vaulted rooms below, probably prisons in their working days, still survive. The walls are broken in places, intersected with the ghosts of windows, and from different angles assume striking varieties of perspective scenery. After the English Civil War much of the building was dismantled and the materials sold off. What survives is enough to give shape to the castle's footprint and enough to ground the visitor in the deep northern history that Penrith has often quietly accommodated. The trains still come and go from the platform opposite, modern travellers crossing paths with a building that watched the Wars of the Roses play out across the fells.