The castle visible today was not the first at Framlingham. An earlier Norman structure was destroyed on Henry II's orders in 1174–76, after the Bigod family joined a revolt against the king. Henry's engineer, Alnoth, demolished the fortifications and filled in the moat at a recorded cost of £16 11s 12d. When Roger Bigod regained royal favor under Richard I and rebuilt the castle in the 1190s, he made a distinctive architectural choice: no central keep. Instead, the Inner Court is defended by a curtain wall of local flint and septaria stone, 10.5 meters high, protected by thirteen square mural towers each approximately 14.3 meters tall, with open backs. The design was innovative for its time, enabling flanking fire against attackers. It is also, scholars have noted, more interested in display than pure military function — the walls were meant to impress as much as to defend. The towers have wall-walks along the top, from which visitors today can look out over the mere and the surrounding parkland.

By the late 13th century, Framlingham was a luxurious home. The Bigods borrowed heavily — first from the Jewish community at Bungay, then from Italian merchants after the expulsion of the Jews — and lived accordingly. In twelve months between 1385 and 1386, the household spent over £1,000, including 28,567 gallons of ale and 70,321 loaves of bread. Wine came from France; spices arrived through London merchants from the Far East; venison was purchased from parks as far away as Northamptonshire. A vineyard operated within the castle grounds. The castle passed eventually to the Howard family, Dukes of Norfolk, who expanded it in fashionable Tudor brick, built pleasure gardens with ornamental ponds and terraced walkways, and added carved brick chimneys — purely ornamental ones that historian R. Allen Brown later described as 'regrettable' — around the Inner Court. When Mary Tudor claimed the throne from Framlingham, the Howard connection was central to her support. When the 4th Duke was executed for treason by Elizabeth I in 1572, the castle reverted to the Crown.

The castle's decline after the Tudors was steep. By 1589, a survey found the stonework, timber, and brickwork all in urgent need of maintenance. Catholic priests and recusants filled the prison from 1580 onward. In 1636, the castle was given to Pembroke College, Cambridge — with the condition that the college demolish the internal buildings and build a poorhouse within the walls. Three attempts were made over the following century; three workhouses failed, one after another, through mismanagement and decay. Meanwhile, the Great Hall was demolished, the chapel destroyed, and medieval stone heads were reset into the walls of the replacement building. The outer towers survived because nobody had the money or reason to pull them down. English Heritage manages the site today. The poorhouse building — the Red House and its later wings — still stands inside the Inner Court, a layered document of the castle's improbable afterlives.

Framlingham is the town where Ed Sheeran grew up. His 2017 single 'Castle on the Hill' takes its title and emotional landscape from this place — the castle visible from the surrounding roads as a destination, a landmark, a fixed point in a childhood geography. The song made Framlingham briefly famous in ways the Howard family and the Bigods could not have anticipated. The castle they built for power and display is now most widely known through a pop song about leaving and returning to Suffolk. The mere to the west, still owned by Framlingham College and managed by the Suffolk Wildlife Trust, reflects the towers in the morning. From the wall-walk, on a clear day, the landscape stretches south and east toward Woodbridge and the Deben estuary, the same view the earls and dukes of Norfolk looked out over across five centuries of Suffolk history.