The castle dates from 1384, when Michael de la Pole, 1st Earl of Suffolk, applied to the Crown for a royal licence to crenellate his manor house. By that point, the age of true military castles was largely over — kings were suspicious of private fortifications and rarely granted such licences unless the structure was more symbolic than strategic. The result is exactly that: a fortified manor house, its moat and towers providing prestige rather than genuine defence. The de la Pole family had come to Wingfield through Catherine Wingfield, daughter of Sir John de Wingfield, chief administrator to Edward the Black Prince. When Catherine married Michael de la Pole, she brought him the Wingfield estates and the connection to the Wingfield family name, from which the dukedom would eventually derive its prestige — if not always its stability.

Among the most extraordinary episodes in Wingfield's history is the long captivity of Charles, Duke of Orléans, who was captured at the Battle of Agincourt in 1415 and held — partly at Wingfield — for decades. His family offered ransom repeatedly. Henry V refused to allow his release, citing Charles's proximity to the French line of succession and the danger his freedom would represent. Charles spent over twenty years in English captivity, eventually returning to France in 1440. He became the father of Louis XII. During his captivity, he wrote poetry — he is remembered as one of the significant medieval French poets, composing in both French and English. The castle where this gifted, politically inconvenient man passed some of his long confinement is now a private house off a Suffolk lane, its moat reflecting the sky.

William de la Pole, 1st Duke of Suffolk, married Alice Chaucer — granddaughter of Geoffrey Chaucer — in 1430 and lived at Wingfield. The literary connection is improbable and entirely real. After William's death, Alice moved to Wallingford Castle. The dukedom continued through various de la Poles until Henry VIII revived and transferred it to his favourite Charles Brandon in 1514, a man who also had Wingfield blood through his great-grandfather. Brandon lived here occasionally with his third wife, Mary Tudor — sister of Henry VIII, briefly Queen of France, returned to England after her royal husband's death to marry Brandon in secret. When Brandon left for Lincolnshire in 1533, Wingfield passed out of the family. Centuries of subsequent owners left it in varying states of repair. In the 1980s, during a period of ownership changes, the castle attracted interest from several musicians considering purchase. Elton John viewed it. So did John Lydon, known as Johnny Rotten of the Sex Pistols. Neither bought it. A contemporary observer noted that 'the de la Poles would surely be interested to know that Wingfield Castle was echoing again to the beat of heavy metal.' The castle went to a London businessman instead, and then to a barrister. It is privately owned and not open to the public.