William de Warenne, the Earl of Surrey, came to England with William the Conqueror in 1066 and was rewarded with estates across the country. Castle Acre was one of them — already an Anglo-Saxon estate center, its previous owner Toki rapidly replaced by Frederick, a Flemish lord and William's brother-in-law. When Frederick died around 1070, William acquired the manor. During the 1070s, he built a motte-and-bailey castle at the strategic point where the River Nar met the Peddars Way, placing a grand double-hall of stone — more country house than fortress — in the center of the inner bailey. He also gave the local church to the Cluniac order; his son the second earl gave the monks land to the west where Castle Acre Priory was eventually built, its construction spanning from roughly 1090 to the 1160s.

Civil war in England in 1135 prompted the third earl, also called William, to improve the castle's defenses considerably. He raised the earthworks, began converting the double-hall into a tall square keep, and replaced the timber ramparts of the inner bailey with stone walls. Then Hamelin de Warenne acquired the castle through his marriage to Isabel de Warenne in 1164 and brought the building work to a halt — the keep was never completed — but finished three large stone gatehouses in the castle and town. One of these gatehouses, the town's bailey gate, survives intact, its two drum towers and portcullis groove still visible, roofless but otherwise largely complete. Historic England describes the castle's earthworks as "among the finest surviving in England." The inner bailey earthwork alone rises up to 9.5 meters high.

The de Warennes held Castle Acre until 1347, and during that time the castle attracted the attention of English royalty. Henry III visited at least four times; Edward I five times. The seventh earl, John de Warenne, complicated things considerably: married to Joan, the niece of Edward II, he chose instead to live with his mistress Maud Nereford, a situation that drew threats of excommunication. In 1316, trying to arrange an annulment, he gave Castle Acre to the Earl of Pembroke as a diplomatic inducement. The castle passed eventually to Richard Fitzalan, Earl of Arundel, in 1347. By 1397, the fortifications were in ruins. Sir Edward Coke carried out repairs in the early 17th century at a cost of £60 — a modest investment for a site of this significance. The castle then passed through the hands of the earls of Leicester, who allowed the walls to be robbed of their stone by neighbors. An antiquarian noted in 1857 that "every house in the neighbourhood has some of the stone-work of the castle...in its walls."

Castle Acre was not just a castle — it was a planned settlement, 225 meters across and covering 4.25 hectares, surrounded by its own circuit of earthwork defenses and walls. The village that grew around the bailey gate was effectively dependent on the castle; historians have compared the arrangement to an enormous outer bailey rather than a conventional medieval town. The Peddars Way's diversion ensured the settlement had traffic. When the priory closed in the Dissolution of the Monasteries and the castle was abandoned, the town's prosperity faded with both. Today the castle and the bailey gate are managed by English Heritage and open to visitors. The castle's earthworks, the bailey gate with its twin drum towers, and the planned street pattern of the village around it — all of it survived to the present, because earthworks are harder to rob for building stone than walls.