The site has worn many shapes. The earliest castle here was a wooden motte-and-bailey raised around 1117 over a Carolingian settlement called Hrothusfeld, recorded in the Werden Abbey register in 889. That timber tower probably burned down after 1259 and was not rebuilt. A stone keep replaced it on the present site, an irregular rectangle of roughly nine meters on a side, built of rubble and lime mortar nearly two meters thick. By the late fourteenth century the castle had grown to thirty meters long and twelve wide, with corner towers added at the south and north. In 1597 fire destroyed the roof. Alexander I von Velen rebuilt the manor house from 1604 to 1606. His son went much further.

Alexander II von Velen joined the army at the outbreak of the Thirty Years' War. He fought for the Catholic League under Tilly and the Counts of Anholt, rose to General Constable of the Catholic League's forces in 1634, and was made Imperial Count in 1641. The Prince-Bishop Ferdinand summed up his fortune bluntly: 'The Count of Vele had a good war in Westphalia. He must have enjoyed a half million.' Between 1643 and 1658 Alexander II poured that money into Raesfeld. He hired the Capuchin architect Michael van Gent, then after van Gent left for Rome continued with Jacob and Johann Schmidt of Roermond. The bill came to roughly eighty thousand Reichstaler. The result was three new wings around the old manor house, a chapel, an outer bailey with the Sterndeuter or Stargazer Tower, and a vast hunting park.

The outer bailey is anchored at its southern end by a five-story tower called the Sterndeuter, the Stargazer. Local tradition says Alexander II used it for astrological studies, which would explain the name. The top floor is surrounded by a gallery; above that sits a Welsche dome capped by a lantern and, above the lantern, a smaller octagonal cupola repeating the form in miniature. The outer bailey itself is plainer than the upper castle, fitting its function as administrative and service rooms. A stone plaque set into the gateway in 1649 inscribed the history of the castle in Latin prose. The arms of Alexander II and his wife Alexandrine von Huyn und Gelen once hung over the entrance and are now displayed in the visitor center.

In 1653 Alexander II ordered the creation of a tiergarten, a hunting park, on a hundred hectares west of the castle. Workers raised an earthen rampart five kilometers long, topped with palisades, and stocked the enclosure with wild boar, roe deer and red deer. Exotic animals followed. The oldest evidence of fallow deer in North Rhine-Westphalia comes from Raesfeld in 1664. In 1670 Prince John Maurice of Nassau-Siegen sent Alexander II 'an American pregnant buffalo cow, as Your Grace is a particular lover of foreign animals and the best.' The park survived as a working zoo into the eighteenth century, then was used only for forestry. A 1729 map rediscovered in the 1990s convinced the regional preservation office that Raesfeld held one of the oldest surviving Renaissance palace gardens in Germany. Today deer roam freely again across 130 hectares behind a modern fence.

The von Velen family died out in the male line in 1733. The castle passed to the Lords of Limburg-Styrum, then to Baron von Boyneburg-Bömelsberg, then through nineteenth-century neglect to outright collapse. Two of the four wings were demolished. Cossack troops chasing the French after the Battle of Leipzig quartered themselves here in the winter of 1813. After the Second World War, the Chambers of Crafts of North Rhine-Westphalia bought the ruins and restored them, and the castle now functions as their further education center. Couples have been able to marry here since 2007. In the family crypt below the chapel rests the leaden heart of Christoph Otto von Velen, who died in Brussels in 1733 and asked that his heart be carried home to Raesfeld. It still sits in its niche beside the choir.