Richard, while still Duke of Gloucester, had supervised the royal heralds for his brother Edward IV from 1469. He understood their disorder. By 1484 he had decided to formalise them into something permanent: the Garter King of Arms, the King of Arms of the Southern parts, the King of Arms of the Northern parts, the King of Arms of Wales, and all the heralds and pursuivants beneath them. The charter created their seal, ordered them to keep a chaplain saying daily mass for himself, his Queen Anne Neville, and his heir Edward of Middleham, and granted them Coldharbour, one of the largest houses in the City of London. Two years later Edward of Middleham was dead. Anne was dead. Richard was dead. Henry Tudor's first Parliament resumed the property grants, and Coldharbour was given to Henry's mother Lady Margaret Beaufort. The heralds, suddenly homeless, scattered their books and tried to survive.

The heralds had a politically dangerous habit: they proclaimed whichever monarch held London at any given moment. When Edward VI died in July 1553, two heralds with trumpets proclaimed Lady Jane Grey queen in Cheapside and Fleet Street. Days later, when popular support swung toward Mary Tudor, the same heralds returned to Cheapside with the Lord Mayor and proclaimed Mary instead. Their excuse was that the Duke of Northumberland had forced them. Mary, remarkably, accepted it. On 18 July 1555 at Hampton Court she signed a new charter granting them Derby Place, a house built in 1503 just off Queen Victoria Street, so they could "assemble together, and consult, and agree amongst themselves" and store their records safely. Four hundred and seventy years later the College is still on the same site.

In 1530 Henry VIII handed the College one of the strangest jobs in English administrative history. The provincial Kings of Arms were sent across the country, armed with royal warrants, to enter houses and churches and "deface and destroy" coats of arms used unlawfully by anyone calling himself a knight, esquire, or gentleman. After the Dissolution of the Monasteries, when the great repositories of local genealogies were broken up, this work became essential. The visitations were not gentle. Hundreds of people were charged and heavily fined. The visitors recorded thousands of legitimate pedigrees and arms in books that survive today. The last visitation was carried out in 1686. For 156 years, the College had operated something like a roving genealogical inquisition across the English shires.

The College's enemies have often been physical. In 1742 a sugar refinery was built against its wall, a constant fire risk that worried the heralds for nearly eighty years until they finally bought the offending building in 1820 for £1,500. In 1939 their records were evacuated to Thornbury Castle in Gloucestershire. On 10 and 11 May 1941, the night of the heaviest German air raid on the City, the College building was almost lost. All the buildings to its east on Queen Victoria Street were levelled. The flames came across the roofs. The College was given up for lost when, at the last possible moment, the wind changed direction and pushed the fire away. The walls survived; restoration after the war required public subscription and significant herald contributions from their own pockets. The gates now at the entrance arrived in 1956 from Goodrich Court in Herefordshire.

Thirteen officers in ordinary make up the modern College: three Kings of Arms, six Heralds, and four Pursuivants, all appointed by the Sovereign on the recommendation of the Earl Marshal, a hereditary office held by the Duke of Norfolk. They are still expected to lead the front of the procession at every State Opening of Parliament, every Garter Service at Windsor, every coronation. They alone, besides the King and Queen, may wear crowns at a coronation, which last happened at the crowning of Charles III in 2023. A new grant of personal arms, including a crest, costs £9,600 as of January 2026; a commercial company pays £29,560. The College is almost entirely self-financed, receiving no regular public funding. From a king's parting reform to a corporation that still drafts coats of arms by hand, the heralds have outlasted nearly every institution that ever threatened them.