Construction began in 1706. The Royal Governor of Virginia in the years that followed was a paper figure: George Hamilton, 1st Earl of Orkney, who collected the salary from London and is not known to have ever crossed the Atlantic. The man who actually lived in the half-finished Palace from 1710 onward was the Lieutenant Governor and acting governor, Alexander Spotswood. Spotswood took the building seriously. He kept improving it through around 1720-1722, adding the forecourt, laying out the formal gardens, and adding the decorative touches that earned the building its name. The colonial House of Burgesses complained about the expense more than once. The cost overruns became something of a running political quarrel between the assembly and the governors. By the time it was finished, the Palace was one of the grandest buildings in colonial America, an architectural argument that the Crown's representative in Virginia mattered.

For most of the 18th century the Palace served seven Royal Governors. Then the Revolution turned it into something stranger: the residence of the new Commonwealth of Virginia's first two governors. Patrick Henry, who had attacked the Stamp Act from the floor of the nearby Capitol, lived here from 1776 to 1779. Thomas Jefferson moved in after him and lived here from 1779 to 1780. Around 1779, Jefferson, never able to resist an architectural problem, drafted a proposal to remodel the Palace in the neoclassical style he was developing for Monticello, adding a temple-like portico to both front and back. He never built it. In 1780, citing security concerns during the Revolutionary War, he urged that the state capital be moved inland to Richmond. The General Assembly agreed. The governor's residence moved with it. The new lodging in Richmond, near the Virginia State Capitol, was much smaller and plainer; that building became the Executive Mansion that Virginia governors still occupy.

The Palace did not survive the year after Yorktown. In the fall of 1781, with Continental and French armies still treating wounds from the long siege, the building was pressed into service as a hospital. On December 22, fire took the main house. The cause is not certainly known; the records simply note the destruction. The brick outbuildings, the kitchen, smokehouse, scullery, and laundry that had supported the household, stood for another eighty years. Then the Civil War came. Occupying forces, North and South, dismantled the brick outbuildings for their building materials. By the war's end the Palace had been erased from the ground. In the 1880s the Chesapeake and Ohio Railway pushed temporary tracks down Duke of Gloucester Street to extend toward Newport News, and those rails passed directly through what had been the Palace grounds. The site was effectively forgotten.

The reconstruction came through W.A.R. Goodwin, the Bruton Parish rector who had also pulled the Capitol and Wren Building out of ruin with John D. Rockefeller Jr.'s money. The architects had more to work with than they might have expected. Excavations exposed the original foundations and cellar, with collapsed architectural fragments that told the story of the lost building from the inside out. Jefferson's drawings and plans from his unbuilt 1779 renovation had survived and revealed the interior layout. A delegation also located records at Badminton House in England, the seat of the Beaufort family, that documented Palace details otherwise lost. The architects deliberately set aside the Colonial Revival assumptions of the 1920s and chose to follow the evidence where it led. The result, completed in the early 1930s and opened to the public as part of Colonial Williamsburg, stands at the north end of Palace Green: the brick facade, the wrought-iron gates topped with the British lion and unicorn, the formal gardens with their boxwood maze, and the supply rooms hung with the muskets and swords that Spotswood used to make his visitors uneasy. It is a reconstruction, but it is a careful one, built where the original stood, on the original foundations, from the best evidence the 20th century could assemble.