Yester Castle was built in the 13th century by the Norman family of Giffard, who had been granted the barony of Yester by the Scottish crown. The first Hugo de Giffard's grandson Hugh, the supposed magician, built the underground chamber that survives in the castle ruins. It is reached by a steep flight of stone steps and opens into a long vaulted hall, lit only by a single small opening at the far end. Stone carvers in the 13th century did not waste effort on rooms that did not need to look grand. Why a remote border laird would have built such a chamber, and why the legend of supernatural builders attached to it, are questions historians and folklorists have argued about for generations. Walter Scott had read both the historical record and the folk tradition before he wove the Goblin Hall into Marmion in 1808, his epic poem of the Battle of Flodden.

The modern village of Gifford grew up not around the castle but a few miles north of it, at the junction of the Colstoun Water (locally called Gifford Water) and the roads to Haddington and Garvald. Its central feature is the Mercat Cross of 1780, still standing in the village centre. The present parish church, named for the parish of Yester, was built in 1710, replacing earlier churches whose ruins lie in the woods near Yester House. Gifford Town Hall began as an early 18th-century private house. The Tweeddale Arms Hotel and the elegant 18th-century houses lining the Avenue give the village its distinctive character: a planned 18th-century estate village rather than an organic medieval settlement. For a brief period in the 18th century, the village had a paper mill that produced bank notes for the Bank of Scotland.

John Witherspoon was born in Gifford on 5 February 1723. He went to Haddington Grammar School, then to the University of Edinburgh where he gained a Master of Arts in 1739 at just sixteen, then to the Presbyterian ministry in Beith, Ayrshire. In 1768 the trustees of a small college in Princeton, New Jersey, sent him an invitation he initially refused and his wife flatly declined. The trustees persisted. Witherspoon eventually accepted and became the 6th president of what was then the College of New Jersey, today Princeton University. He transformed it from a regional Presbyterian seminary into one of the leading colleges in colonial America. When the Revolution came, Witherspoon went with it. He served in the Continental Congress as a representative from New Jersey, and on 4 July 1776 he signed the Declaration of Independence as the only ordained clergyman among the signers. His students at Princeton included James Madison, the fourth President of the United States, and many of the next generation of American leaders. He died in 1794, a Scottish minister buried in American soil, having shaped the founding of a country.

John Knox, the great Protestant reformer who broke the Scottish church from Rome and from the Marian government in the 1560s, was also born in or near Gifford around 1514 (dates between 1505 and 1515 have been proposed), though the exact site is disputed; most traditions place his birth in nearby Haddington, specifically the Giffordgate area or the Nungate. Either way, the early modern history of two countries was shaped by men born within a few miles of each other in this small corner of East Lothian. Other Gifford figures are less world-historical but worth knowing. Willie Wood, the double Commonwealth Games bowls gold medallist, learned the sport here and holds the record for competing in seven Commonwealth Games. The composer Gian Carlo Menotti, who twice won the Pulitzer Prize for Music, owned Yester House for many years until his death in 2007. The artist William Stewart MacGeorge, of the Kirkcudbright School, was born in Gifford in 1861.

The Gifford and Garvald Light Railway Company built a branch line to the village that opened in stages in the early 20th century. The line was intended to extend to Garvald in the Lammermuirs but never quite got there, and the Gifford section closed in 1947 after flooding washed away a bridge. The remaining line to East Saltoun lasted until the Beeching cuts of the 1960s. Today the village survives by being beautiful and being close enough to Edinburgh to attract residents who commute the 25 miles by car. Yester Primary School draws children from across the parish. Two golf courses, Gifford Golf Club and Castle Park, alternate hosting the annual Gifford Cup. In 2017 the community bought 55 acres of woodland north-west of the village with help from the Scottish Land Fund. Gifford Community Woodland, managed by a local charity, is now restoring native woodland for biodiversity, learning, and the pleasures of walking among trees whose roots reach back into the same hillside where, eight centuries ago, Hugh de Giffard was reputed to have summoned his goblins.