The original Fort George stood in Inverness itself, on a hillock beside the River Ness incorporating portions of a medieval castle dating back to Malcolm III in the late 11th century. General George Wade had completed it in 1726 to house 400 troops. During the 1745 rising, three thousand Jacobites besieged the fort, and after mines reached the outer walls, the garrison surrendered in February 1746. The victorious Jacobites promptly blew the place up. After the Crown's decisive victory at Culloden two months later, Colonel William Skinner received a contract to build a replacement -- but this time, the government was not going to take chances. The new fort would be impregnable, positioned where it could control the sea approach to Inverness, and supplied by its own harbour.

Construction began in 1748 with the Adam brothers -- John, Robert, and later James -- acting as contractors, overseeing roughly a thousand soldiers who provided both labour and site defence. The original budget was set at a precise 92,673 pounds, 19 shillings, and one penny. The final cost exceeded 200,000 pounds, a vast sum for the era. The fortifications are a textbook example of defence in depth: stone-faced walls, angled and faceted with projecting bastions and redoubts so that every wall face is covered by fire from guns atop other walls. The walls sit on barrel-vaulted casemates forming underground bunkers capable of sheltering the entire garrison. Approaching from the landward side, an attacker would cross loose shingle, then face a ravelin, a raised walkway with drawbridge, and finally a wide ditch forming a killing ground exposed to gunfire from multiple angles.

Fort George was completed in 1769 and has served continuously as a military garrison ever since. Following the Childers Reforms of 1881, it became the depot of the Seaforth Highlanders. In 1961, the Queen's Own Highlanders formed their depot here, and in 2007 the barracks became home to the Black Watch, 3rd Battalion, Royal Regiment of Scotland. The Highlanders' Museum in the former Lieutenant Governors' House displays uniforms, weapons, medals, and the poignant First World War memorial plaques known as "death pennies." In November 2016, the Ministry of Defence announced the site would close in 2032, with Defence Minister Sir Michael Fallon offering the dryly historical observation that it was no longer needed because "the Highland rebellions are over." The closure has since been delayed to 2033.

Much of the site is open to the public through Historic Environment Scotland, which uses part of the barracks to display reconstructions of life in the fort's early decades. The Grand Magazine houses the Seafield Collection of Arms and serves as a stage for actors recreating the stories of 18th-century soldiers. The regimental chapel is open to visitors and features many regimental colours and memorials. The site received 76,169 visitors in 2019. Standing on the ramparts looking out over the Moray Firth, with dolphins sometimes visible in the water below, it is possible to appreciate the paradox of the place: a fortress designed for violence that has spent its entire existence in peace.