The name dates from the sixth century, when an Irish monk known as Saint Munn, also called Fintan of Taghmon, came across the sea from Ireland and founded a church at the head of the loch. The Gaelic name for the village that grew up there is Cill Mhunnu, the church of St Munn. An early carved stone at Kilmun suggests a place of worship as far back as the sixth or seventh century; the visible remains of a church on the site date from the twelfth. By the fifteenth century Kilmun was significant enough as a local centre of Christianity that the adjacent loch took the name Holy Loch, and the rising Clan Campbell adopted it as their spiritual home. Kilmun Parish Church still stands where Saint Munn's church is said to have been.

On the southern shore, in the village of Sandbank, Alexander Robertson began repairing boats in 1876 in a small workshop. Alexander Robertson and Sons grew into one of the foremost wooden boat builders on the Clyde. The yard's golden years were in the early twentieth century when it built classic 12 and 15 metre racing yachts, including Shimna, the first 15-metre yacht designed by the great William Fife in 1907. The yard built more than 55 boats in the run-up to the First World War and remained busy through the Great Depression. During the Second World War it turned out high-speed Fairmile motor boats for the Admiralty. After the war it built two challengers for the America's Cup: Sceptre in 1958 and Sovereign in 1964. Over 104 years Robertson's yard built 482 numbered boats, many still sailing. The site is now the Holy Loch Marina.

In March 1959, President Dwight D. Eisenhower mentioned to Prime Minister Harold Macmillan, at a meeting at Camp David, that the United States needed an overseas base for refitting submarine-launched ballistic missiles. Negotiations with the British government followed. Holy Loch was selected over alternatives at Faslane, Largs, Rosneath, and Rothesay, partly because of its sheltered anchorage and partly because Glasgow airport was within reach. Agreement was reached at the end of 1960. The arrival of the first submarine tender, originally planned for December, was rescheduled to 3 March 1961 because of British political divisions and expected Campaign for Nuclear Disarmament protests. For thirty-one years the loch was home to US Submarine Squadron 14. The shore command leased 342 housing units for personnel and their dependents. At its peak around 3,000 Americans lived in the area.

Two events during the deployment stand out. On 29 November 1970 a fire broke out on the tender USS Canopus, killing three of her complement. Four years later, on 3 November 1974, the nuclear ballistic missile submarine USS James Madison collided with a Soviet submarine, believed to be a Victor-class attack boat, during a dive just after leaving Holy Loch. The Madison was dented and scraped along nine feet of her hull, and spent a week back at the loch for inspection. Among the medical officers who served at Submarine Squadron 14 was Laurel Clark, known to her shipmates as Doc Salton, who later became a NASA astronaut and was one of the seven killed in the Space Shuttle Columbia disaster on 1 February 2003. The end of the Cold War made the base unnecessary. The last submarine tender left in November 1991, the base closed the following June, and the closure caused significant economic decline in Dunoon and the surrounding villages.