The historian John Stow, walking London in the 1590s for his Survey of London, described what was then a memory: "a fair and large library, furnished with books, pertaining to the Guildhall and college." Stow recorded what had happened to it. During the reign of Edward VI, around 1549, the entire collection was "sent for" by Edward Seymour, Duke of Somerset and Lord Protector of England. His agents loaded the books onto carts and took them away to furnish Somerset House, the duke's new palace on the Strand. They never came back. By 1550 the empty library building had been let to Sir John Aylif, surgeon to Henry VIII, as a market house for the sale of clothes. The first London public library had become a clothes warehouse. Only one book from Whittington's original collection has ever returned to Guildhall: a thirteenth-century manuscript of Petrus Riga's Aurora, a metrical Latin version of the Bible, purchased from an antiquarian dealer centuries after the looting.

Three centuries passed before the City of London Corporation tried again. In 1824 they appointed a committee to consider "a Library of all matters relating to this City, the Borough of Southwark, and the County of Middlesex," funded out of the Privy Purse rather than the rates. The library opened in 1828 with just 1,700 volumes, available only to members of the Corporation. The collection grew. Membership widened. By the late 1860s the library had outgrown its space, and the Corporation made an unusual decision: the new building would be open to the general public, free of charge. Horace Jones, the City Architect, designed the new library in Perpendicular Gothic style to complement the medieval Guildhall next door. Construction ran from 1868 to 1872, and the doors opened to the public in 1873. By then the collection numbered 60,000 volumes, covering London's history, architecture, topography, and early printed plays from the Chapman bequest. Some of those old books still carry shelf marks like "Bay A" and "Bay H," survivors of the old library's classification scheme.

On the night of 29 to 30 December 1940, German incendiary bombs fell across the City of London. The Blitz that night destroyed dozens of churches, livery halls, and warehouses. At Guildhall Library, around 25,000 volumes were lost when some of the storerooms burned. The library building itself, remarkably, was not extensively damaged. After the war, the library acquired extra stack space in the Guildhall crypt and continued operating. As part of the post-war reconstruction scheme, the firm of Sir Giles Scott, Son and Partners was commissioned to design a wholly new library. The present Guildhall Library, in the West Wing of the Guildhall complex, opened on 21 October 1974. It took seven weeks to wheel many trolley loads of books to their new home, legally this time. A Country Life magazine writer suggested it might be the most efficient machine for information retrieval in the world. But the librarians kept the old pneumatic tube system: nothing else moved a slip of paper across the building faster.

The Guildhall Library now holds what may be the deepest collection of London-specific reference material anywhere. There is a complete run of the London Gazette from 1665 to the present, capturing royal proclamations, bankruptcy notices, and military dispatches across 360 years of British history. There are 18th-century poll books from before the secret ballot, listing how every named voter voted in their constituency. A complete set of House of Commons papers from 1740 covers parliamentary business across nearly three centuries. The local and trade directories run from 1677, an unrivalled run that lets researchers trace London businesses generation by generation. Five major archive collections live here: those of the livery companies, the Stock Exchange, Christ's Hospital, St Paul's Cathedral, and the Lloyd's Marine Collection. The library also holds specialist collections on Sir Thomas More, Charles Lamb, John Wilkes, Samuel Pepys, and the history of horology. The combined holdings of the Antiquarian Horological Society, the Worshipful Company of Clockmakers, and the Corporation's own collections give the library an international standing in the literature of clocks.

In 1926 the librarians devised the London Classification, a bespoke system for arranging material about the city, and that scheme is still used today, including by the Barbican Library's London Collection for items available to borrow. In 2009 and 2010, the Prints and Maps and Manuscripts collections moved to the London Metropolitan Archives, but many major manuscript holdings remain at Guildhall. Until 2023, the library shared its building with the City Business Library, which had originally been the Commercial Reference Room within Guildhall itself before moving out in 1970. The City Business Library, now the Small Business Research and Enterprise Centre, moved again to Basinghall Street in February 2023. Guildhall Library remains a public reference library, free to use. Anyone can walk in, sit down, and ask for a 17th-century pamphlet on a political controversy, a trade directory listing a chandler in 1740, or a livery company's records of who joined when. Six centuries after Whittington left the money, the library still does what he hoped a London library might do.