On February 4, 1819, a group of Romney residents adopted a constitution for what they called the Polemic Society of Romney. It became the Romney Literary Society - the first organization of its kind in what is now West Virginia and one of the earliest in the United States. The library began with two books. By 1861 it had grown to roughly 3,000 volumes. In 1820 the society pushed Romney Academy to add classical languages to its curriculum, effectively making it the first institution of higher education in what would become West Virginia's Eastern Panhandle. In 1846 the society built a campus building that housed the new Romney Classical Institute and its library. For four decades, the intellectual life of an isolated mountain town centered on a small library run by volunteers.

When the Civil War reached Romney, the town changed hands repeatedly - by some counts more than fifty times - between Union and Confederate forces. The Romney Classical Institute and its library sat right in the path of every passing army. Union troops occupied the building, removed the books, used pages for fires and bedding, and scattered or destroyed the rest. Many of the society's members were Confederate soldiers killed in combat. When the war ended in 1865, the survivors found that only 400 volumes could be recovered, of which only 200 were still suitable for the library shelves. The community that had spent forty years building an Appalachian library had to start over.

On May 15, 1869, the Romney Literary Society reorganized. The members decided to transfer the old Classical Institute campus to the West Virginia Schools for the Deaf and Blind, which still occupies it today, and to construct a new brick hall in downtown Romney for the society and its library. Construction took 1869 and 1870. The new Literary Hall stood at the intersection of North High Street and West Main Street, across from the Hampshire County Courthouse. The society rebuilt its library with 700 recovered or newly purchased volumes. For a decade between 1870 and 1880 the new building was the center of Romney's intellectual life. Then, as the original membership aged, interest dwindled. The society held its last recorded meeting on February 15, 1886.

After 1886 Literary Hall passed to the Clinton Lodge of Ancient Free and Accepted Masons - the first Masonic lodge chartered in Hampshire County, going back to a Grand Lodge of Virginia charter on December 13, 1825. The Masons met in the building, alongside the Order of the Eastern Star, for most of a century. From 1937 to the early 1940s the building also housed a New Deal era community library, with hours expanded to five afternoons a week through funding from the National Youth Administration and the Works Progress Administration. In 1973 the lodge was preparing to demolish the building when Romney lawyer Ralph Haines stepped in. He bought Literary Hall from the lodge, provided the Masons with land for a new temple at Washington and Center Streets, restored the old building to its original condition, and used it as his law office and a private museum. Literary Hall was added to the National Register of Historic Places on May 29, 1979 - 110 years after construction had begun.