The Waldomore went up in 1839 on a four-acre tract that ran from Pike Street down to Elk Creek. Waldo P. Goff, born in 1796, was the fifth son in a New York family that had migrated to Harrison County in 1804. He became a lawyer and politician of the kind that filled the early statehouses of antebellum Virginia - a Virginia State Senator from 1833 to 1837, sheriff of Harrison County in 1851, and an officeholder of one local kind or another for much of his adult life. Three of his brothers also served in the legislatures of Virginia and, later, West Virginia. Waldo and Harriet raised their family in the new house. Their most famous son, Nathan Goff Jr., was born in the second-floor rooms on February 9, 1843.

Nathan Goff Jr. went on to serve as Secretary of the Navy under President Rutherford B. Hayes - one of the few West Virginians to hold a Cabinet post in the 19th century. He ran for governor of West Virginia in 1876 and lost to Henry Mathews. He served in the U.S. House of Representatives from 1883 to 1889 and later in the U.S. Senate. In 1904, he built the seven-story Waldo Hotel directly across the street from the Waldomore - a deliberate gesture by a man who had grown up in one house and wanted to dominate the city block with another. From the upper floors of the Waldo Hotel, the family home where Nathan had been born would have looked small. From the front porch of the Waldomore, the hotel would have looked enormous. Both buildings still stand.

In 1930, May Goff Lowndes - a Goff descendant - donated the Waldomore to the City of Clarksburg. The gift came with a single binding condition: the building could only be used as a public library or museum, and for no other purpose. The city accepted the terms. From 1931 to 1976, the Waldomore was Clarksburg's public library. The Neo-Classical proportions that had once framed family dinners now framed circulation desks and reading tables. In 1976 a new, larger library was built next door on the same property, and the Waldomore was repurposed once again - this time as a special-collections building under the library's umbrella. It was listed on the National Register of Historic Places on October 4, 1978.

Today the Waldomore houses materials relating to West Virginia culture and history, a collection for genealogical research, and the personal papers of Gray Barker. Barker was a Clarksburg-area writer who turned a 1952 flying saucer sighting into a career investigating UFO reports across the United States. His 1956 book "They Knew Too Much About Flying Saucers" introduced the phrase Men in Black to American popular culture - the menacing, dark-suited government agents who, in Barker's account, intimidated UFO witnesses into silence. The story spawned movies, comic books, and decades of conspiracy folklore. Barker's papers - correspondence, photographs, manuscripts, files on cases he investigated - landed at the Waldomore after his death in 1984. A Neo-Classical mansion, a Goff family library, and the archive of a UFO writer turn out to share the same address.