Reverend Thomas Craven had studied at Miami University in Oxford, Ohio, and had seen what education could do. He was a Baptist minister, an abolitionist, and by the 1840s, a man with a plan. The community around Lancaster, Indiana, already had the raw materials: anti-slavery sentiment had run deep there since the 1830s. Local residents had formed the Neil's Creek Anti-Slavery Society in 1839 and the Neil's Creek Abolitionist Baptist Church in 1846. Lancaster sat along the Underground Railroad route that carried people fleeing slavery north from Madison on the Ohio River toward Indianapolis. Several families on the school's future board of trustees were active participants in that network. Craven proposed building a school that would educate everyone. The Hoyt-Whipple family helped organize the effort, Craven donated the land, and on November 27, 1848, the Eleutherian Institute held its first classes.

Craven's son, Reverend John G. Craven, served as the first teacher and principal. The early years were lean. A second teacher, John C. Thompson, lasted only one year before the school could no longer afford his salary. James and Lucy Nelson maintained the dormitory. Despite the financial strain, enrollment grew. By the mid-1850s, African American students comprised roughly a quarter to a third of the student body. In 1854, the school was renamed Eleutherian College and began offering college-level coursework, making it the second college west of the Allegheny Mountains to do so, after Oberlin College, and the first in Indiana to provide interracial education. By 1860, the college enrolled some 200 students, including approximately 50 African Americans, some of whom had traveled from the Deep South despite laws that should have made their very presence in Indiana impossible.

The building that still stands on College Hill was constructed between 1854 and 1856 in the Greek Revival style, a deliberate architectural choice echoing the democratic ideals of ancient Athens. The three-story stone structure features a square bell tower centered above a gable-fronted entrance, finished limestone windowsills and lintels contrasting with rough-hewn stone walls. A wooden pediment crowns the third-floor windows. The choice of stone was practical as well as symbolic: it was meant to endure. A dormitory was erected nearby around 1854, and a log cabin on the property served as John Craven's home. The campus was modest, but the building itself projected permanence, as if the founders wanted to say that their experiment in equality was not temporary.

The Civil War changed everything. After 1861, no Black students enrolled at Eleutherian. Some former students enlisted in the 6th Indiana Infantry Regiment and returned to campus for military training on the college grounds. The buildings hosted meetings and concerts during the war years. When John Craven left in 1861, the school passed through a series of principals but never regained its momentum. The college closed in 1874. Craven returned in 1878 to revive the building as a private high school and teachers' training school, staying until 1887. The Lancaster township trustees bought the main building in 1888 for use as a public school, which served the community until 1938. After that, the building sat largely forgotten for decades.

In 1973, Historic Madison, a Jefferson County preservation group, received the school as a gift and sold it to new owners in 1990 who formed Historic Eleutherian College Inc. in 1996. The restoration returned the building to its 1850s appearance, and it now serves as a local history museum. The school was listed on the National Register of Historic Places in 1993 and declared a National Historic Landmark in 1997. In 2004, Indiana installed a state historical marker on the grounds. The college's legacy lives in its alumni: Moses Broyles, a formerly enslaved man from Kentucky who purchased his own freedom, became pastor of the Second Baptist Church in Indianapolis. Sarah B. Hoyt studied at Oberlin and became a high school principal. Rebecca J. Thompson chaired the mathematics department at Franklin College for thirty-eight years. These were the students that the word eleutheros was coined for, the ones who proved that freedom and equality were not abstractions but achievements.