A huge building that seems out of place in the rolling countryside of the Northeast Kingdom, the Old Stone House has a long and treasured history. It was built between 1834 and 1836 by Alexander Lucius Twilight, famous for being the first man of African-American descent to receive a bachelor's degree from an American institution of higher learning (Middlebury College in Vermont). An educator himself, Twilight was hired as principal of the Orleans County Grammar School in Brownington. He expanded the teaching facilities of what was also called the Brownington Academy, which was so successful that it started to attract students from other New England states as well as eastern Canada. Receiving no help from local authorities, Twilight designed and raised the funds to build this structure to serve as a dormitory for his students; he named it Athenian Hall.

The oldest granite public building in the state, it is now called the Old Stone House, and is the centerpiece of a museum of the same name that is run by the Brownington Historic District. Twilight would later win a seat in Vermont's state legislature, making him the first man of African-American descent to be elected to a state assembly in the United States.