Grover worked on the Erie Canal as a youth and then as a store clerk at age fifteen. He went on to teach at the New York Institute for the Blind, where his older brother William taught and then served as principal. Grover wanted to attend college but did not have the funds.