As with George Washington’s father, Augustine, James Madison Sr. was a prominent tobacco planter. He owned a large plantation named Montpelier. He also served as a colonel in the Virginia militia.