That was where John Brown had nearly started the war, two years earlier, and Harpers Ferry remained very important. Union men held it when the war began, but they were chased out by Virginia militia in the summer of 1861. Harpers Ferry would change hands so many times that it became, perhaps, the single most occupied place in the United States. It took the town, and its people, a long time to shake off the effects of the war.