The decision to leave a financial adviser really depends upon each individual. Some people look at the annual return of their portfolio to make this determination. Others feel the financial planner’s advice is too risky or too complex, or they feel he is just moving money around to make commissions on investment trades or sales.