In 20:14-16 we find Avimelech getting rid of Avraham and Sarah and giving them cattle, money, land, etc. Why did Avraham accept it? When S'dom offered him money Avraham refused it claiming people would think S'dom made him rich instead of Hashem. Why here was he not concerned people might say Avimelech made him rich rather than Hashem?
And if you want to differentiate and say S'dom was bad and Avimelech was not, so such an impression that his wealth came from evil sources is much worse, Avraham already said in passuk 11 that this is a place with no fear of heavan.. so what is the difference?
Maybe the answer comes in passuk 17. It says that Avraham prayed for Avimelech and his people.
This money is now money that was not given to Avraham. he was not made wealthy by Avimelech. he earned that money. he provided a service. He worked for his money.
Once he worked for the money, nobody can claim he was made wealthy or that he got it from improper sources. he earned it. Nobody can take away from you an honest days work.