I have to agree with Paul and others, if its something personal about you, then politely pass them on to someone else (find a suitable replacement ahead of time if you do this). If its something about your training style, then try to modify how you approach the client during sessions.
Do look upon this as a learning experience and not something that is bad. Personal training means interacting with the public. Ask yourself “what can I learn from this?” If you look at it this way, you take yourself out of the situation and are better able to discuss the issue with your client