Write him a letter telling him exactly how you feel.
A person can't argue at a letter.
This way you can get everything out without forgetting anything.
Communication is the biggest factor when it comes to swinging.
So if you don't want to talk to him, that's a bigger problem than the actually swinging situation.
