I do not see why you would have to leave Toribash at all. As you said, I am probably the only person that would care at all about you leaving, and I do care. There is in my mind, nothing that makes you a bad person or an 'asshole', as I have had a lot of fun talking to you and playing TB with you, and wish to continue doing those very things. You are a genuinely nice person and I have actually had a great opportunity in meeting you.
But in the end of it all, it is completely your decision as to whether you quit this game or not, and your decision is final. Only do what you think is right for you to do, and do not leave just because one or two people think badly of you - in this community the good outweigh the bad, and you are one of the good, even best additions to this community I have seen for a damn long time.
If you do choose to leave, then I can only support that decision and wish the best for you in your life outside this community, and hope that you know that you are always welcome back if you choose to return.