That is because hit sounds probably don't get properly updated by the game when removed. Just like fpt and I believe some other items.
So those randoms that you bump into every now and then are people you've played against before and have your customs downloaded already. If they perform the same steps your hit sounds will also be gone for them.
Only reason to fully fix it for everybody is by uploading a default hit sound to your hit sounds before deactivating them. All sounds can be found in Toribash>Data>Sounds, I assume body_hit1 till body_hit5 are the default hit sounds.