A caveat: This is all just from my own personal understanding and experience. I'm nit any kind of expert on thismstuff and I could easily be disproven by others whom know a lot more.
Okay first off, sorry that guy was a dick to you. You didn't deserve that, and he was probably scared.
Second, yeah there are plenty of things you can do. First off, to understand why this is happening. Shadow men are just a natural pitfall of this world, they're around all the time just like pickpockets and germs, and all you can do is to make yourself less and less vulnerable. Just like bullies or any other predator, these guys are going to pick on you because you're making it rewarding for them, albeit accidentally.
So what you can do is, first, know that they can't actually harm you. In the worst worst case it coukd ever get no worse than a small scratch or one bruise, of which I'm sure you've had plenty and it's no big deal by now. What they CAN do that's dangerous is to trick you into hurting yourself, either by running around in the dark like a maniac or by keeping you from sleeping. But honestly both of these things never happen in more than 90% of cases.
Almost always they just want to cause fear and suffering to you.
So you need to just become a different kind of person who knows he no longer needs to be afraid of them. What helped for me was to learn the limits of their power, of which you already have some idea, and to learn spiritual defense methods, which even if they are BS, made me feel confident enough not to be a target anymore. I would recommend learning how to create a Merkaba (just search the term) and understanding how to send energy around to each of your chakras to sort of "strengthen your spine" and make yourself stand taller.