Okay, this one's been bugging me for a long time. Staring is not a form of assault.
The concept of assault rests on the basis that people have an inherent right to control what goes on with their own bodies, and that doing anything to another person's body without their consent is wrong. Rights conflict in the real world, which is why we need to establish rules. The classic example is "your right to swing your fist ends at my nose".
I have a right to control my own body, and that includes what direction my eyes face. Since my eyes are devices for detecting radiation rather than for emitting it, what direction my eyes face impacts no one. In the conflict between my right to point my eyes where I please, and your right not to have someone point their eyes in your direction, the right to control one's own body trumps the right to force someone else to do something with their body you don't like.
I don't live in a society where we worship god-kings who can demand I downcast my gaze when they grace my presence. I will not live in such a society.
The last time someone tried to treat my gaze as an assault, I was quite ready to meet their explicitly threatened violence with violence of my own. I have no regrets about my actions in that incident.