I am guessing either you look like a tramp or this wasn't a particularly high class establishment.
The guy shouldn't have been rude but you have to see it from their point of view also. They probbaly get stiffed often, and maybe it's the staff who get penalised when this happens.
Whether to go back depends upon whether you like the food. No other factor. Don't cut off your nose to spite your face.