I fully understand where you're coming from but this is how I feel about it.
I photograph a lot of celebrities and pro athletes but only on their professional time and not their personal time. You can argue that since they are famous they're in the public eye and that opens them up to 24 / 7 access. I don't feel that way. If they're not working or doing a public event etc I think they should be left alone to do whatever they wish in their personal lives. Some celebrities like Britney Spears want and seek the attention while others like Julia Roberts don't. I would use discretion and have on many occasions.
For me the best way to understand was to mentally step into their shoes for awhile and see how long it would take me to get annoyed. It's not part of my nature to do something to someone else that I'm not willing to do to myself.
Paparazzi may have the right to photograph them but that doesn't make it right
