"As they had done with the Swastika, the Nazis simply adopted the Totenkopf from the historical tradition and used it for their own purposes, leaving it marked with a stigma that has continued to the present." <-- This is also from wikipedia perhaps the special interest groups out there would like to supress this historical fact as well?
http://en.wikipedia.org/wiki/Totenkopf
This is a very important factor here, It is not a nazi symbol at all, it is just as harmless as the Iron Cross that appears on a huge percent of clothing worn today by men and women alike such as west coast choppers et all.
If you like the shirt , wear it with pride and do not let any special interest groups or individuals push you to do otherwise.