If I understand your question, you can't without deleting the categories, or editing the sidebar.php as suggested. But I doubt seriously that your category listings are affecting the page load. If you have a lot of photos on your blog, then I'd suggest cutting back on the number of posts that display per page.
