SiteMap Generator

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • OzMan84
    Confirmed User
    • Nov 2003
    • 4740

    #1

    SiteMap Generator

    is there such thing or how do i make one manually?

    for SE purposes
    XferClick - 300,000 UV daily and growing quickly. Majority US, EU, CA and AU traffic. Contact me for more information
    ICQ: 304568058
  • StarkReality
    Confirmed User
    • May 2004
    • 4444

    #2
    http://www.auditmypc.com/xml-sitemap.asp is a free one for example (uses java) or just google for "sitemap generator", there are loads around.

    About sitemaps in general: Great for new sites, be careful with established sites since adding a sitemap to a website online for years will hurt your rankings without any real advantage

    Comment

    • wtfent
      Confirmed User
      • Nov 2003
      • 3790

      #3
      http://www.xml-sitemaps.com/
      ThisWillShockYou.com DVD Store - TWSY UNCENSORED
      ICQ# 194020367 E-mail: shockingbucks(AT)gmail.com
      Promote something different!! Shocking Bucks

      Comment

      • fallenmuffin
        Confirmed User
        • Nov 2005
        • 8170

        #4
        http://www.xml-sitemaps.com/
        http://www.rorweb.com/rormap.htm
        http://www.sitemapdoc.com/
        http://www.auditmypc.com/xml-sitemap.asp (Java, you've been warned)
        http://www.devintelligence.com/sitemapgenerator/ (Free desktop app)
        http://www.neuroticweb.com/recursos/sitemap/
        http://www.sitemapbuilder.net/

        And so many more...

        Comment

        • uno
          RIP Dodger. BEST.CAT.EVER
          • Dec 2002
          • 18450

          #5
          www.google.com

          search: google, sitemaps

          look for a result on google.com, bam.
          -uno
          icq: 111-914
          CrazyBabe.com - porn art
          MojoHost - For all your hosting needs, present and future. Tell them I sent ya!

          Comment

          • OzMan84
            Confirmed User
            • Nov 2003
            • 4740

            #6
            half my pages arent getting picked up with these scripts, any idea to why??
            XferClick - 300,000 UV daily and growing quickly. Majority US, EU, CA and AU traffic. Contact me for more information
            ICQ: 304568058

            Comment

            • georgeyw
              58008 53773
              • Jul 2005
              • 9865

              #7
              google has one and it's not really all that friendly either...
              TripleXPrint on Megan Fox
              "I would STILL suck her pussy until her face caved in. And then blow her up and do it again!"

              Comment

              • DamageX
                Marketing & Strategy
                • Jun 2001
                • 14293

                #8
                Originally posted by StarkReality
                be careful with established sites since adding a sitemap to a website online for years will hurt your rankings without any real advantage
                Elaborate, please. Doesn't make much sense to me, without further info on this.
                Whitehat is for chumps

                If you don't do it, somebody else will - true story!

                Comment

                • munki
                  Do Fun Shit.
                  • Dec 2004
                  • 13393

                  #9
                  http://www.download.com/3001-2379_4-10419602.html

                  SiteMapperBuilder.net is a personal fave of mine.

                  I have the simplest tastes. I am always satisfied with the best.” -Oscar Wilde

                  Comment

                  • GrouchyAdmin
                    Now choke yourself!
                    • Apr 2006
                    • 12085

                    #10
                    Just so you know, Google will also accept RSS feeds in lieu of a generator; it's not as good for granularity, but it does work quite well for your basic blog.

                    Comment

                    • GeorgeM
                      Confirmed User
                      • Apr 2006
                      • 4969

                      #11
                      thx for links

                      http://www.devintelligence.com/sitemapgenerator/

                      Comment

                      • StarkReality
                        Confirmed User
                        • May 2004
                        • 4444

                        #12
                        Originally posted by DamageX
                        Elaborate, please. Doesn't make much sense to me, without further info on this.
                        Just very painful personal experiences. Sitemaps are just good for getting a large site crawled quicker, plus they avoid pages getting lost due to a bad/broken linking structure, but there are several disadvantages:

                        1. You need to use real 404 pages, redirects aren't possible any more since you need to report a 404 and not a 200.

                        2. Sitemaps do not help your site except for being crawled better. If you have an established well linked site, there is no need for a sitemap at all.

                        3. Adding sitemaps to your google account allows google to see which sites you are running, even if you use different IPs. Since almost all adult sites are "thin affiliates" by google's terms, getting one of them banned would mean you get the others kicked as well.

                        I lost a part of a network because of this. One site was kicked, all other sites in the google account followed. Those not added in the account stayed in google. Bad coincidence ? I don't think so, 11 sites from the network were kicked, 23 stayed and the 11 were those in the google account.

                        4. By giving pages a certain importance, your suggestions compete with google's internal evaluation. It really messes with your rankings, seen it happen more than once.

                        Don't get me wrong, if you want to use a old style html sitemap, fine, but the XML ones are a big NO NO except on a new or badly linked site. If a site is built and linked well, there are zero advantages of a xml sitemap. So, why use one ?

                        Comment

                        • fris
                          I have to go potty
                          • Aug 2002
                          • 55792

                          #13
                          use googles webmaster tools
                          Since 1999: 69 Adult Industry awards for Best Hosting Company and professional excellence.


                          My Newest Theme

                          Comment

                          • DamageX
                            Marketing & Strategy
                            • Jun 2001
                            • 14293

                            #14
                            Originally posted by StarkReality
                            Just very painful personal experiences. Sitemaps are just good for getting a large site crawled quicker, plus they avoid pages getting lost due to a bad/broken linking structure, but there are several disadvantages:

                            1. You need to use real 404 pages, redirects aren't possible any more since you need to report a 404 and not a 200.

                            2. Sitemaps do not help your site except for being crawled better. If you have an established well linked site, there is no need for a sitemap at all.

                            3. Adding sitemaps to your google account allows google to see which sites you are running, even if you use different IPs. Since almost all adult sites are "thin affiliates" by google's terms, getting one of them banned would mean you get the others kicked as well.

                            I lost a part of a network because of this. One site was kicked, all other sites in the google account followed. Those not added in the account stayed in google. Bad coincidence ? I don't think so, 11 sites from the network were kicked, 23 stayed and the 11 were those in the google account.

                            4. By giving pages a certain importance, your suggestions compete with google's internal evaluation. It really messes with your rankings, seen it happen more than once.

                            Don't get me wrong, if you want to use a old style html sitemap, fine, but the XML ones are a big NO NO except on a new or badly linked site. If a site is built and linked well, there are zero advantages of a xml sitemap. So, why use one ?
                            Thanks for taking the time to explain this. It makes more sense now.
                            Whitehat is for chumps

                            If you don't do it, somebody else will - true story!

                            Comment

                            • he-fox
                              Confirmed User
                              • Jun 2006
                              • 2884

                              #15
                              Originally posted by wtfent

                              Comment

                              • GooSearch
                                Confirmed User
                                • Oct 2006
                                • 2538

                                #16
                                Xenu's Link Sleuth
                                http://home.snafu.de/tilman/xenulink.html

                                kick ass free proggy
                                GooSearch Real Text Ads Without The Drama
                                Coming Back Bigger and Better Then Ever

                                Comment

                                • OzMan84
                                  Confirmed User
                                  • Nov 2003
                                  • 4740

                                  #17
                                  ok like i said up above, half my pages are not getting found with the crawling....any ideas?
                                  XferClick - 300,000 UV daily and growing quickly. Majority US, EU, CA and AU traffic. Contact me for more information
                                  ICQ: 304568058

                                  Comment

                                  • DamageX
                                    Marketing & Strategy
                                    • Jun 2001
                                    • 14293

                                    #18
                                    Originally posted by OzMan84
                                    ok like i said up above, half my pages are not getting found with the crawling....any ideas?
                                    Yeah, get more backlinks to the individual pages.
                                    Whitehat is for chumps

                                    If you don't do it, somebody else will - true story!

                                    Comment

                                    • GooSearch
                                      Confirmed User
                                      • Oct 2006
                                      • 2538

                                      #19
                                      Originally posted by OzMan84
                                      ok like i said up above, half my pages are not getting found with the crawling....any ideas?
                                      html site map linked from your index page.. hardlinks to your sitemap.. and then u'll see a more complete spider./.. works for me
                                      GooSearch Real Text Ads Without The Drama
                                      Coming Back Bigger and Better Then Ever

                                      Comment

                                      • mattz
                                        Confirmed User
                                        • Dec 2001
                                        • 7697

                                        #20
                                        Originally posted by StarkReality
                                        http://www.auditmypc.com/xml-sitemap.asp is a free one for example (uses java) or just google for "sitemap generator", there are loads around.

                                        About sitemaps in general: Great for new sites, be careful with established sites since adding a sitemap to a website online for years will hurt your rankings without any real advantage
                                        Thanks for the tip

                                        Comment

                                        • martinsc
                                          Too lazy to set a custom title
                                          • Jun 2005
                                          • 27043

                                          #21
                                          Originally posted by StarkReality
                                          Just very painful personal experiences. Sitemaps are just good for getting a large site crawled quicker, plus they avoid pages getting lost due to a bad/broken linking structure, but there are several disadvantages:

                                          1. You need to use real 404 pages, redirects aren't possible any more since you need to report a 404 and not a 200.

                                          2. Sitemaps do not help your site except for being crawled better. If you have an established well linked site, there is no need for a sitemap at all.

                                          3. Adding sitemaps to your google account allows google to see which sites you are running, even if you use different IPs. Since almost all adult sites are "thin affiliates" by google's terms, getting one of them banned would mean you get the others kicked as well.

                                          I lost a part of a network because of this. One site was kicked, all other sites in the google account followed. Those not added in the account stayed in google. Bad coincidence ? I don't think so, 11 sites from the network were kicked, 23 stayed and the 11 were those in the google account.

                                          4. By giving pages a certain importance, your suggestions compete with google's internal evaluation. It really messes with your rankings, seen it happen more than once.

                                          Don't get me wrong, if you want to use a old style html sitemap, fine, but the XML ones are a big NO NO except on a new or badly linked site. If a site is built and linked well, there are zero advantages of a xml sitemap. So, why use one ?
                                          thanks
                                          Make Money

                                          Comment

                                          • OzMan84
                                            Confirmed User
                                            • Nov 2003
                                            • 4740

                                            #22
                                            Originally posted by GooSearch
                                            html site map linked from your index page.. hardlinks to your sitemap.. and then u'll see a more complete spider./.. works for me
                                            so make a nice little site map linking to all 150+ pages correct?
                                            XferClick - 300,000 UV daily and growing quickly. Majority US, EU, CA and AU traffic. Contact me for more information
                                            ICQ: 304568058

                                            Comment

                                            • OzMan84
                                              Confirmed User
                                              • Nov 2003
                                              • 4740

                                              #23
                                              Originally posted by OzMan84
                                              so make a nice little site map linking to all 150+ pages correct?
                                              bump bump just need to know if i was on the right track with that
                                              XferClick - 300,000 UV daily and growing quickly. Majority US, EU, CA and AU traffic. Contact me for more information
                                              ICQ: 304568058

                                              Comment

                                              • OzMan84
                                                Confirmed User
                                                • Nov 2003
                                                • 4740

                                                #24
                                                http://www.hqerosescorts.com/sitemap.htm

                                                is that what you guys ment?
                                                XferClick - 300,000 UV daily and growing quickly. Majority US, EU, CA and AU traffic. Contact me for more information
                                                ICQ: 304568058

                                                Comment

                                                • Empress_JoinRightNow
                                                  Confirmed User
                                                  • Aug 2005
                                                  • 1207

                                                  #25
                                                  google sitemaps are more effective and it's automatic...

                                                  Comment

                                                  • bonkerz2007
                                                    Confirmed User
                                                    • Sep 2005
                                                    • 794

                                                    #26
                                                    Create two files:
                                                    sitemap.php
                                                    .htaccess

                                                    sitemap.php =

                                                    Code:
                                                    <?php
                                                    
                                                    #To ignore #404 Page Not Found error, and do not include them in sitemap
                                                    error_reporting(0);
                                                    
                                                    # The script is free of charge. 
                                                    # Mandatory disclaimer: THIS SCRIPT CARRIES NO WARRANTY OR GUARRANTEE WHAT SO EVER. USE IS AT YOUR OWN RISK.
                                                    
                                                    # The site url to crawl. Remember to include the slash ( / ) at the end. (EX: http://www.site.com/)
                                                    $siteurl = 'http://'.$_SERVER["HTTP_HOST"].'/';
                                                    
                                                    # The frequency of updating. Valid settings are: always, hourly, daily, weekly, monthly, yearly, never
                                                    # The value "always" should be used to describe documents that change each time they are accessed. The value "never" should be used to describe archived URLs.
                                                    $frequency = "weekly";
                                                    
                                                    
                                                    # Priority of page in relation to other parts of your site. A number from 0.1 to 1.0 is acceptable.
                                                    $priority = "0.5";
                                                    
                                                    # Include last modification date. Options are: true or false
                                                    # The date of last modification of the file. This date should be in W3C Datetime format. This format allows you to omit the time portion, if desired, and use YYYY-MM-DD.
                                                    $lastmodification = true;
                                                    
                                                    # File extensions to include in sitemap.
                                                    $extensions = array("htm", "html", "php", "asp", "pdf");
                                                    
                                                    # Try to index dynamic web pages that have a parameter in there url (?). Valid options
                                                    # are true or false. Use this at your own risk, could capture session info
                                                    # which possibly could cause problems during the Google index process.
                                                    $index_dynamic_pages_params = true;
                                                    
                                                    # First do a check that allow_url_fopen is set to on
                                                    if(ini_get("allow_url_fopen") != 1)
                                                    	die("The php.ini directive 'allow_url_fopen' must be set to 'On' for this script to function.\nPlease set this to 'On' in your php.ini file.\n");
                                                    
                                                    # Make url compatible with Google sitemap Specifications
                                                    # As with all XML files, any data values (including URLs) must use entity escape codes for the characters listed in the table below. 
                                                    
                                                    
                                                    #  --------------------------------------
                                                    # |   Character   | Simbol | Escape Code |
                                                    # |---------------|--------|-------------|
                                                    # |   Ampersand   |    &   |    &amp;    |
                                                    # |---------------|--------|-------------|
                                                    # | Single Quote  |    '   |    &apos;   |
                                                    # |---------------|--------|-------------|
                                                    # | Double Quote  |    "   |    &quot;   |
                                                    # |---------------|--------|-------------|
                                                    # | Greater Than  |    >   |    &gt;     |
                                                    # |---------------|--------|-------------|
                                                    # |  Less Than    |    <   |    &lt;     |
                                                    # |---------------|--------|-------------|
                                                    
                                                    function googlesitemap_compatible($url) {
                                                        $url = str_replace("&","&amp;",$url);
                                                    	$url = str_replace("'","&apos;",$url);
                                                    	$url = str_replace('"',"&quot;",$url);
                                                    	$url = str_replace(">","&gt;",$url);
                                                    	$url = str_replace("<","&lt;",$url);
                                                    	return $url;
                                                    }
                                                    
                                                    
                                                    
                                                    # Gets a URLs path minus the actual filename + query.
                                                    function getPath($url) {
                                                        if($GLOBALS['index_dynamic_pages_params'] == true) {
                                                            $url = explode("?", $url);
                                                            $url = $url[0];
                                                        }
                                                    	
                                                    	$temp = explode("/", $url);
                                                    	$fnsize=strlen($temp[(count($temp) - 1)]);
                                                    	return substr($url, 0, strlen($url) - $fnsize);
                                                    }
                                                    
                                                    # Cleans up a path so that extra / slashes are gone, .. are translated, etc
                                                    function cleanPath($url) {
                                                    	$new = array();
                                                    	$url = explode("/", trim($url));
                                                    	foreach($url as $p) {
                                                    		$p = trim($p);
                                                    		if($p != "" && $p != ".") {
                                                    			if($p == "..") {
                                                    				if(is_array($new))
                                                    					$new = array_pop($new);
                                                    			} else {
                                                    				$new = array_merge((array) $new, array($p));
                                                    			}
                                                    		}
                                                    	}
                                                    	
                                                    	$url = $new[0]."/";
                                                    	for($i=1; $i < count($new); $i++)
                                                    		$url .= "/".$new[$i];
                                                    	
                                                    	return $url;
                                                    }
                                                    
                                                    # Checks if URL has specified extension, if so returns true
                                                    function checkExt($url, $ext) {
                                                    	# Strip out parameter info from a script (?)
                                                    	if($GLOBALS['index_dynamic_pages_params'] == true) {
                                                    		$url = explode("?", $url);
                                                    		$url = $url[0];
                                                    	}
                                                    	
                                                    	$text=substr($url, strlen($url) - (strlen($ext) + 1), strlen($url));
                                                    	if($text == ".".$ext)
                                                    		return true;
                                                    	else
                                                    		return false;
                                                    }
                                                    
                                                    # Retrieve Site URLs
                                                    function getUrls($url, $string) {
                                                    	$type = "href";
                                                    	# Regex to chop out urls
                                                    	preg_match_all("|$type\=\"?'?`?([[:alnum:]:?=&@/._-]+)\"?'?`?|i", $string, $matches);
                                                    	$ret[$type] = $matches[1];
                                                    
                                                    	# Make all URLS literal (full path)
                                                    	for($i = 0; $i < count($ret['href']); $i++) {
                                                    		if(! preg_match( '/^(http|https):\/\//i' , $ret['href'][$i]))
                                                    			$ret['href'][$i] = getPath($url)."/".$ret['href'][$i];
                                                    	
                                                    		$ret['href'][$i] = cleanPath($ret['href'][$i]);
                                                    	}
                                                    	
                                                    	return $ret;
                                                    }
                                                    
                                                    function addUrls($urls) {
                                                    	if(is_array($urls))
                                                    		for($i=0; $i < count($urls['href']); $i++) {
                                                    			$skip = 0;
                                                    			
                                                    			# Cycle through to make sure url is unique
                                                    			for($x=0; $x < count($GLOBALS['urls']); $x++)
                                                    				if($GLOBALS['urls'][$x] == $urls['href'][$i]) {
                                                    					$skip = 1;
                                                    					break;
                                                    				}
                                                    			
                                                    			# Check extension
                                                    			$extgood = 0;
                                                    			foreach($GLOBALS['extensions'] as $ext)
                                                    				if(checkExt($urls['href'][$i], $ext))
                                                    					$extgood = 1;
                                                    			
                                                    			# And finally make sure its in the current website
                                                    			if(! stristr($urls['href'][$i], $GLOBALS['siteurl']))
                                                    				$skip = 1;
                                                    
                                                    			if($skip == 0 && $extgood == 1)
                                                    				$GLOBALS['urls'][] = $urls['href'][$i];
                                                    		}
                                                    }
                                                    
                                                    function getNextUrl($oldurl) {
                                                    	if($oldurl == "")
                                                    		return $GLOBALS['urls'][0];
                                                    		
                                                    	for($i=0; $i < count($GLOBALS['urls']); $i++)
                                                    		if($GLOBALS['urls'][$i] == $oldurl)
                                                    			if(isset($GLOBALS['urls'][($i+1)]))
                                                    				return $GLOBALS['urls'][($i+1)];
                                                    			else
                                                    				return false;
                                                    	
                                                    	return false;
                                                    }
                                                    
                                                    $urls = array($siteurl);
                                                    
                                                    #start to generate inline sitemap
                                                    echo '<?xml version="1.0" encoding="UTF-8"?>'."\n".'<!--Google Site Map File Generated by http://xml-sitemap-generator.com/ '.date("D, d M Y G:i:s T").' -->'."\n".'<urlset xmlns="http://www.google.com/schemas/sitemap/0.84">'."\n";
                                                    
                                                    # if possible the script will writte on website a copy of generated sitemap
                                                    # start to writte generated sitemap
                                                    # Make sure you have writte permission to file sitemap_temp.xml
                                                    
                                                    
                                                    if($fp = fopen("sitemap_temp.xml", "w")) {
                                                      $open_file = 'sucess';
                                                      fputs($fp, '<?xml version="1.0" encoding="UTF-8"?>'."\n".'<!--Google Site Map File Generated by http://xml-sitemap-generator.com/ '.date("D, d M Y G:i:s T").' -->'."\n".'<urlset xmlns="http://www.google.com/schemas/sitemap/0.84">'."\n");
                                                    }
                                                    
                                                    # Log 404 Error Page Not Found
                                                    # Make sure you have writte access to file 404error.txt
                                                    if($fp_err = fopen("404error.txt", "w")) $open_file_404 = 'sucess';
                                                    	
                                                    $turl = "";
                                                    # Cycle through tree and build a url list
                                                    while($turl = getNextUrl($turl)) {
                                                    	# Extend script time limit
                                                    	set_time_limit(3000);
                                                    
                                                    	# Read html file into memory
                                                    	if($html = file($turl)) {
                                                    		$html = stripslashes(implode($html));
                                                            echo '<url>'."\n\t".'<loc>'.googlesitemap_compatible($turl).'</loc>'."\n";
                                                    		if($lastmodification == true)
                                                        		echo "\t".'<lastmod>'.date("Y-m-d").'</lastmod>'."\n";
                                                    		echo "\t".'<changefreq>'.$frequency.'</changefreq>'."\n\t".'<priority>'.$priority.'</priority>'."\n".'</url>'."\n";
                                                    		# Get site urls from html and add new unique url to list if needed
                                                    		addUrls(getUrls($turl, $html));
                                                            #writte the same thing above on website if you have permission to writte
                                                    		if($open_file == 'sucess') {
                                                    			fputs($fp,'<url>'."\n\t".'<loc>'.googlesitemap_compatible($turl).'</loc>'."\n");
                                                    			if($lastmodification == true)
                                                        			fputs($fp,"\t".'<lastmod>'.date("Y-m-d").'</lastmod>'."\n");
                                                    			fputs($fp,"\t".'<changefreq>'.$frequency.'</changefreq>'."\n\t".'<priority>'.$priority.'</priority>'."\n".'</url>'."\n");
                                                    		}
                                                    	} else {
                                                    		# check if 404error.txt was sucsefuly opened
                                                    		if($open_file_404 == 'sucess') fputs($fp_err, $turl."\n");
                                                    	}
                                                    }
                                                    
                                                    echo '</urlset>';	
                                                    if($open_file == 'sucess') {
                                                    	fputs($fp, '</urlset>');
                                                    	fclose($fp);
                                                    	# Make sure you have writte access to file sitemap_OK.xml
                                                    	# To track evolution of your google sitemap you can replace sitemap_temp.xml with "sitemap_OK_".date(d-m-y).".xml"
                                                    	copy('sitemap_temp.xml','sitemap_OK.xml');
                                                    	
                                                    if($open_file_404 == 'sucess') fclose($fp_err);
                                                    }
                                                    
                                                    
                                                    
                                                    ?>
                                                    .htaccess=

                                                    Code:
                                                    RewriteEngine on
                                                    RewriteRule (.*)\.xml(.*) $1.php$2 [nocase]

                                                    Comment

                                                    • TheDoc
                                                      Too lazy to set a custom title
                                                      • Jul 2001
                                                      • 13827

                                                      #27
                                                      Paysites can benefit from the use of sitemaps due to the broken linking structures and affiliate codes. You don't have to register the sitemap.

                                                      You can use a sitemap at a simple level and not include every page in your site. If pages (menus) are easy to spider and find, then no reason to include them in the sitemap.

                                                      Paysites however can tell google that tour(s) are priority over the warning page. Doing this makes your tour list in google rather than the warning page under your keyword listings.
                                                      ~TheDoc - ICQ7765825
                                                      It's all disambiguation

                                                      Comment

                                                      Working...