SiteMap Generator

[OzMan84]

is there such thing or how do i make one manually?

for SE purposes

[StarkReality]

http://www.auditmypc.com/xml-sitemap.asp is a free one for example (uses java) or just google for "sitemap generator", there are loads around.

About sitemaps in general: Great for new sites, be careful with established sites since adding a sitemap to a website online for years will hurt your rankings without any real advantage

[wtfent]

http://www.xml-sitemaps.com/

[fallenmuffin]

http://www.xml-sitemaps.com/
http://www.rorweb.com/rormap.htm
http://www.sitemapdoc.com/
http://www.auditmypc.com/xml-sitemap.asp (Java, you've been warned)
http://www.devintelligence.com/sitemapgenerator/ (Free desktop app)
http://www.neuroticweb.com/recursos/sitemap/
http://www.sitemapbuilder.net/

And so many more...

[uno]

www.google.com

search: google, sitemaps

look for a result on google.com, bam.

[OzMan84]

half my pages arent getting picked up with these scripts, any idea to why??

[georgeyw]

google has one and it's not really all that friendly either...

[DamageX]

Originally Posted by StarkReality

be careful with established sites since adding a sitemap to a website online for years will hurt your rankings without any real advantage

Elaborate, please. Doesn't make much sense to me, without further info on this.

[munki]

http://www.download.com/3001-2379_4-10419602.html

SiteMapperBuilder.net is a personal fave of mine.

[GrouchyAdmin]

Just so you know, Google will also accept RSS feeds in lieu of a generator; it's not as good for granularity, but it does work quite well for your basic blog.

[GeorgeM]

thx for links

http://www.devintelligence.com/sitemapgenerator/

[StarkReality]

Originally Posted by DamageX

Elaborate, please. Doesn't make much sense to me, without further info on this.

Just very painful personal experiences. Sitemaps are just good for getting a large site crawled quicker, plus they avoid pages getting lost due to a bad/broken linking structure, but there are several disadvantages:

1. You need to use real 404 pages, redirects aren't possible any more since you need to report a 404 and not a 200.

2. Sitemaps do not help your site except for being crawled better. If you have an established well linked site, there is no need for a sitemap at all.

3. Adding sitemaps to your google account allows google to see which sites you are running, even if you use different IPs. Since almost all adult sites are "thin affiliates" by google's terms, getting one of them banned would mean you get the others kicked as well.

I lost a part of a network because of this. One site was kicked, all other sites in the google account followed. Those not added in the account stayed in google. Bad coincidence ? I don't think so, 11 sites from the network were kicked, 23 stayed and the 11 were those in the google account.

4. By giving pages a certain importance, your suggestions compete with google's internal evaluation. It really messes with your rankings, seen it happen more than once.

Don't get me wrong, if you want to use a old style html sitemap, fine, but the XML ones are a big NO NO except on a new or badly linked site. If a site is built and linked well, there are zero advantages of a xml sitemap. So, why use one ?

[fris]

use googles webmaster tools

[DamageX]

Originally Posted by StarkReality

Just very painful personal experiences. Sitemaps are just good for getting a large site crawled quicker, plus they avoid pages getting lost due to a bad/broken linking structure, but there are several disadvantages:

1. You need to use real 404 pages, redirects aren't possible any more since you need to report a 404 and not a 200.

2. Sitemaps do not help your site except for being crawled better. If you have an established well linked site, there is no need for a sitemap at all.

3. Adding sitemaps to your google account allows google to see which sites you are running, even if you use different IPs. Since almost all adult sites are "thin affiliates" by google's terms, getting one of them banned would mean you get the others kicked as well.

I lost a part of a network because of this. One site was kicked, all other sites in the google account followed. Those not added in the account stayed in google. Bad coincidence ? I don't think so, 11 sites from the network were kicked, 23 stayed and the 11 were those in the google account.

4. By giving pages a certain importance, your suggestions compete with google's internal evaluation. It really messes with your rankings, seen it happen more than once.

Don't get me wrong, if you want to use a old style html sitemap, fine, but the XML ones are a big NO NO except on a new or badly linked site. If a site is built and linked well, there are zero advantages of a xml sitemap. So, why use one ?

Thanks for taking the time to explain this. It makes more sense now.

[he-fox]

Originally Posted by wtfent

http://www.xml-sitemaps.com/

[GooSearch]

Xenu's Link Sleuth
http://home.snafu.de/tilman/xenulink.html

kick ass free proggy

[OzMan84]

ok like i said up above, half my pages are not getting found with the crawling....any ideas?

[DamageX]

Originally Posted by OzMan84

ok like i said up above, half my pages are not getting found with the crawling....any ideas?

Yeah, get more backlinks to the individual pages.

[GooSearch]

Originally Posted by OzMan84

ok like i said up above, half my pages are not getting found with the crawling....any ideas?

html site map linked from your index page.. hardlinks to your sitemap.. and then u'll see a more complete spider./.. works for me

[mattz]

Originally Posted by StarkReality

http://www.auditmypc.com/xml-sitemap.asp is a free one for example (uses java) or just google for "sitemap generator", there are loads around.

About sitemaps in general: Great for new sites, be careful with established sites since adding a sitemap to a website online for years will hurt your rankings without any real advantage

Thanks for the tip

[martinsc]

Originally Posted by StarkReality

Just very painful personal experiences. Sitemaps are just good for getting a large site crawled quicker, plus they avoid pages getting lost due to a bad/broken linking structure, but there are several disadvantages:

1. You need to use real 404 pages, redirects aren't possible any more since you need to report a 404 and not a 200.

2. Sitemaps do not help your site except for being crawled better. If you have an established well linked site, there is no need for a sitemap at all.

3. Adding sitemaps to your google account allows google to see which sites you are running, even if you use different IPs. Since almost all adult sites are "thin affiliates" by google's terms, getting one of them banned would mean you get the others kicked as well.

I lost a part of a network because of this. One site was kicked, all other sites in the google account followed. Those not added in the account stayed in google. Bad coincidence ? I don't think so, 11 sites from the network were kicked, 23 stayed and the 11 were those in the google account.

4. By giving pages a certain importance, your suggestions compete with google's internal evaluation. It really messes with your rankings, seen it happen more than once.

Don't get me wrong, if you want to use a old style html sitemap, fine, but the XML ones are a big NO NO except on a new or badly linked site. If a site is built and linked well, there are zero advantages of a xml sitemap. So, why use one ?

thanks

[OzMan84]

Originally Posted by GooSearch

html site map linked from your index page.. hardlinks to your sitemap.. and then u'll see a more complete spider./.. works for me

so make a nice little site map linking to all 150+ pages correct?

[OzMan84]

Originally Posted by OzMan84

so make a nice little site map linking to all 150+ pages correct?

bump bump just need to know if i was on the right track with that

[OzMan84]

http://www.hqerosescorts.com/sitemap.htm

is that what you guys ment?

[Empress_JoinRightNow]

google sitemaps are more effective and it's automatic...

[bonkerz2007]

Create two files:
sitemap.php
.htaccess

sitemap.php =

<?php

#To ignore #404 Page Not Found error, and do not include them in sitemap
error_reporting(0);

# The script is free of charge. 
# Mandatory disclaimer: THIS SCRIPT CARRIES NO WARRANTY OR GUARRANTEE WHAT SO EVER. USE IS AT YOUR OWN RISK.

# The site url to crawl. Remember to include the slash ( / ) at the end. (EX: http://www.site.com/)
$siteurl = 'http://'.$_SERVER["HTTP_HOST"].'/';

# The frequency of updating. Valid settings are: always, hourly, daily, weekly, monthly, yearly, never
# The value "always" should be used to describe documents that change each time they are accessed. The value "never" should be used to describe archived URLs.
$frequency = "weekly";


# Priority of page in relation to other parts of your site. A number from 0.1 to 1.0 is acceptable.
$priority = "0.5";

# Include last modification date. Options are: true or false
# The date of last modification of the file. This date should be in W3C Datetime format. This format allows you to omit the time portion, if desired, and use YYYY-MM-DD.
$lastmodification = true;

# File extensions to include in sitemap.
$extensions = array("htm", "html", "php", "asp", "pdf");

# Try to index dynamic web pages that have a parameter in there url (?). Valid options
# are true or false. Use this at your own risk, could capture session info
# which possibly could cause problems during the Google index process.
$index_dynamic_pages_params = true;

# First do a check that allow_url_fopen is set to on
if(ini_get("allow_url_fopen") != 1)
	die("The php.ini directive 'allow_url_fopen' must be set to 'On' for this script to function.\nPlease set this to 'On' in your php.ini file.\n");

# Make url compatible with Google sitemap Specifications
# As with all XML files, any data values (including URLs) must use entity escape codes for the characters listed in the table below. 


#  --------------------------------------
# |   Character   | Simbol | Escape Code |
# |---------------|--------|-------------|
# |   Ampersand   |    &   |    &amp;    |
# |---------------|--------|-------------|
# | Single Quote  |    '   |    &apos;   |
# |---------------|--------|-------------|
# | Double Quote  |    "   |    &quot;   |
# |---------------|--------|-------------|
# | Greater Than  |    >   |    &gt;     |
# |---------------|--------|-------------|
# |  Less Than    |    <   |    &lt;     |
# |---------------|--------|-------------|

function googlesitemap_compatible($url) {
    $url = str_replace("&","&amp;",$url);
	$url = str_replace("'","&apos;",$url);
	$url = str_replace('"',"&quot;",$url);
	$url = str_replace(">","&gt;",$url);
	$url = str_replace("<","&lt;",$url);
	return $url;
}



# Gets a URLs path minus the actual filename + query.
function getPath($url) {
    if($GLOBALS['index_dynamic_pages_params'] == true) {
        $url = explode("?", $url);
        $url = $url[0];
    }
	
	$temp = explode("/", $url);
	$fnsize=strlen($temp[(count($temp) - 1)]);
	return substr($url, 0, strlen($url) - $fnsize);
}

# Cleans up a path so that extra / slashes are gone, .. are translated, etc
function cleanPath($url) {
	$new = array();
	$url = explode("/", trim($url));
	foreach($url as $p) {
		$p = trim($p);
		if($p != "" && $p != ".") {
			if($p == "..") {
				if(is_array($new))
					$new = array_pop($new);
			} else {
				$new = array_merge((array) $new, array($p));
			}
		}
	}
	
	$url = $new[0]."/";
	for($i=1; $i < count($new); $i++)
		$url .= "/".$new[$i];
	
	return $url;
}

# Checks if URL has specified extension, if so returns true
function checkExt($url, $ext) {
	# Strip out parameter info from a script (?)
	if($GLOBALS['index_dynamic_pages_params'] == true) {
		$url = explode("?", $url);
		$url = $url[0];
	}
	
	$text=substr($url, strlen($url) - (strlen($ext) + 1), strlen($url));
	if($text == ".".$ext)
		return true;
	else
		return false;
}

# Retrieve Site URLs
function getUrls($url, $string) {
	$type = "href";
	# Regex to chop out urls
	preg_match_all("|$type\=\"?'?`?([[:alnum:]:?=&@/._-]+)\"?'?`?|i", $string, $matches);
	$ret[$type] = $matches[1];

	# Make all URLS literal (full path)
	for($i = 0; $i < count($ret['href']); $i++) {
		if(! preg_match( '/^(http|https):\/\//i' , $ret['href'][$i]))
			$ret['href'][$i] = getPath($url)."/".$ret['href'][$i];
	
		$ret['href'][$i] = cleanPath($ret['href'][$i]);
	}
	
	return $ret;
}

function addUrls($urls) {
	if(is_array($urls))
		for($i=0; $i < count($urls['href']); $i++) {
			$skip = 0;
			
			# Cycle through to make sure url is unique
			for($x=0; $x < count($GLOBALS['urls']); $x++)
				if($GLOBALS['urls'][$x] == $urls['href'][$i]) {
					$skip = 1;
					break;
				}
			
			# Check extension
			$extgood = 0;
			foreach($GLOBALS['extensions'] as $ext)
				if(checkExt($urls['href'][$i], $ext))
					$extgood = 1;
			
			# And finally make sure its in the current website
			if(! stristr($urls['href'][$i], $GLOBALS['siteurl']))
				$skip = 1;

			if($skip == 0 && $extgood == 1)
				$GLOBALS['urls'][] = $urls['href'][$i];
		}
}

function getNextUrl($oldurl) {
	if($oldurl == "")
		return $GLOBALS['urls'][0];
		
	for($i=0; $i < count($GLOBALS['urls']); $i++)
		if($GLOBALS['urls'][$i] == $oldurl)
			if(isset($GLOBALS['urls'][($i+1)]))
				return $GLOBALS['urls'][($i+1)];
			else
				return false;
	
	return false;
}

$urls = array($siteurl);

#start to generate inline sitemap
echo '<?xml version="1.0" encoding="UTF-8"?>'."\n".'<!--Google Site Map File Generated by http://xml-sitemap-generator.com/ '.date("D, d M Y G:i:s T").' -->'."\n".'<urlset xmlns="http://www.google.com/schemas/sitemap/0.84">'."\n";

# if possible the script will writte on website a copy of generated sitemap
# start to writte generated sitemap
# Make sure you have writte permission to file sitemap_temp.xml


if($fp = fopen("sitemap_temp.xml", "w")) {
  $open_file = 'sucess';
  fputs($fp, '<?xml version="1.0" encoding="UTF-8"?>'."\n".'<!--Google Site Map File Generated by http://xml-sitemap-generator.com/ '.date("D, d M Y G:i:s T").' -->'."\n".'<urlset xmlns="http://www.google.com/schemas/sitemap/0.84">'."\n");
}

# Log 404 Error Page Not Found
# Make sure you have writte access to file 404error.txt
if($fp_err = fopen("404error.txt", "w")) $open_file_404 = 'sucess';
	
$turl = "";
# Cycle through tree and build a url list
while($turl = getNextUrl($turl)) {
	# Extend script time limit
	set_time_limit(3000);

	# Read html file into memory
	if($html = file($turl)) {
		$html = stripslashes(implode($html));
        echo '<url>'."\n\t".'<loc>'.googlesitemap_compatible($turl).'</loc>'."\n";
		if($lastmodification == true)
    		echo "\t".'<lastmod>'.date("Y-m-d").'</lastmod>'."\n";
		echo "\t".'<changefreq>'.$frequency.'</changefreq>'."\n\t".'<priority>'.$priority.'</priority>'."\n".'</url>'."\n";
		# Get site urls from html and add new unique url to list if needed
		addUrls(getUrls($turl, $html));
        #writte the same thing above on website if you have permission to writte
		if($open_file == 'sucess') {
			fputs($fp,'<url>'."\n\t".'<loc>'.googlesitemap_compatible($turl).'</loc>'."\n");
			if($lastmodification == true)
    			fputs($fp,"\t".'<lastmod>'.date("Y-m-d").'</lastmod>'."\n");
			fputs($fp,"\t".'<changefreq>'.$frequency.'</changefreq>'."\n\t".'<priority>'.$priority.'</priority>'."\n".'</url>'."\n");
		}
	} else {
		# check if 404error.txt was sucsefuly opened
		if($open_file_404 == 'sucess') fputs($fp_err, $turl."\n");
	}
}

echo '</urlset>';	
if($open_file == 'sucess') {
	fputs($fp, '</urlset>');
	fclose($fp);
	# Make sure you have writte access to file sitemap_OK.xml
	# To track evolution of your google sitemap you can replace sitemap_temp.xml with "sitemap_OK_".date(d-m-y).".xml"
	copy('sitemap_temp.xml','sitemap_OK.xml');
	
if($open_file_404 == 'sucess') fclose($fp_err);
}



?>

.htaccess=

RewriteEngine on
RewriteRule (.*)\.xml(.*) $1.php$2 [nocase]

[TheDoc]

Paysites can benefit from the use of sitemaps due to the broken linking structures and affiliate codes. You don't have to register the sitemap.

You can use a sitemap at a simple level and not include every page in your site. If pages (menus) are easy to spider and find, then no reason to include them in the sitemap.

Paysites however can tell google that tour(s) are priority over the warning page. Doing this makes your tour list in google rather than the warning page under your keyword listings.