Quote:
Originally posted by davidd
I downloaded and installed it a short while ago. I will let you know my results in a week or so. So far, I like what I see, as you have consolidated a lot of my cron's and hodge podge of shell scripts into one unit.
Update the install .html with:
If you are executing this file using SSI, you must add the location of your PHP binary to the top of the googlebot.php (i.e. #!/usr/local/bin/php), and googlebot.php must be CHMOD 755.
The one thing that I thought was kind of ->un-cool<- was you added output to the html pages via googlebot.php. So upon execution, the bot will see:
CJ GoogleBot Activity V1.0 is running on this site
<!-- CJ GoogleBot Activity V1.0 is running on this site -->
I am cool with self promotion, but not in the search engine game. If the above was used as an indicator by the Google people for blacklisting or deeper inspection, a large number of people would be penalized unknowingly.
My comments, should always be deemed as constructive criticism, I am not in the game of slamming people's work.
-dd
|
Hi dd,
Thanks for the installation file addition.
As for the tag line (html comment) - that was left in by mistake, It was put there just to test the thing was being included and I forgot to take it out - so ive also changed that.
Quote:
|
This too should not be looked upon as flaming. What is the purpose of having a self-promotion tagline when the "stats" aren't publicly displayed?
|
AdultNex - basically just repeated what dd said, so same to you really - its sorted now.
Read previous posts doofas
Quote:
|
how stable is that script?
|
pretty stable, not to be uses on sites that get crawled 500+ times a day (unless your gonna clear your own log every 30 minutes or something)
Quote:
|
What would I do with this information if I used the script?
|
It was written for two reasons (my own reasons):
A) To monitor GoogleBot (how it acts)
B) To monitor the pages of my site that were crawled
What I found out...
A) Googlebot (usually crawler2) will pick up a link, say from hotscripts.com and find its way to my site, crawler4, say, would then come back and crawl that link properly - also found some other stuff like crawler12 is a deep crawler (takes most of your site when it crawls)
B) I found that my recently updated pages were crawled in a matter of minutes (pretty sweet how it knows)
What you can do with the information you get....
A) The same as what I have done
B) Wipe your ass on it