GoogleBot Activity Script

[cj-design]

Hi guys,

I'm just finishing off a script that will allow you to log each and every crawl that Google makes on your website. Some screenshots can be found here:

http://www.cj-design.com/downloads/g...activity/demo/

The script will be up for free download very soon (just finishing off the installation guide) - so if you dont have anything like this on your site, stay tuned.



P.S The script is written in PHP therefore can only be run in pages that can include PHP files (i.e .php, .php3, .shtml, .shtm)

[hudson]

[davidd]

Originally posted by cj-design
Hi guys,

I'm just finishing off a script that will allow you to log each and every crawl that Google makes on your website. Some screenshots can be found here:

http://www.cj-design.com/downloads/g...activity/demo/

The script will be up for free download very soon (just finishing off the installation guide) - so if you dont have anything like this on your site, stay tuned.



P.S The script is written in PHP therefore can only be run in pages that can include PHP files (i.e .php, .php3, .shtml, .shtm)

What are the features?

Is it any better than:

grep googlebot access.log | cut -f4 -d'/' | sed s/HTTP//g | sort | uniq



This is a serious question, as here is my count from today alone, across my domains:

% grep googlebot * |wc -l
18974

[cj-design]

Its for webmasters that dont understand those commands, therefore it is layed out in a more attractive interface. Features are pretty minimal at the moment as its only V1.0; they include; sort the log, export the log, clear the log. In the future you will be able to search the log, say you want to see which pages "crawler8.googlebot.com" indexed.

Doesn't require MySQL either, so pretty simple all around.

[hudson]

grep googlebot access.log | cut -f4 -d'/' | sed s/HTTP//g | sort | uniq

shhhhhh...top secret!!

This is a serious question, as here is my count from today alone, across my domains:

% grep googlebot * |wc -l
18974

hmmmm...want to buy some scripts?

[flashfreak]

nice but too easy

[cj-design]

I would hope that most of the webmasters on these forums would find the script 'easy' as you put it.

[Wiz]

Your script can handle 150 to 250k pages? ;)

[hudson]

Originally posted by Wiz
Your script can handle 150 to 250k pages? ;)

yo...domains....so you got 40k pages in a domain, well...only one stat file there

[cj-design]

Originally posted by Wiz
Your script can handle 150 to 250k pages? ;)

If it was driven by MySQL, yeh - coming soon of course.

[Brujah]

Originally posted by cj-design
Hi guys,

I'm just finishing off a script that will allow you to log each and every crawl that Google makes on your website. Some screenshots can be found here:

http://www.cj-design.com/downloads/g...activity/demo/

The script will be up for free download very soon (just finishing off the installation guide) - so if you dont have anything like this on your site, stay tuned.



P.S The script is written in PHP therefore can only be run in pages that can include PHP files (i.e .php, .php3, .shtml, .shtm)

Looks great, keep us informed

[cj-design]

OK,

I typed like a bitch to finish it, download as you please:

http://www.cj-design.com/index.php?id=downloads&page=13

Report bugs to me via email - should be ok though

[AdultNex]

Hrm, I thought I remembered going to your site before... I was just on your site yesterday looking for a quote generator.

[cj-design]

Originally posted by AdultNex
Hrm, I thought I remembered going to your site before... I was just on your site yesterday looking for a quote generator.

Ah the classic CJ Random Quote - never fails.

[Kevin2]

cj-design you have some great scripts on your site

[cj-design]

Originally posted by Kevin2
cj-design you have some great scripts on your site

Thanks. Some of them are a bit out of date now due to PHP moving on and me becomming a better programmer, but they do the job!

[hudson]

Originally posted by cj-design


Ah the classic CJ Random Quote - never fails.

what's the classic CJ Random Quote...

[cj-design]

do you really need to ask, everybody, this guy has never heard of the classic CJ Random Quote.... everybody? Oh...

Well here it is in action:

http://www.cj-design.com/downloads/randomquote/demo/

[hudson]

Originally posted by cj-design
do you really need to ask, everybody, this guy has never heard of the classic CJ Random Quote.... everybody? Oh...

Well here it is in action:

http://www.cj-design.com/downloads/randomquote/demo/

nice site...very nice! ;-)

[cj-design]

Originally posted by hudson


nice site...very nice! ;-)

cheers mate, its not often you post on these forums and recieve appraisals.

[hudson]

Originally posted by cj-design


cheers mate, its not often you post on these forums and recieve appraisals.

hehe...

[davidd]

Originally posted by cj-design
Its for webmasters that dont understand those commands, therefore it is layed out in a more attractive interface. Features are pretty minimal at the moment as its only V1.0; they include; sort the log, export the log, clear the log. In the future you will be able to search the log, say you want to see which pages "crawler8.googlebot.com" indexed.

Doesn't require MySQL either, so pretty simple all around.

I downloaded and installed it a short while ago. I will let you know my results in a week or so. So far, I like what I see, as you have consolidated a lot of my cron's and hodge podge of shell scripts into one unit.

Update the install .html with:

If you are executing this file using SSI, you must add the location of your PHP binary to the top of the googlebot.php (i.e. #!/usr/local/bin/php), and googlebot.php must be CHMOD 755.

The one thing that I thought was kind of ->un-cool<- was you added output to the html pages via googlebot.php. So upon execution, the bot will see:

CJ GoogleBot Activity V1.0 is running on this site
<!-- CJ GoogleBot Activity V1.0 is running on this site -->

I am cool with self promotion, but not in the search engine game. If the above was used as an indicator by the Google people for blacklisting or deeper inspection, a large number of people would be penalized unknowingly.

My comments, should always be deemed as constructive criticism, I am not in the game of slamming people's work.

-dd

[blackmonsters]

What would I do with this information if I used the script?

[Trax]

how stable is that script?

[DarkJedi]

where to download it ?

[AdultNex]

Originally posted by davidd


I downloaded and installed it a short while ago. I will let you know my results in a week or so. So far, I like what I see, as you have consolidated a lot of my cron's and hodge podge of shell scripts into one unit.

Update the install .html with:

If you are executing this file using SSI, you must add the location of your PHP binary to the top of the googlebot.php (i.e. #!/usr/local/bin/php), and googlebot.php must be CHMOD 755.

The one thing that I thought was kind of ->un-cool<- was you added output to the html pages via googlebot.php. So upon execution, the bot will see:

CJ GoogleBot Activity V1.0 is running on this site
<!-- CJ GoogleBot Activity V1.0 is running on this site -->

I am cool with self promotion, but not in the search engine game. If the above was used as an indicator by the Google people for blacklisting or deeper inspection, a large number of people would be penalized unknowingly.

My comments, should always be deemed as constructive criticism, I am not in the game of slamming people's work.

-dd

This too should not be looked upon as flaming. What is the purpose of having a self-promotion tagline when the "stats" aren't publicly displayed?

[cj-design]

Originally posted by davidd


I downloaded and installed it a short while ago. I will let you know my results in a week or so. So far, I like what I see, as you have consolidated a lot of my cron's and hodge podge of shell scripts into one unit.

Update the install .html with:

If you are executing this file using SSI, you must add the location of your PHP binary to the top of the googlebot.php (i.e. #!/usr/local/bin/php), and googlebot.php must be CHMOD 755.

The one thing that I thought was kind of ->un-cool<- was you added output to the html pages via googlebot.php. So upon execution, the bot will see:

CJ GoogleBot Activity V1.0 is running on this site
<!-- CJ GoogleBot Activity V1.0 is running on this site -->

I am cool with self promotion, but not in the search engine game. If the above was used as an indicator by the Google people for blacklisting or deeper inspection, a large number of people would be penalized unknowingly.

My comments, should always be deemed as constructive criticism, I am not in the game of slamming people's work.

-dd

Hi dd,

Thanks for the installation file addition.

As for the tag line (html comment) - that was left in by mistake, It was put there just to test the thing was being included and I forgot to take it out - so ive also changed that.

This too should not be looked upon as flaming. What is the purpose of having a self-promotion tagline when the "stats" aren't publicly displayed?

AdultNex - basically just repeated what dd said, so same to you really - its sorted now.

where to download it ?

Read previous posts doofas

how stable is that script?

pretty stable, not to be uses on sites that get crawled 500+ times a day (unless your gonna clear your own log every 30 minutes or something)

What would I do with this information if I used the script?

It was written for two reasons (my own reasons):

A) To monitor GoogleBot (how it acts)

B) To monitor the pages of my site that were crawled

What I found out...

A) Googlebot (usually crawler2) will pick up a link, say from hotscripts.com and find its way to my site, crawler4, say, would then come back and crawl that link properly - also found some other stuff like crawler12 is a deep crawler (takes most of your site when it crawls)

B) I found that my recently updated pages were crawled in a matter of minutes (pretty sweet how it knows)

What you can do with the information you get....

A) The same as what I have done
B) Wipe your ass on it

[acctman]

it's free

[pradaboy]

thnx for sharing man... this is great!