GoogleBot Activity Script
 

Hi guys,

I'm just finishing off a script that will allow you to log each and every crawl that Google makes on your website. Some screenshots can be found here:

http://www.cj-design.com/downloads/g...activity/demo/

The script will be up for free download very soon (just finishing off the installation guide) - so if you dont have anything like this on your site, stay tuned.

:thumbsup

P.S The script is written in PHP therefore can only be run in pages that can include PHP files (i.e .php, .php3, .shtml, .shtm)

:glugglug :thumbsup

What are the features?

Is it any better than:

grep googlebot access.log | cut -f4 -d'/' | sed s/HTTP//g | sort | uniq

:)

This is a serious question, as here is my count from today alone, across my domains:

% grep googlebot * |wc -l
18974

Its for webmasters that dont understand those commands, therefore it is layed out in a more attractive interface. Features are pretty minimal at the moment as its only V1.0; they include; sort the log, export the log, clear the log. In the future you will be able to search the log, say you want to see which pages "crawler8.googlebot.com" indexed.

Doesn't require MySQL either, so pretty simple all around.

shhhhhh...top secret!!

hmmmm...want to buy some scripts? :glugglug

nice but too easy

I would hope that most of the webmasters on these forums would find the script 'easy' as you put it.

:winkwink:

Your script can handle 150 to 250k pages? ;)

yo...domains....so you got 40k pages in a domain, well...only one stat file there :2 cents:

If it was driven by MySQL, yeh - coming soon of course.

Looks great, keep us informed :)

OK,

I typed like a bitch to finish it, download as you please:

http://www.cj-design.com/index.php?id=downloads&page=13

Report bugs to me via email - should be ok though

Hrm, I thought I remembered going to your site before... I was just on your site yesterday looking for a quote generator.

Ah the classic CJ Random Quote - never fails.

cj-design you have some great scripts on your site :thumbsup

Thanks. Some of them are a bit out of date now due to PHP moving on and me becomming a better programmer, but they do the job!

what's the classic CJ Random Quote...

:) do you really need to ask, everybody, this guy has never heard of the classic CJ Random Quote.... everybody? Oh...

Well here it is in action:

http://www.cj-design.com/downloads/randomquote/demo/

nice site...very nice! ;-)

cheers mate, its not often you post on these forums and recieve appraisals.

:winkwink:

hehe...:1orglaugh

I downloaded and installed it a short while ago. I will let you know my results in a week or so. So far, I like what I see, as you have consolidated a lot of my cron's and hodge podge of shell scripts into one unit.

Update the install .html with:

If you are executing this file using SSI, you must add the location of your PHP binary to the top of the googlebot.php (i.e. #!/usr/local/bin/php), and googlebot.php must be CHMOD 755.

The one thing that I thought was kind of ->un-cool<- was you added output to the html pages via googlebot.php. So upon execution, the bot will see:

CJ GoogleBot Activity V1.0 is running on this site
<!-- CJ GoogleBot Activity V1.0 is running on this site -->

I am cool with self promotion, but not in the search engine game. If the above was used as an indicator by the Google people for blacklisting or deeper inspection, a large number of people would be penalized unknowingly.

My comments, should always be deemed as constructive criticism, I am not in the game of slamming people's work.

-dd

What would I do with this information if I used the script?

how stable is that script?

where to download it ?

This too should not be looked upon as flaming. What is the purpose of having a self-promotion tagline when the "stats" aren't publicly displayed?

Hi dd,

Thanks for the installation file addition.

As for the tag line (html comment) - that was left in by mistake, It was put there just to test the thing was being included and I forgot to take it out - so ive also changed that.


AdultNex - basically just repeated what dd said, so same to you really - its sorted now.


Read previous posts doofas

pretty stable, not to be uses on sites that get crawled 500+ times a day (unless your gonna clear your own log every 30 minutes or something)

It was written for two reasons (my own reasons):

A) To monitor GoogleBot (how it acts)

B) To monitor the pages of my site that were crawled

What I found out...

A) Googlebot (usually crawler2) will pick up a link, say from hotscripts.com and find its way to my site, crawler4, say, would then come back and crawl that link properly - also found some other stuff like crawler12 is a deep crawler (takes most of your site when it crawls)

B) I found that my recently updated pages were crawled in a matter of minutes (pretty sweet how it knows)

What you can do with the information you get....

A) The same as what I have done
B) Wipe your ass on it

it's free

:thumbsup thnx for sharing man... this is great!