Indexing pages

Does google index my pages every time it crawls?

[hthota]

I have a date stamp on my pages and can see the last time google indexed the page. I also subscribe to Googlebot alerts and get an e-mail notification everytime the spider crawls on the page.

One of my pages was last indexed on Dec 10th. I got an alert that showed that the spider crawled on my page on Dec 31st. But the page still hasn't been indexed after 31st. Does google index my pages every time it crawls?

[dirkz]

> Does google index my pages every time it crawls?

No. Sometimes it just looks for changes, only to come back later.

[ThomasB]

It may take several days to weeks after spidering till the updated/new page is shown in Google. No reason to worry.

[gators]

How and where do you subscribe to Googlebot alerts?

[thehittmann]

I was thinking the same thing as Gators, could we have more info on Googlebot alerts please.

[dirkz]

How does it work?

[thehittmann]

you have to upload the page to your server and then place a link to the file on your main/index page of your site. When the spider grabs the file it emails you the user agent.

Good in theory......but I couldnt get the test to work so I gave up on it.

[NickCoons]

It shouldn't be difficult to have something in your pages that checks the user agent and then reports back. If you have a moderately-sized site, though, you'll likely be crawled often. For instance, Google pulls about 50 pages per day from one of my sites, and I know it happens. I definately don't want to be emailed each time Googlebot visits.

I have it setup so that it checks the user-agent and categorizes the impressions. Then I can look at a table throughout the day that shows page views by users, Googlebot, Ink, etc. Seems the easiest, and least intrusive way.

[ThomasB]

maybe a little off-topic, but here's a quick and dirty php hack:

<?php
if (strstr($_SERVER['HTTP_USER_AGENT'], "Googlebot")) {
$body = "Googlebot just visited " . $_SERVER["PHP_SELF"];
mail("[e-mail]", "Alert", $body);
};
?>

[JoeyBall]

How often should googlebot acess i site. I am getting google around every 2 to 3 days. Should i get worried?

[ThomasB]

it's fine.
Be happy, other sites get spidered just about once a month.

[dirkz]

> you have to upload the page to your server and then place a link to the file on your main/index page of your site. When the spider grabs the file it emails you the user agent.

I guess you need PHP for this to work? My second guess: it only works when Googlebot follows the link?

[ThomasB]

I guess you need PHP for this to work?

For my solution: Yes. For the other solution I don't know.

My second guess: it only works when Googlebot follows the link?

It looks like that for the other solution. And then you also don't know when google spidered your site, just when google spidered their site, which was linked from yours. I'd go for a php/perl/cgi/... solution on your server. Then there's no doubt and it should work very reliable.

[thehittmann]

No the other method is also php and sends back when about 10 different SE's spider your site. The php file is set up kind of like the one posted in here. But It wouldnt work for me, guess my server doesnt allow php. Never done anything php before so i have no idea.

[g1smd]

You can easily test to se if you have PHP on your server.

Put this in a file called test.php as a text file:

<?php echo("HELLO");?>

Upload that file to your webserver and use your browser to look at that page.

If you see just HELLO then PHP is working.

If you see <?php echo("HELLO");?> then it is not.

[thehittmann]

thx for that, i'll give that a try now

[thehittmann]

hmmm it doesnt do either of what you said it just asks me to try and download the file.

[g1smd]

That also means that they have not set up the server so that .PHP files can be parsed. The PHP extansion seems to have some other MIME type than the one that is required. It should be text/html to work as expected.

The server is just sending the content as a file, and your browser then wants to download and store it, rather than analyse it and then display the contents.

[Kwix]

I have a defacto notification setup. On a custom 404 I have it set up to email me when somebody(user or user-agent) hits a non existant page. Since Google seems to search for pages long after they have disappeared I get a sort of notification of when this occurs. Also, SLURP (Inktomi) keeps hitting the same pages over and over. Oddly, FAST seems to get it right on the first couple of tries.