GoogleBot didn't visit me after 23rd

Anyone same here?

[jamesyap]

I have a new site and GoogleBot seems to visiting my site almost everyday since the start of december. But suddently it stop on 23rd and never come back. No even christmas. Anyone has the same problem.

I recall I add a robot.txt at that period that ask all robots to exclude a specific file which do meta refreshes to sites located outside. I just don't think this robot.txt is the main reason that stop google from coming back. Anywhere, I have remove it.

Just want to see if anyone facing the same problem here.

[jamesyap]

hi javascripter,

your 'index & follow' is valid and I use it last time with my site. But nowadays, pages without robot.txt and that tag is consider to have the tag on. :) Youd don't need to change any code to get listed.

[mfishy]

Googlebot is now crawling my sites. First time in days.

[tristan]

yep, also spidering here again :)

[Danielo21]

That means that the google dance is far away again, doesn't it?

[xlcus]

Googlebot is now crawling my sites. First time in days

Freshbot or standard bot? (What do people here call it?). Freshbot just hit one of my sites in the last 30 minutes. First time since the 23rd.

[h_b_k]

Yes, fresh bot is back crawling my site.

I would prefer it, if the bot would start to crawl the pages every 3rd day only. No idea, why it was crawling dayly for weeks about 100% of pages while there is a 3 day rhythm for freshness!?

I am using SSI and therefore my sites have no real modification date.
Now I set a meta tag
<meta name="revisit-after" content="3 days">
to the pages I often change and
<meta name="revisit-after" content="14 days">
<meta name="revisit-after" content="28 days">
to other pages.

hope, the bot would respect my humbly wishes. ;-)

[edited by: h_b_k at 5:38 pm (utc) on Dec. 30, 2002]

[jamesyap]

Hey, how do you guys know google bot came by? from the log files? or the fresh tag in google.com? I din see my site appearing. :(

[h_b_k]

from my log files

[javascripter]

I'm a little confused about that too. What does it say in the log files when googlebot hits you? Is a particular IP address? And do I have to look in my logs or can I look in my sitemeter? My site meter shows me who's hitting the site and from what place they are being referred by.

Ken

[Danielo21]

You have to look in your logs

[h_b_k]

no, there is no special referrer url to identify google bots

you have to look at your log files

more answer for you at
[webmasterworld.com...]

[jamesyap]

Ya I saw it in my log files :) 30 december. Happy although it crawl only the main page!

Javascripter, log files means apache server log files, you will usually have them available if you have a paid hosting, either download from the FTP (if Plesk control panel) or from the control panel (using CPanel).

For free hosting, usually there are no server logs. All web tracking tools will not show you if googlebot have reach u.

Log file has entry something like this -
64.68.82.14 - - [30/Dec/2002:04:29:53 -0800] "GET / HTTP/1.0" 200 - "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

[tristan]

check your logfiles, you can search for the string "googlebot"
in my logs it looks like this:

64.68.82.67 - - [30/Dec/2002:20:34:35 -0500] "GET / HTTP/1.0" 200 1540 "-" "Goog
lebot/2.1 (+http://www.googlebot.com/bot.html)"

64.* is the freshbot, the monthly index bot is somewhere 213.*

[h_b_k]

no tristan, not 213.*

fresh bot ip:
64.68.82.*

deep crawler:
216.239.46.*

[jamesyap]

Please shout in this forum when you guys see the fresh tag appearing in google search! I want to see my site reappearing in no time

[javascripter]

Thanks for all your help.

[Danielo21]

same here, googlebot visited me 6 hours ago today

[pardo]

Seems like the Freshbot did reach the other side of the ocean as well. Green flag from Holland for two websites i can confirm a visit for a couple of pages just some hours ago.

[jamesyap]

Funny, fresh bot came twice on 30th and 31st while 216.x.x.x bot came on late at night of 30th. They just take away the robot.txt and main page. I am wondering why 216 is coming ... it had came before on 10th. So? Does this sounds like there are no dance for december?

Anyway, it is already 2003 in my area. For those still 2002, I am one year ahead of you! :) Happy New YeaR ...