Forum Moderators: open
crawler10.googlebot.com - - [07/Apr/2003:10:57:20 -0700] "GET /1Crim.html HTTP/1.0" 301 258 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
crawler10.googlebot.com - - [08/Apr/2003:02:11:12 -0700] "GET /robots.txt HTTP/1.0" 200 188 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
crawler10.googlebot.com - - [08/Apr/2003:02:11:13 -0700] "GET / HTTP/1.0" 200 - "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
These are the only two requests.
Pendanticist.
Last update frehbot continued to visit my sites as per norm - however during the update the fresh tags disappeared from www but were showing in www2 - www3 only.
BTW - Cant see fresh dates at the moment in www,www1 or www2 (of course that happens sometimes anyway)
Note - I cant see fresh tags - but I can see fresh pages - strange
Just a thought.....
It drove me crazy - one of my sites had a complete re-design days before the last update and was getting fresh results in www2 and www3 but when the update had completed freshbot was not going so deep and the results never showed in www.
I'm new to this game. How can I check to see if the freshbot is visiting my site?
The two posts just below yours contain two examples of googlebot UA (User Authorization) strings.
To check them (UA strings) you'll need to be able to read your access_log files. Do you currently have access to them? Check with your host server to see just what you are entitled to use.
Access_Log Files house every impression every visitor (human or otherwise) makes.
Some stats programs/application either provided by your host server or yourself (in certain cases) show similar data - but not as detailed as those you can inspect yourself via access_log files.
Doing a Site Search [webmasterworld.com] using 'access_log files' pulls up some additional readings.
Pendanticist.
access_log.3:216.239.46.82 - - [20/Mar/2003:19:11:25 +0100] "GET /zblue/index.htm HTTP/1.0" 200 10441 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)
Googlebot crawled 144 pages ... how long do they take to include it in the database? .. Do I have to expect another crawl this month before I'm included in the search database?