Forum Moderators: open
All the sites that I have access to here are all deep crawl by the freshbot 64.* don't know why.
The other one came and grab index page once awhile.
My own very new site was deepcrawl by the freshbot for sure. Never by the regular 216.*.
Also, if it interest anyone here, dynamic IP address doesn't cause any problem. Googlebot came just one to two hours after I reboot my machine. So they do resolve correctly. Problem I face is because my client's site is already down one month before I took over. That is why it disappear from the index.
We still have sites down from the worm... thanks a bunch Google.
Last hit from the bot - well after the update started?
64.68.82.14 - - [26/Jan/2003:04:54:49 -0600] "GET /science_education_software/software_reviews/oceans.html HTTP/1.0" 304 - "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
Still crawling
64.68.82.14 - - [26/Jan/2003:07:07:11 -0500] "GET /webpagename.htm HTTP/1.0" 200 9445 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
What's REALLY strange is the site got scanned this morning by
all three major engines at ONCE:
66.77.73.244 - - [26/Jan/2003:03:05:17 -0500] "GET /robots.txt HTTP/1.0" 200 41 "-" "FAST-WebCrawler/3.x Multimedia (mm dash crawler at fast dot no)"
64.68.82.35 - - [26/Jan/2003:03:16:25 -0500] "GET /robots.txt HTTP/1.0" 200 41 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
66.196.65.21 - - [26/Jan/2003:03:20:44 -0500] "GET /robots.txt HTTP/1.0" 200 41 "-" "Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)"
The regular fast atw did a full scan yesterday too!
zeus
I thought that i was probably just seeing things as it was late. ;)
[edited by: Marcia at 2:40 pm (utc) on Jan. 26, 2003]
There are more than just those 3 databases. At the moment I have changes in 3 of the 7 servers that I'm checking. The dance is over when the results are the same right across the board.
Regarding the freshbot. My News page is in the new index with a Jan 12 snapshot showing in the cache. The new indexing is based on crawls done from early Dec through to mid-Jan. Very cool GG.