ciml
msg:57228 | 1:57 pm on Jul 19, 2004 (gmt 0) |
Thanks Panze, it looks like it was Tuesday or Wedensday last week.
|
Chndru
msg:57229 | 2:18 pm on Jul 19, 2004 (gmt 0) |
[robotstxt.org...] needs update?
|
Patrick Taylor
msg:57230 | 2:32 pm on Jul 19, 2004 (gmt 0) |
It switched for me on 13th July (server in California).
|
ThomasB
msg:57231 | 10:13 pm on Jul 19, 2004 (gmt 0) |
Wow, the second change in G that resulted by the idea of one black hat WebmasterWorld member. Find out the other one as I don't want to mention the name. ;)
|
adfree
msg:57232 | 8:23 am on Jul 20, 2004 (gmt 0) |
There is no change needed in my robots files if I do not list this bot for any disallowments, is there?
|
ciml
msg:57233 | 8:30 am on Jul 20, 2004 (gmt 0) |
adfree, even if you do mention Googlebot in your /robots.txt you should have no problem. [google.com...]
| When deciding which pages to crawl on a particular host, Googlebot will obey the first record in the robots.txt file with a User-Agent starting with "googlebot". If no such entry exists, it will obey the first entry with a User-Agent of "*". |
|
It's the URI portion of the User_Agent header that's changed, so people using "user_agent cloaking" that relies on matching the entire string may be caught out by this.
|
DaveN
msg:57234 | 10:03 am on Jul 20, 2004 (gmt 0) |
sweet that is so funny!
|
volatilegx
msg:57235 | 10:22 pm on Jul 20, 2004 (gmt 0) |
We've been talking about it for days. [webmasterworld.com...]
|
yowza
msg:57236 | 10:55 pm on Jul 20, 2004 (gmt 0) |
Here's an even older thread with some comments by GoogleGuy:
[webmasterworld.com...]
that I found today when I noticed the change. I was surprised to see it has been happening since March 3rd because I didn't see any change until July 9th.
|
ciml
msg:57237 | 10:11 am on Jul 21, 2004 (gmt 0) |
Thanks volatilegx, you guys certainly beat us to that one. That's interesting yowza, I see the same as you, only later.
|
Jesse_Smith
msg:57238 | 3:59 am on Aug 15, 2004 (gmt 0) |
What do I change
if ($ENV{'HTTP_USER_AGENT'} =~ /googlebot/) {
|
|
to? 7/28/04 is the last day that code generated any logs.
|
Jesse_Smith
msg:57239 | 4:22 am on Aug 15, 2004 (gmt 0) |
*Tries changing it to
if ($ENV{'HTTP_USER_AGENT'} =~ /google/) {
|
|
* and with in minutes Google shows up in the log!
|
ciml
msg:57240 | 5:27 am on Aug 15, 2004 (gmt 0) |
Jesse, I think that's because Googlebot has a capital "G", while www.google.com or www.googlebot.com is in lower case in Google's user_agent.
|
|
