Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: open
Did anybody see something like that in the logs? I tried a search here at WebmasterWorld and didn't turn up anything. Nor did I see it before in our logs.
This happened a day after a full crawl of our sites, although we have the respective *.js in place since the beginning of the year.
Im curious about what 'Googlebot/Test' is up for.
It does seem to obey the robots.txt file.
So because a rarely-seen spider didn't pull your particular .js files, you assume it obeys robots.txt?
Requests just one page, same one all 5 times.
only js in the file is a third party counter.
This is a very old page that hasn't been changed/updated in over a year except for site wide navigation changes.
GG did say he wasn't familiar with this bot and was going to check it out, but AFAIK he hasn't mentioned it again.
Just my thoughts
If you want to find out which is happening, just place a few dummy functions in your js files. In those dummy functions open a window using an url that does not exist made up of two strings.
It might be necessary, to be certain, to connect the dummy functions to a small links somewhere on the page.
Hopefully, someone will be able to provide an answer rather than just speculate and ruminate.
[edited by: Powdork at 4:37 pm (utc) on Mar. 20, 2004]
Enough reasons for me to say that you're either wrong or simply just trying to blow another bubble.
Regarding GBot/Test ignoring robots.txt: it wouldn't surprise me. Doh! A bot that doesn't identify itself with a user agent that contains correct contact info and/or correct whois info +plus+ GoogleGuy claiming that he doesn't know anything about a Testbot ... sorry, but this stinks ...
There are a number of reasons both of you could experience what you have experienced and neither one would be wrong.
Google has made some MAJOR changes in the last month or so in the links they crawl and index. Probably the biggest change in two years from what I monitor.