| 4:58 pm on Mar 18, 2004 (gmt 0)|
Me too. Posted a new thread too, but mods haven't released it. Maybe they won't now, since one is already started.
It does seem to obey the robots.txt file.
| 5:02 pm on Mar 18, 2004 (gmt 0)|
| 5:05 pm on Mar 18, 2004 (gmt 0)|
So because a rarely-seen spider didn't pull your particular .js files, you assume it obeys robots.txt?
| 5:10 pm on Mar 18, 2004 (gmt 0)|
Yes, I have GoogleBot/Test spidering several times my .js files.
| 5:26 pm on Mar 18, 2004 (gmt 0)|
|So because a rarely-seen spider didn't pull your particular .js files, you assume it obeys robots.txt? |
1. The bot is not at all rare, see several threads in f39, too.
2. Yes, from my observations I assume it obeys robots.txt
3. You -or anyone else- need to prove the contrary. No respect for robots.txt is commonly regarded as something, let's say, unethical for robots. In dubio pro reo.
| 6:08 pm on Mar 18, 2004 (gmt 0)|
are any of you using adwords? It could checking that you don't have popups...
| 6:13 pm on Mar 18, 2004 (gmt 0)|
| 6:16 pm on Mar 18, 2004 (gmt 0)|
I wonder if this will put an end to the js redirected doorway pages that seem to be doing so well now.
| 6:47 pm on Mar 18, 2004 (gmt 0)|
| 6:53 pm on Mar 18, 2004 (gmt 0)|
Log all your browser requests with a proxy/sniffer.
| 6:55 pm on Mar 18, 2004 (gmt 0)|
this bot has come around 5 times since 3/4, it pulled robots.txt the first time only, last visit was 3/16.
Requests just one page, same one all 5 times.
only js in the file is a third party counter.
This is a very old page that hasn't been changed/updated in over a year except for site wide navigation changes.
GG did say he wasn't familiar with this bot and was going to check it out, but AFAIK he hasn't mentioned it again.
| 6:55 pm on Mar 18, 2004 (gmt 0)|
The ones I have seen use onmouseover, so load the page with the cursor outside the window.
| 7:05 pm on Mar 18, 2004 (gmt 0)|
fearing that my language-based redirect will be interpreted as spam, therefore scripts disallowed. Hope there is no penalty on disallowed scripts.
| 7:12 pm on Mar 18, 2004 (gmt 0)|
Maybe they're looking for those automatic popups used by frauds to set affiliate link cookies, even if the visitor doesn't click on the link.
But why would the search engine care if one affiliate steals commissions from another one? Wouldn't they say, "hahha, serves them right!".
| 7:19 pm on Mar 18, 2004 (gmt 0)|
| 7:21 pm on Mar 18, 2004 (gmt 0)|
Yeah, I think a witch hunt about spiders visiting .js files would be a red herring. Maybe they're just looking for the deep web.
| 9:17 pm on Mar 18, 2004 (gmt 0)|
Or they could try to improve on their abilities to check wether a browser gets different content than the bot. This could - like powdork mentioned - very well put an end to js-redirected doorways and would in most cases improve serps - which is the communicated goal if I am not mistaken.
Just my thoughts
| 12:42 pm on Mar 20, 2004 (gmt 0)|
Well, this bot keeps coming back and is bloating my log files so I did try to exclude it in robots.txt but it doesn't check robots.txt anymore (it did at first) and just keeps sucking up bandwidth.
Would someone please teach this bot some manners!
| 1:21 pm on Mar 20, 2004 (gmt 0)|
| 1:36 pm on Mar 20, 2004 (gmt 0)|
If you want to find out which is happening, just place a few dummy functions in your js files. In those dummy functions open a window using an url that does not exist made up of two strings.
It might be necessary, to be certain, to connect the dummy functions to a small links somewhere on the page.
Hopefully, someone will be able to provide an answer rather than just speculate and ruminate.
| 4:27 pm on Mar 20, 2004 (gmt 0)|
I have reported some sites that would (I would think) warrant a manual penalty because they are polluting the serps with pages that don't exist. My guess is that they haven't been removed by hand because Google wants to catch them with their automated technology. That leads me to believe they may be planning on executing the js.
[edited by: Powdork at 4:37 pm (utc) on Mar. 20, 2004]
| 4:34 pm on Mar 20, 2004 (gmt 0)|
Does this mean the pages will now be spidered with this new bot?
| 4:47 pm on Mar 20, 2004 (gmt 0)|
GG has made mention in the past that Googlebot will follow links in js.
| 6:15 pm on Mar 20, 2004 (gmt 0)|
Enough reasons for me to say that you're either wrong or simply just trying to blow another bubble.
Regarding GBot/Test ignoring robots.txt: it wouldn't surprise me. Doh! A bot that doesn't identify itself with a user agent that contains correct contact info and/or correct whois info +plus+ GoogleGuy claiming that he doesn't know anything about a Testbot ... sorry, but this stinks ...
| 7:37 pm on Mar 20, 2004 (gmt 0)|
Check the IP address... it is not google. It is some guy who has named his browser :Googlebot: Ban him from your site and move on.
| 8:07 pm on Mar 20, 2004 (gmt 0)|
|Check the IP address... it is not google |
| 12:08 am on Mar 21, 2004 (gmt 0)|
If you've read many of my posts you would know that I very rarely make definitive statements. When I do, you can bet your life that I can back them up.
When you have read and verified the stickymail I am about to send you, I trust you will apologise for calling me a liar.
| 2:05 am on Mar 21, 2004 (gmt 0)|
Now now, let's keep it NICE.
There are a number of reasons both of you could experience what you have experienced and neither one would be wrong.
Google has made some MAJOR changes in the last month or so in the links they crawl and index. Probably the biggest change in two years from what I monitor.
| 4:10 am on Mar 21, 2004 (gmt 0)|
| This 58 message thread spans 2 pages: 58 (  2 ) > > |