184.108.40.206 ua Mozilla/4.0+(compatible;+MSIE+5.0;+Windows
| 3:59 pm on Nov 29, 2000 (gmt 0)|
Crawled some of my stuff, I thought it was a surfer, but I didn't see it come from any engine, and the amount of stuff it grabbed really through me for a loop, especially the regularity of the requessts.
Does any one know if this is a spider, or if some surfer just couldn't get enough of my redirect system, and had to click on everything?
| 4:22 pm on Nov 29, 2000 (gmt 0)|
220.127.116.11 - 18.104.22.168
U S WEST Interact Services
600 Stinson Blvd NE
Minneapolis, MN 55413
U S WEST ISOps firstname.lastname@example.org
Of course you already knew this.
Does it usually mean when the host is unreachable that it most likely is a dial-up account?
| 4:32 pm on Nov 29, 2000 (gmt 0)|
I have a program (teleport pro) that I can run from my desktop which spiders an entire site and doesn't leave a referrer if you want to remain anonymous. It will follow links up to 3 deep. It will leave your dial-up ip address though. Perhaps this guy is using the same.
| 4:35 pm on Nov 29, 2000 (gmt 0)|
Dial up accounts, and dsl accounts, I believe look a little different, but then, I'm certainly not an expert.
The reason I posted this was the activity on my logs...it seemed more like a spider than a surfer, and I handed it the information I give to web surfers, and not to spiders...
What I've been doing is custom serving different entities, and since this "user" didn't seem to come from a search engine, I got confused, since it seemed like a surfer with the ip/ua combo it gave...but then when it grabbed so much stuff...makes me wonder.
Perhaps the gods of seo will be along shortly to help us...;)
| 4:12 am on Nov 30, 2000 (gmt 0)|
What I was trying to say here is that there is a program which will come to your site and look like a spider. It gives no UA (if you don't want it to) and no referrer. The only way to determine whether it is a search engine or a surfer running his own bot is by ip. As long as you are cloaking by ip delivery, you are OK.
| 4:13 am on Nov 30, 2000 (gmt 0)|
Edited by: eljefe3
| 2:50 pm on Nov 30, 2000 (gmt 0)|
Thanks. Teleport pro? I'll have to make sure I follow up on that. I've been filtering by all I can, usually both, sometimes one, sometimes the other, depending on how easy the data appears...
With some of the engines, surfers doing odd things, I've begun to get very paranoid...after a while, if anything comes through and looks even slightly abnormal, I spend at least an hour tracking it down, checking the ip, and trying to assure myself I didn't just redirect the latest version of slurp :)
Thanks again for your help.