Welcome to WebmasterWorld Guest from 23.20.147.6

Forum Moderated by: Ocean10000 & incrediBILL & keyplyr

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
bdcindexer
didn't grab much, but....
1 carfac 5:52 pm Dec 20, 2002

5 lazyz 10:14 am Dec 20, 2002
libwww-perl/5.63 bot from tracerlock.com
libwww-perl/5.63 not just from Inktomi or FAST
2 Falcon 1:05 am Dec 20, 2002
im4u
oh, really?
2 bull 1:00 am Dec 20, 2002
Why does AV crawl with Scooter AND Mercator
please explain
2 jec2002 4:26 am Dec 19, 2002
Indy Library Explained
Indy Library is a programming toolset that can be anything.
2 WarDead 4:10 am Dec 19, 2002
Found a great User Agent Database & Tool
user agent database reverse dns reverse IP
1 FoodPlaces 11:07 pm Dec 17, 2002
Rumours-Agent
Site downloader?
6 pendanticist 7:19 pm Dec 17, 2002
DA5.0 come to my site and downloal stuff. I don mind traffic, but ..
2 Penguinsnob 7:17 pm Dec 17, 2002
Google bot won't crawl all the page
google bot only crawls certain parts of my site
1 Penguinsnob 4:00 pm Dec 17, 2002
Page Beamer
Web Accelorator
2 carfac 8:09 pm Dec 16, 2002
Hungry Bot
16 menyak 1:35 pm Dec 16, 2002
ArcNet - over 1000 requests?
7 billdaly 8:14 pm Dec 15, 2002
Summation Script
2 DavidT 7:40 pm Dec 15, 2002
links manager robot Identity
3 Need_More_Hits 3:18 pm Dec 15, 2002
Mysterious User Agents
[3] ( 1 2 3 )
76 ratman 4:17 pm Dec 14, 2002
FreeFind.com viral marketing
banned at the firewall
10 Finder 7:52 pm Dec 13, 2002
lcabotAccept
who does this belong to?
7 Finder 7:40 pm Dec 13, 2002
Mercator and Altavista?
4 HuhuFruFru 7:12 pm Dec 13, 2002
Szukacz violating robots.txt
Polish search engine
2 Finder 7:07 pm Dec 13, 2002
Coincidence?
3 wilderness 6:39 pm Dec 13, 2002
4.38.68.69 unknown crawler
3 Orange_XL 8:25 am Dec 13, 2002
Python-urllib/2.0a1
5 troels_nybo_nielsen 3:51 pm Dec 12, 2002
attempts on form processing
why does this occur
3 sfielder 4:44 pm Dec 11, 2002
Unknown bot - amzn_assoc
anyone sen it?
25 upside 9:59 am Dec 11, 2002