Welcome to WebmasterWorld Guest from 54.211.227.36

Forum Moderated by: Ocean10000 & incrediBILL & keyplyr

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
No, Spider! Stay!
What does it mean when they grab your robots.txt page and then leave?
4 brina 5:33 pm Nov 22, 2002
Who is this?
Just grabbed my whole site twice
9 Powdork 5:05 pm Nov 22, 2002
WebEco/1.0
Anyone know what it is?
2 Kerrin 7:10 am Nov 22, 2002
phpgetter
strange activity from 208.17.76.138
4 PandaM 3:12 am Nov 22, 2002
New one to me Quibot 3.0
3 cmoewes 8:59 pm Nov 21, 2002
NetNoseCrawler/v1.0
no robots.txt
2 Finder 8:36 pm Nov 21, 2002
"HTML Text Download Class" Agent
Does not respect robots.txt
3 carfac 4:35 pm Nov 21, 2002
Sneaky Fast Spider
4 Josk 4:33 pm Nov 21, 2002
Anyone know this bot?
216.55.138.108
8 Weblamer 10:10 pm Nov 20, 2002
AV Scooter 1.0
images
16 wilderness 11:35 pm Nov 19, 2002
server1.business2www.com b2w/0.1
9 Brett_Tabke 10:10 am Nov 18, 2002
Scooter 3.2 IS Ruining My Marriage
Someone tell me what is going on please
9 brina 6:07 pm Nov 17, 2002
Does anyone know anything about these three user agents?
Download Ninja 7.0, Iria/1.07a, and Pockey/4.10.2(Win32; GUI; ix86)
3 GaryK 3:12 pm Nov 17, 2002
webcollage/1.87
no robots.txt
7 bull 5:15 pm Nov 16, 2002
IBM Crawl_Application
198.81.209.19
2 WebGuerrilla 12:54 am Nov 16, 2002
Links 2.1pre3
legitimate browser
2 Finder 9:41 pm Nov 15, 2002
curl/7.7.1
no robots. grabbed my odp dir
7 bull 11:05 pm Nov 13, 2002
New Spam bot?
7 willtell 2:24 am Nov 13, 2002
What in the world is this?
someone requesting www.intel.com and got 200 code.....
12 kstprod 9:25 pm Nov 12, 2002
parallelContextFocusCrawler1.1
not a good guy
2 carfac 8:57 am Nov 12, 2002
Help with an IP
6 carfac 5:48 pm Nov 11, 2002
ebay robot hitting hard!
Why?
22 Liane 4:32 pm Nov 11, 2002
attempted to access form mail
8 Busynut 12:17 am Nov 9, 2002
PHP/4.1.2 user agent
What's this?
3 Scooter24 10:41 am Nov 8, 2002
Was this the Googlebot?
Or was it Someone from Google Surfing Our Site?
10 The_Toecutter 7:33 am Nov 8, 2002

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members