Welcome to WebmasterWorld Guest from 54.145.213.148

Forum Moderated by: Ocean10000 & incrediBILL & keyplyr

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
Who's Libby
and why does she want to be my friend?
3 idiotgirl 8:09 pm Jul 1, 2002
Is this valid robots.txt?
valid robots.txt * asterisk disallow
3 pwc21 3:36 pm Jul 1, 2002
display error pages to banned users
.htaccess too aggressive
6 idiotgirl 7:04 pm Jun 30, 2002
Spider? choking on javascript
Huge increase in 404's
2 Robert_Scott 10:20 am Jun 29, 2002
Possible Spambot wih user agent "MFC Library"
2 Bluestreak 10:39 pm Jun 27, 2002
Annoying and unusual bot activity from other countries
3 Bluestreak 2:43 pm Jun 27, 2002
List of recent spiders
Unknown to me
10 Thierry_Zoller 9:24 pm Jun 26, 2002
VB OpenUrl
Anyone seen this one?
3 jdMorgan 8:04 pm Jun 25, 2002
IBM Almaden Research Center
mystery
5 wilderness 1:15 am Jun 25, 2002
cc0.inktomi.com
ua = libwww-perl/5.49
10 misosoph 10:06 pm Jun 24, 2002
Is this AltaVista's image robot?
qascoot1.sv.av.com
2 misosoph 4:52 pm Jun 21, 2002
IP Lists
[2] ( 1 2 )
36 Brett_Tabke 4:51 pm Jun 20, 2002
FirstGov.gov Spider
7 msgraph 10:26 am Jun 20, 2002
Teradex Mapper from 216.129.82.203
Grabbed my ODP-listed pages yesterday.
4 mbauser2 10:19 am Jun 20, 2002
Anyone know of a spider named Iron
hit my server hard
2 Big_Balou 10:30 pm Jun 19, 2002
New user agent?
Is this new or just new to me?
5 Pushycat 5:08 am Jun 18, 2002
Bot looking for Exploits?
10 Bluestreak 4:49 pm Jun 16, 2002
CS9zuj883s28.1?
CS9zuj883s28.1 user agent
2 Kev 9:19 am Jun 16, 2002
CAP LOCKs Robot from China
Internet Explore 5.x
5 misosoph 9:04 pm Jun 15, 2002
Googlebot Agent Header ¦ Dialup user
Warning to acces_log grep'ers
6 Thierry_Zoller 11:01 am Jun 15, 2002
Is this who i think it is?
7 Dpeper 2:56 am Jun 15, 2002
Another FAST spider?
test?
3 lorax 8:48 pm Jun 14, 2002
How to report a potential Bot abuse?
Would love to have one in my draft folder just in case ;)
2 hanuman 4:15 pm Jun 14, 2002
Reading robots.txt or root directory, then leaving
3 Bluestreak 4:02 pm Jun 14, 2002
New company Name
Any Ideas for our new company name?
2 canucks1980 8:07 pm Jun 13, 2002

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members