Welcome to WebmasterWorld Guest from 54.161.25.142

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
Grub Crawler has new name
4 Josk 7:57 pm Feb 11, 2002
I want MATTIE's IP Address!
Can anyone provide the answer?
6 hellier 5:28 pm Feb 11, 2002
Longest UserAgent
why-dont-u-you-just-go-away-and-crawl-somewhere-else?
13 Josk 1:20 pm Feb 9, 2002
Inktomi/Goo Spider
7 msgraph 11:59 pm Feb 6, 2002
iaea.org spider
new theory
8 yobb 7:41 pm Feb 5, 2002
ia_archiver not honoring robots.txt
anyone else having same problem
6 Marshall 3:11 pm Feb 3, 2002
somebody knows this : Szukacz
6 Steph 6:03 pm Feb 2, 2002
Slurp/si - Slurp/cat
8 skirril 3:51 am Feb 1, 2002
A couple of new Wisenuts and an Ask Jeeves
3 volatilegx 4:37 pm Jan 31, 2002
Interesting article on blocking bots
2 volatilegx 2:57 pm Jan 31, 2002
New Alta UA?
1 msgraph 1:34 pm Jan 31, 2002
Overture Spider?
4 Josk 2:50 am Jan 31, 2002
National Directory, Business.com, Splatsearch.com ?
Does anybody know the names or IPs of their spiders?
2 snark 1:48 am Jan 31, 2002
Scooter Spider is Confused?
filling my error logs with junk
3 idiotgirl 1:07 am Jan 31, 2002
Nice js solution for email harvesters
6 littleman 3:29 pm Jan 30, 2002
Pesky bot/snooper with a bogus referrer.
13 littleman 1:28 am Jan 30, 2002
Nico
Anybody seen this one?
2 wilderness 5:35 pm Jan 27, 2002
New AV IP
out of Colorado
6 msgraph 12:58 am Jan 25, 2002
Crawler able to follow javascript generated links
4 bottester 5:02 pm Jan 24, 2002
How to kill the Mattie MP3 Spider
This Spider is eating my bandwidth
3 hellier 3:43 pm Jan 23, 2002
cr009r01.sac2.fastsearch.net who is this please
cr009r01.sac2.fastsearch.net who is this please
9 rilee12 10:11 am Jan 22, 2002
About/0.1libwww-perl/5.47
7 Marcia 12:03 am Jan 20, 2002
new IP range for Fast spider
4 PeteU 3:13 pm Jan 19, 2002
Whatever happened to WiseWire?
Are they dead?
3 volatilegx 11:09 pm Jan 15, 2002
A flurry of New bots (for me)
over the past few days
5 wilderness 5:46 pm Jan 15, 2002

Featured Threads

Hot Threads This Week

Hot Threads This Month