Welcome to WebmasterWorld Guest from 54.162.211.233

Forum Moderated by: Ocean10000 & incrediBILL & keyplyr

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
Spider from multiple IP's all at once
Spider from multiple IP's all at once
5 sammy 6:13 am Dec 29, 2002
I need help building a list
I am building a list of spiders that we should let in and would like help
6 anoryu 6:10 am Dec 29, 2002
Teradex Mapper violating robots.txt
in fact, they never bothered to check
2 Finder 6:08 am Dec 29, 2002
Fast PartnerSite Crawler Question
4 Jon_King 2:49 pm Dec 24, 2002
Microsoft URL Control
A warning and a request for information
29 Crazy_Fool 12:45 am Dec 23, 2002
Universal Payment Gateway
UP.Browser/3.2.9.1-UPG1 UP.Link/3.2
4 wilderness 7:50 pm Dec 22, 2002
Comscore Networks
216.148.246.70
1 wilderness 4:52 am Dec 22, 2002
New Spam Bot?
Hit by "Lite Bot 0816b"
5 121focus 10:03 am Dec 21, 2002
bdcindexer
didn't grab much, but....
1 carfac 5:52 pm Dec 20, 2002

5 lazyz 10:14 am Dec 20, 2002
libwww-perl/5.63 bot from tracerlock.com
libwww-perl/5.63 not just from Inktomi or FAST
2 Falcon 1:05 am Dec 20, 2002
im4u
oh, really?
2 bull 1:00 am Dec 20, 2002
Why does AV crawl with Scooter AND Mercator
please explain
2 jec2002 4:26 am Dec 19, 2002
Indy Library Explained
Indy Library is a programming toolset that can be anything.
2 WarDead 4:10 am Dec 19, 2002
Found a great User Agent Database & Tool
user agent database reverse dns reverse IP
1 FoodPlaces 11:07 pm Dec 17, 2002
Rumours-Agent
Site downloader?
6 pendanticist 7:19 pm Dec 17, 2002
DA5.0 come to my site and downloal stuff. I don mind traffic, but ..
2 Penguinsnob 7:17 pm Dec 17, 2002
Google bot won't crawl all the page
google bot only crawls certain parts of my site
1 Penguinsnob 4:00 pm Dec 17, 2002
Page Beamer
Web Accelorator
2 carfac 8:09 pm Dec 16, 2002
Hungry Bot
16 menyak 1:35 pm Dec 16, 2002
ArcNet - over 1000 requests?
7 billdaly 8:14 pm Dec 15, 2002
Summation Script
2 DavidT 7:40 pm Dec 15, 2002
links manager robot Identity
3 Need_More_Hits 3:18 pm Dec 15, 2002
Mysterious User Agents
[3] ( 1 2 3 )
76 ratman 4:17 pm Dec 14, 2002
FreeFind.com viral marketing
banned at the firewall
10 Finder 7:52 pm Dec 13, 2002

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members