Forum Moderators: goodroi
I have been visiting this site over the past couple of months as I am constantly monitoring my site logs and watching for the next or newest bandwidth thieves to come along.
This listing nails off quite a number of standard issue spam e-mail harvesters, site rippers and dumb search engines that gulp tons of your bandwidth and yet they will never return any visitors because no one goes to their 'directories'.
The only search engines that I have not deliberately blocked are Google, Microsoft, Yahoo, Ask.com and a few others. These are the bread and butter search engines so we really don't want to discourage their indexing of our sites.
To really kill off the ultimate pest in the world, I make use of the .htaccess file. It kills off any of the loser robots that don't want to play friendly and take the 'get lost your not wanted here' of the robots.txt file.
This list has passed the WebmasterWorld Robot Validator test and the REAL WORLD test on my own site... which handles approximately 1 million page views per day.
I will post up my .htaccess file in another message just so you can pick and choose which or both to deploy * or maybe spot some other bots in this list that you have not encountered yet *
# Robots.txt begins here
# Bandwidth thieves to keep out :)
User-agent: (CK-SillyDog)
User-agent: .NET CLR
User-agent: .NET CLR 1.0
User-agent: .NET CLR 1.0.3705
User-agent: .NET CLR 1.1
User-agent: .NET CLR 1.1.4322
User-agent: _iRc
User-agent: 000000000
User-agent: 4all
User-agent: abot
User-agent: Accept
User-agent: accoona
User-agent: AceFTP
User-agent: adressendeutschland
User-agent: Ahead
User-agent: Alarm
User-agent: Alexibot
User-agent: Amfi
User-agent: Anal
User-agent: andit
User-agent: Anon-Encrypt
User-agent: Anon_Proxy
User-agent: Ansearch
User-agent: AnswerBus
User-agent: API Request
User-agent: appie
User-agent: Aqua_Products
User-agent: Arachmo
User-agent: arianna
User-agent: asterias
User-agent: Atlocal
User-agent: Atropos
User-agent: Auto
User-agent: AWV0.8d
User-agent: b2w/0.1
User-agent: bach-idx
User-agent: BackDoorBot
User-agent: Baiduspide
User-agent: Bandit
User-agent: BasicHTTP
User-agent: Bison
User-agent: Black
User-agent: BlowFish
User-agent: BoardPulse
User-agent: BoardViewer
User-agent: boitho
User-agent: Bond
User-agent: Bookmark search tool
User-agent: boris
User-agent: BotALot
User-agent: BotName
User-agent: Brother
User-agent: BuiltBotTough
User-agent: Bullseye
User-agent: BunnySlippers
User-agent: CafeKelsa
User-agent: Cafi
User-agent: Capture
User-agent: Cegbfeieh
User-agent: cfetch
User-agent: Chaos
User-agent: Check
User-agent: Cheese
User-agent: Cherry
User-agent: CherryPicker
User-agent: CherryPickerElite/1.0
User-agent: CherryPickerSE/1.0
User-agent: ChinaClaw
User-agent: CK-SillyDog
User-agent: Clam
User-agent: classified
User-agent: Clush
User-agent: COAST
User-agent: collage
User-agent: Collect
User-agent: Concept
User-agent: Convera
User-agent: Copier
User-agent: Copy
User-agent: CopyRightCheck
User-agent: CoralWebPrx
User-agent: cosmos
User-agent: costa
User-agent: craft
User-agent: crawler
User-agent: Crescent
User-agent: Crescent Internet ToolPak
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: Custo
User-agent: CuteFTP
User-agent: Cyberian
User-agent: DA 7
User-agent: DARK
User-agent: dcbot
User-agent: Diamond
User-agent: Dillo
User-agent: DISCo
User-agent: DISCo Pump 3.2
User-agent: Dispatch
User-agent: Ditto
User-agent: DittoSpyder
User-agent: Download
User-agent: Drecombot
User-agent: DTS Agent
User-agent: Dual
User-agent: dumbot
User-agent: EARTHCOM
User-agent: EasyDL
User-agent: eCatch
User-agent: EdcompBot
User-agent: ELinks
User-agent: Email
User-agent: EmailCollector
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: Enfi
User-agent: Enhancer
User-agent: Enterprise_Search
User-agent: Enterprise_Search/1.0
User-agent: Epiphany
User-agent: EroCrawler
User-agent: ERROR
User-agent: es
User-agent: Express
User-agent: Extract
User-agent: ExtractorPro
User-agent: EyeNetIE
User-agent: FairAd Client
User-agent: FANG
User-agent: fast
User-agent: FAST_Data
User-agent: Faxo
User-agent: FDM
User-agent: Fetch
User-agent: Fetch API Request
User-agent: FindAnISP
User-agent: Finder
User-agent: findlinks
User-agent: Flaming AttackBot
User-agent: Flash
User-agent: Flip
User-agent: Foobot
User-agent: Foto
User-agent: Franklin Locator
User-agent: Franklin Locator 1.8
User-agent: FrontPage
User-agent: futile
User-agent: Fyber
User-agent: Gaisbot
User-agent: Generic
User-agent: geni
User-agent: Geona
User-agent: GetHTML
User-agent: GetRight
User-agent: GetRight/4.2
User-agent: GetWeb
User-agent: Giga
User-agent: Girafabot
User-agent: glx v
User-agent: Go!Zilla
User-agent: GoForIt
User-agent: Googlebot-Image
User-agent: GoSearch
User-agent: Got-It
User-agent: Grab
User-agent: Grafula
User-agent: grub
User-agent: grub-client
User-agent: Gruppo
User-agent: GT::WWW
User-agent: Gulliver/1.3
User-agent: Gypsy
User-agent: hack
User-agent: Harvest
User-agent: Harvest/1.5
User-agent: Hatena Antenna
User-agent: Hatena_Antenna
User-agent: Helmdall
User-agent: heritrix
User-agent: hloader
User-agent: HLoader
User-agent: HMView
User-agent: Holmes
User-agent: holy_teacher
User-agent: HTML_Works
User-agent: HTMLParser
User-agent: HTTP OLE Control v.1.0
User-agent: httplib
User-agent: HTTrack
User-agent: human
User-agent: humanlinks
User-agent: hverify
User-agent: Hyper
User-agent: ia_archiver
User-agent: ia_archiver/1.6
User-agent: ichiro
User-agent: Iltrovatore
User-agent: Image
User-agent: Indy
User-agent: InetURL
User-agent: InfoNav
User-agent: InfoNaviRobot
User-agent: Innerprise
User-agent: InterGET
User-agent: Internet\x
User-agent: Intraformant
User-agent: IRLbot
User-agent: Iron33/1.0.2
User-agent: ISC_Systems
User-agent: ISSpider
User-agent: Jakarta
User-agent: Java
User-agent: Java 1
User-agent: Java/1
User-agent: Java/1.4
User-agent: Java/1.4.1
User-agent: Java/1.4.1_04
User-agent: Java1
User-agent: Jenny
User-agent: JennyBot
User-agent: Jetbot
User-agent: Jetbot/1.0
User-agent: JetCar
User-agent: JoBo
User-agent: JOC
User-agent: JOC Web Spider
User-agent: Jyxobot
User-agent: kashmir
User-agent: Kazoom
User-agent: Kenjin
User-agent: Kenjin Spider
User-agent: Keyword
User-agent: Keyword Density/0.9
User-agent: KnowItAll
User-agent: KRAEHE
User-agent: Kumm
User-agent: larbin
User-agent: Leech
User-agent: LexiBot
User-agent: libWeb/clsHTTP
User-agent: LibWeb/clsHTTP
User-agent: Lincoln
User-agent: LinkextractorPro
User-agent: Linkman
User-agent: LinkScan
User-agent: LinkScan/8.1a Unix
User-agent: LinkWalker
User-agent: Lint
User-agent: List
User-agent: LMQueue
User-agent: LNSpiderguy
User-agent: Locust
User-agent: loop
User-agent: LTH_Browser
User-agent: lwp-request
User-agent: lwp-trivial
User-agent: Lwp-trivial
User-agent: lwp-trivial/1.34
User-agent: LWP::Simple
User-agent: Mac_Finder
User-agent: magic
User-agent: majestic
User-agent: Mammoth
User-agent: Mass
User-agent: Mass Downloader/
User-agent: Mass Downloader/2.2
User-agent: Mata Hari
User-agent: Mata.Hari
User-agent: McBot
User-agent: Mechan
User-agent: Metaspinner
User-agent: Meter
User-agent: Microsoft URL Control
User-agent: Microsoft URL Control - 5.01.4511
User-agent: Microsoft URL Control - 6.00.8169
User-agent: Microsoft.URL
User-agent: MIDown
User-agent: MIIxpc
User-agent: MIIxpc/4.2
User-agent: Mirago
User-agent: MIRE
User-agent: Missigua
User-agent: Mister PiX
User-agent: Mister\ PiX
User-agent: MJ12bot
User-agent: MnoGo
User-agent: Moget
User-agent: moget
User-agent: moget/2.1
User-agent: Mojeek
User-agent: MojeekBot
User-agent: Moozilla
User-agent: mouse.com
User-agent: Mozdex
User-agent: Mozilla/3.0 (Macintosh; I; PPC)
User-agent: Mozilla/3.Mozilla/2.01
User-agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)
User-agent: Mozilla/8.0
User-agent: MS Search
User-agent: MSIECrawler
User-agent: MSProxy
User-agent: muscope
User-agent: MyApp
User-agent: NASA_Search
User-agent: naver
User-agent: Naver
User-agent: Navroad
User-agent: NearSite
User-agent: NetAnts
User-agent: NetCrawler
User-agent: NetMechanic
User-agent: NetSpider
User-agent: netXplorer
User-agent: NetZIP
User-agent: Neutrino
User-agent: NEWT
User-agent: Next
User-agent: NICE
User-agent: NICErsPRO
User-agent: Ninja
User-agent: Nintendo
User-agent: nogoop
User-agent: Nokodo
User-agent: Nomad
User-agent: NPbot
User-agent: Nutch
User-agent: NWSpider
User-agent: Objects
User-agent: Octopus
User-agent: Offline Explorer
User-agent: Offline\ Explorer
User-agent: Offline\ Navigator
User-agent: OmniExplorer
User-agent: Openbot
User-agent: Openfind
User-agent: Openfind data gathere
User-agent: Oracle
User-agent: Oracle Ultra Search
User-agent: Oregano
User-agent: osis
User-agent: pamuk
User-agent: Patwebbot
User-agent: Pavuk
User-agent: PcBrowser
User-agent: PEERbot
User-agent: PerMan
User-agent: PHP/4.
User-agent: PHP_version_tracker
User-agent: Pica
User-agent: PicoSearch/1.0
User-agent: pigs
User-agent: PipeLine
User-agent: Pluck
User-agent: Plucker/Py-1.0
User-agent: Plucker/Py-1.4
User-agent: PocketHTTP
User-agent: PocketHTTP/1.0.2
User-agent: Pompos
User-agent: Preload
User-agent: Privoxy
User-agent: Program_Shareware
User-agent: ProPowerBot/2.14
User-agent: ProWebWalker
User-agent: Prowl
User-agent: Psbot
User-agent: psbot
User-agent: Python
User-agent: Python-urllib
User-agent: Python-urllib/1.0
User-agent: Python-urllib/1.15
User-agent: QueryN
User-agent: QueryN Metasearch
User-agent: Quest
User-agent: Qweer
User-agent: Radiation Retriever 1.1
User-agent: Reap
User-agent: ReGet
User-agent: RepoMonkey
User-agent: RepoMonkey Bait & Tackle/v1.01
User-agent: request
User-agent: Research
User-agent: Ripper
User-agent: RMA
User-agent: Rogue
User-agent: RPT-HTTPClient
User-agent: RSSOwl
User-agent: Sample
User-agent: Sauger
User-agent: savvybot
User-agent: SCAgent
User-agent: Schizozilla
User-agent: Schlong
User-agent: Schmozilla
User-agent: SCrawl
User-agent: script_type
User-agent: searchpreview
User-agent: SeznamBot
User-agent: Shaz
User-agent: Sherlock
User-agent: Shot
User-agent: Simple
User-agent: SiteBar
User-agent: SiteSnagger
User-agent: SiteSucker
User-agent: SiteSucker/1.3.1
User-agent: Sleipnir
User-agent: Sly
User-agent: Smurf
User-agent: Snag
User-agent: Sohu
User-agent: sootle
User-agent: Spade
User-agent: Span
User-agent: SpankBot
User-agent: spanner
User-agent: Speed
User-agent: Spin
User-agent: sproose
User-agent: Spurl
User-agent: Spyder
User-agent: Squid
User-agent: Sqworm
User-agent: Stack
User-agent: Stanford
User-agent: Stanford Comp Sci
User-agent: Stanford CompClub
User-agent: Stanford CompSciClub
User-agent: Stanford Spiderboys
User-agent: Statbot
User-agent: State
User-agent: STEALTH
User-agent: Stress
User-agent: Strip
User-agent: subot
User-agent: Suck
User-agent: Supervi
User-agent: Surf4Me
User-agent: Surfbot
User-agent: SUS
User-agent: suzuran
User-agent: Suzuran
User-agent: Sweep
User-agent: sygol
User-agent: Szukacz
User-agent: Szukacz/1.4
User-agent: TAke
User-agent: Talkro
User-agent: tamu
User-agent: Tarantula
User-agent: Teleport
User-agent: Teleport Pro/1.29
User-agent: Teleport Pro/1.29.1820
User-agent: TeleportPro
User-agent: Telesoft
User-agent: TerrawizBot
User-agent: Test
User-agent: The Intraformant
User-agent: TheNomad
User-agent: Titan
User-agent: ToCrawl
User-agent: toCrawl/UrlDispatcher
User-agent: Todobr
User-agent: Track
User-agent: True
User-agent: True_Robot
User-agent: True_Robot/1.0
User-agent: Tunnel
User-agent: Turing
User-agent: turingos
User-agent: Turnitin
User-agent: Twat
User-agent: Tweak
User-agent: Tygo
User-agent: Ultraseek
User-agent: unknown
User-agent: URL Control
User-agent: URL_Spider_Pro
User-agent: URLy
User-agent: URLy Warning
User-agent: User-Agent:
User-agent: USyd
User-agent: Vagabondo
User-agent: Valet
User-agent: Valid
User-agent: Vamp
User-agent: VCI
User-agent: VCI WebViewer VCI WebViewer Win32
User-agent: Visual
User-agent: Void
User-agent: Voila
User-agent: VoilaBot
User-agent: VoilaBot BETA 1.2
User-agent: vspider
User-agent: W3C
User-agent: w3m
User-agent: walhello
User-agent: Walker
User-agent: Watch
User-agent: Weather
User-agent: Web Image Collector
User-agent: Web.Image
User-agent: WebAuto
User-agent: WebBandit
User-agent: WebBandit/3.50
User-agent: WebCapture
User-agent: WebCapture 3.0
User-agent: WebCatS
User-agent: WebCopier
User-agent: WebCopier v2.8
User-agent: WebCopier v3.0
User-agent: WebCopier v4.0
User-agent: WebCopy/
User-agent: WebCopy/0.98b7
User-agent: webcrawl
User-agent: Webdup
User-agent: Webdup/
User-agent: Webdup/0.9
User-agent: WebEnhancer
User-agent: WebGo
User-agent: WebLeacher
User-agent: WebmasterWorld
User-agent: WebmasterWorld Extractor
User-agent: WebmasterWorldForumBot
User-agent: WebPictures
User-agent: WebPix
User-agent: WebRobot
User-agent: WebSauger
User-agent: Website Quester
User-agent: Webster
User-agent: Webster Pro
User-agent: WebStripper
User-agent: WebStripper/2.10
User-agent: WebStripper/2.62
User-agent: WebVac
User-agent: WebZIP
User-agent: WebZip
User-agent: WebZip/4.0
User-agent: Wells
User-agent: Wget
User-agent: WGet/
User-agent: Wget/1.5.3
User-agent: Wget/1.6
User-agent: Wget/1.7
User-agent: Wget/1.8.1
User-agent: Wget/1.8.2
User-agent: Whiz
User-agent: Widow
User-agent: Win67
User-agent: Works
User-agent: WWW-Collector-E
User-agent: WWWC
User-agent: WWWOFFLE
User-agent: Xaldon
User-agent: Xaldon WebSpider
User-agent: Xaldon WebSpider 2.7.b6
User-agent: Xenu
User-agent: Xenu's
User-agent: Xenu's Link Sleuth 1.1c
User-agent: YahooSeeker
User-agent: Yamm
User-agent: Yotta
User-agent: zagre
User-agent: Zeal
User-agent: Zeus
User-agent: Zeus 32297 Webster Pro V2.9 Win32
User-agent: Zeus Link Scout
User-agent: ZunZun
Disallow: /
# Robots.txt ends here
Hope this helps some other folks out there. This is simply my way of thanking everyone for their contributions to the security of my servers over the past couple of months.
SlothV
I am a relatively new player to the Internet and was looking for some guidance on the use of robots.txt
Id like to use this on a couple of websites that I am building so if its OK with you I will.
Anyone intending to copy this for their own site would better double check each individual entry very carefully, not to mention the syntax of the resulting file.
Now... if I let every search engine, bot, mail harvester, etc etc rip around my site, then I would be out about 20 gigabytes of bandwidth per month.
So it is really just a question of economy and to preserve the level of service to my 'regular human' visitor traffic so that their experience on my site is not affected by my server being burdened or swamped down with unproductive traffic.
The bottom line is that all you really need in life is Google, Yahoo, Microsoft and SCooter from Altavista.
Now... in particular there are a number of fake bots out there that like to mask themselves via their USER AGENT to pretend they are legitimate bots.
Imfortunately there is no way for ME or my server to confirm with 100% accuracy that it is indeed orginating from the legitimate site. Whereas by restricting access down to just the four major contenders I can easily identify when a bogus IP is running around claiming to be from Google or Yahoo etc... then I just nuke that specific IP via the .htaccess file
The problem that exist nowadays is that the SCUMBAGS also use the AGENT names of legit robots to try to hide or obscure the fact that they are raping a web site.
So in my particular case it is better to err by blocking all - and thereby eliminate both the real bot and the fake bot... then to leave a doorway open for the fake bots to enter while pretending they are your bot.
Hope this explains things better.
Now it is agreed that the SCUMBAG bots are never going to obey a robots.txt file. That is why I also back up the robots.txt file with a powerful .htaccess file that fine tunes things and ensures that the bots identified on my robots.txt file do not pass go and collect $500.
I tried to post my .htacces file to a messge here... but at 48k in size it kept choking the message editor and freaking it out. So I will attempt to find a way to post the .htaccess file so that you can also use it to fine tune your systems.
So the bottom line is that the robots.txt file does work for the honest bots that are out there. But if a bot chooses to deliberately disobey it... then they are open season as far as I am concerned for inclusion in the .htaccess hammer of death..
I have attached in the next message the SetEnvIfNoCase User-Agent section of my .htaccess file so you can see how it supports the rules of the robots.txt file.
Options +FollowSymlinks
RewriteEngine On
RewriteBase /
RewriteCond %{THE_REQUEST} wget [NC,OR]
RewriteCond %{THE_REQUEST} passthru [NC,OR]
RewriteCond %{THE_REQUEST} formmail [NC,OR]
RewriteCond %{THE_REQUEST} admin_forums.php [NC,OR]
RewriteCond %{THE_REQUEST} perform=backup [NC,OR]
RewriteCond %{THE_REQUEST} highlight=\%2527 [NC,OR]
RewriteCond %{THE_REQUEST} highlight=%2527 [NC,OR]
RewriteCond %{THE_REQUEST} passthru [NC,OR]
RewriteCond %{THE_REQUEST} wget\%20 [NC,OR]
RewriteCond %{THE_REQUEST} echr [NC,OR]
RewriteCond %{THE_REQUEST} x04H [NC,OR]
RewriteCond %{THE_REQUEST} echo%20; [NC,OR]
RewriteCond %{THE_REQUEST} xc9 [NC,OR]
RewriteCond %{THE_REQUEST} configdir [NC,OR]
RewriteCond %{THE_REQUEST} vti_bin [NC,OR]
RewriteCond %{THE_REQUEST} vti_pvt [NC,OR]
RewriteCond %{THE_REQUEST} cltreq.asp [NC,OR]
RewriteCond %{THE_REQUEST} rush= [NC,OR]
RewriteCond %{THE_REQUEST} esystem [NC,OR]
RewriteCond %{HTTP_REFERER} ^(.*)aolimages [NC,OR]
RewriteCond %{HTTP_REFERER} ^(.*)lillas.splinder.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^(.*)scot.altermedia.info [NC,OR]
RewriteCond %{HTTP_REFERER} ^(.*)images.google [NC,OR]
RewriteCond %{HTTP_REFERER} ^(.*)www.enation.org [NC,OR]
RewriteCond %{HTTP_REFERER} ^(.*)sum0.net [NC,OR]
RewriteCond %{HTTP_REFERER} ^(.*)domaindlx.com [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Tcs/1" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "freefind" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "EmeraldShield" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "libwww-perl" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "lwp-trivial" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Turnitin" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "trivial" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "User-Agent: Mozilla" [NC,OR]
RewriteCond %{HTTP_COOKIE}% s:(.*):\%22test1\%22\%3b [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "LWP::Simple" [NC]
RewriteRule ^.*$ [127.0.0.1...] [L,R=301]
The next message will contain part II of the three parts
SetEnvIfNoCase User-Agent "^(.*)Jakarta" bad_bot
SetEnvIfNoCase User-Agent "^(.*)_iRc" bad_bot
SetEnvIfNoCase User-Agent "^(.*)000000000" bad_bot
SetEnvIfNoCase User-Agent "^(.*)4all" bad_bot
SetEnvIfNoCase User-Agent "^(.*)abot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Accept" bad_bot
SetEnvIfNoCase User-Agent "^(.*)accoona" bad_bot
SetEnvIfNoCase User-Agent "^(.*)AceFTP" bad_bot
SetEnvIfNoCase User-Agent "^(.*)adressendeutschland" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Ahead" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Alarm" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Alexibot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Anal" bad_bot
SetEnvIfNoCase User-Agent "^(.*)andit" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Amfi" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Anon-Encrypt" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Anon_Proxy" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Ansearch" bad_bot
SetEnvIfNoCase User-Agent "^(.*)AnswerBus" bad_bot
SetEnvIfNoCase User-Agent "^(.*)appie" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Arachmo" bad_bot
SetEnvIfNoCase User-Agent "^(.*)arianna" bad_bot
SetEnvIfNoCase User-Agent "^(.*)asterias" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Atlocal" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Atropos" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Auto" bad_bot
SetEnvIfNoCase User-Agent "^(.*)bach-idx" bad_bot
SetEnvIfNoCase User-Agent "^(.*)BackDoorBot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Baiduspide" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Bandit" bad_bot
SetEnvIfNoCase User-Agent "^(.*)BasicHTTP" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Bison" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Black" bad_bot
SetEnvIfNoCase User-Agent "^(.*)BlowFish" bad_bot
SetEnvIfNoCase User-Agent "^(.*)BoardPulse" bad_bot
SetEnvIfNoCase User-Agent "^(.*)BoardViewer" bad_bot
SetEnvIfNoCase User-Agent "^(.*)boitho" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Bond" bad_bot
SetEnvIfNoCase User-Agent "^(.*)boris" bad_bot
SetEnvIfNoCase User-Agent "^(.*)BotALot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)BotName" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Brother" bad_bot
SetEnvIfNoCase User-Agent "^(.*)BuiltBotTough" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Bullseye" bad_bot
SetEnvIfNoCase User-Agent "^(.*)BunnySlippers" bad_bot
SetEnvIfNoCase User-Agent "^(.*)CafeKelsa" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Cafi" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Capture" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Cegbfeieh" bad_bot
SetEnvIfNoCase User-Agent "^(.*)cfetch" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Chaos" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Check" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Cheese" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Cherry" bad_bot
SetEnvIfNoCase User-Agent "^(.*)ChinaClaw" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Clam" bad_bot
SetEnvIfNoCase User-Agent "^(.*)classified" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Clush" bad_bot
SetEnvIfNoCase User-Agent "^(.*)COAST" bad_bot
SetEnvIfNoCase User-Agent "^(.*)collage" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Collect" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Concept" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Convera" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Copier" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Copy" bad_bot
SetEnvIfNoCase User-Agent "^(.*)CoralWebPrx" bad_bot
SetEnvIfNoCase User-Agent "^(.*)cosmos" bad_bot
SetEnvIfNoCase User-Agent "^(.*)costa" bad_bot
SetEnvIfNoCase User-Agent "^(.*)craft" bad_bot
SetEnvIfNoCase User-Agent "^(.*)crawler" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Crescent" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Custo" bad_bot
SetEnvIfNoCase User-Agent "^(.*)CuteFTP" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Cyberian" bad_bot
SetEnvIfNoCase User-Agent "^(.*)DA 7" bad_bot
SetEnvIfNoCase User-Agent "^(.*)DARK" bad_bot
SetEnvIfNoCase User-Agent "^(.*)dcbot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Diamond" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Dillo" bad_bot
SetEnvIfNoCase User-Agent "^(.*)DISCo" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Dispatch" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Ditto" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Download" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Drecombot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)DTS Agent" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Dual" bad_bot
SetEnvIfNoCase User-Agent "^(.*)EARTHCOM" bad_bot
SetEnvIfNoCase User-Agent "^(.*)EasyDL" bad_bot
SetEnvIfNoCase User-Agent "^(.*)eCatch" bad_bot
SetEnvIfNoCase User-Agent "^(.*)EdcompBot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)ELinks" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Email" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Enfi" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Enhancer" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Epiphany" bad_bot
SetEnvIfNoCase User-Agent "^(.*)EroCrawler" bad_bot
SetEnvIfNoCase User-Agent "^(.*)ERROR" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Express" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Extract" bad_bot
SetEnvIfNoCase User-Agent "^(.*)EyeNetIE" bad_bot
SetEnvIfNoCase User-Agent "^(.*)FANG" bad_bot
SetEnvIfNoCase User-Agent "^(.*)FAST_Data" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Faxo" bad_bot
SetEnvIfNoCase User-Agent "^(.*)FDM" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Fetch" bad_bot
SetEnvIfNoCase User-Agent "^(.*)FindAnISP" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Finder" bad_bot
SetEnvIfNoCase User-Agent "^(.*)findlinks" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Flash" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Flip" bad_bot
SetEnvIfNoCase User-Agent "^(.*)focus" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Foobot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Foto" bad_bot
SetEnvIfNoCase User-Agent "^(.*)FrontPage" bad_bot
SetEnvIfNoCase User-Agent "^(.*)futile" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Fyber" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Gaisbot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Generic" bad_bot
SetEnvIfNoCase User-Agent "^(.*)geni" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Geona" bad_bot
SetEnvIfNoCase User-Agent "^(.*)GetHTML" bad_bot
SetEnvIfNoCase User-Agent "^(.*)GetRight" bad_bot
SetEnvIfNoCase User-Agent "^(.*)GetWeb" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Giga" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Girafabot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)glx v" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Go!Zilla" bad_bot
SetEnvIfNoCase User-Agent "^(.*)GoForIt" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Googlebot-Image" bad_bot
SetEnvIfNoCase User-Agent "^(.*)GoSearch" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Got-It" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Grab" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Grafula" bad_bot
SetEnvIfNoCase User-Agent "^(.*)grub" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Gruppo" bad_bot
SetEnvIfNoCase User-Agent "^(.*)GT::WWW" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Gypsy" bad_bot
SetEnvIfNoCase User-Agent "^(.*)hack" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Harvest" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Hatena_Antenna" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Helmdall" bad_bot
SetEnvIfNoCase User-Agent "^(.*)heritrix" bad_bot
SetEnvIfNoCase User-Agent "^(.*)HLoader" bad_bot
SetEnvIfNoCase User-Agent "^(.*)HMView" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Holmes" bad_bot
SetEnvIfNoCase User-Agent "^(.*)holy_teacher" bad_bot
SetEnvIfNoCase User-Agent "^(.*)HTML_Works" bad_bot
SetEnvIfNoCase User-Agent "^(.*)HTMLParser" bad_bot
SetEnvIfNoCase User-Agent "^(.*)httplib" bad_bot
SetEnvIfNoCase User-Agent "^(.*)HTTrack" bad_bot
SetEnvIfNoCase User-Agent "^(.*)human" bad_bot
SetEnvIfNoCase User-Agent "^(.*)hverify" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Hyper" bad_bot
SetEnvIfNoCase User-Agent "^(.*)ichiro" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Iltrovatore" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Image" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Indy" bad_bot
SetEnvIfNoCase User-Agent "^(.*)InetURL" bad_bot
SetEnvIfNoCase User-Agent "^(.*)InfoNav" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Innerprise" bad_bot
SetEnvIfNoCase User-Agent "^(.*)InterGET" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Internet\x" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Intraformant" bad_bot
SetEnvIfNoCase User-Agent "^(.*)IRLbot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)ISC_Systems" bad_bot
SetEnvIfNoCase User-Agent "^(.*)ISSpider" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Java 1" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Java/1" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Java1" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Jenny" bad_bot
SetEnvIfNoCase User-Agent "^(.*)JetCar" bad_bot
SetEnvIfNoCase User-Agent "^(.*)JoBo" bad_bot
SetEnvIfNoCase User-Agent "^(.*)JOC" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Jyxobot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)kashmir" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Kazoom" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Kenjin" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Keyword" bad_bot
SetEnvIfNoCase User-Agent "^(.*)KnowItAll" bad_bot
SetEnvIfNoCase User-Agent "^(.*)KRAEHE" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Kumm" bad_bot
SetEnvIfNoCase User-Agent "^(.*)larbin" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Leech" bad_bot
SetEnvIfNoCase User-Agent "^(.*)LexiBot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)LibWeb/clsHTTP" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Lincoln" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Linkman" bad_bot
SetEnvIfNoCase User-Agent "^(.*)LinkScan" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Lint" bad_bot
SetEnvIfNoCase User-Agent "^(.*)List" bad_bot
SetEnvIfNoCase User-Agent "^(.*)LMQueue" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Locust" bad_bot
SetEnvIfNoCase User-Agent "^(.*)loop" bad_bot
SetEnvIfNoCase User-Agent "^(.*)LTH_Browser" bad_bot
SetEnvIfNoCase User-Agent "^(.*)lwp-request" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Lwp-trivial" bad_bot
SetEnvIfNoCase User-Agent "^(.*)LWP::Simple" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Mac_Finder" bad_bot
SetEnvIfNoCase User-Agent "^(.*)magic" bad_bot
SetEnvIfNoCase User-Agent "^(.*)majestic" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Mammoth" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Mass" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Mata.Hari" bad_bot
SetEnvIfNoCase User-Agent "^(.*)McBot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Mechan" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Metaspinner" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Meter" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Microsoft.URL" bad_bot
SetEnvIfNoCase User-Agent "^(.*)MIDown" bad_bot
SetEnvIfNoCase User-Agent "^(.*)MIIxpc" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Mirago" bad_bot
SetEnvIfNoCase User-Agent "^(.*)MIRE" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Missigua" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Mister\ PiX" bad_bot
SetEnvIfNoCase User-Agent "^(.*)MJ12bot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)MnoGo" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Moget" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Mojeek" bad_bot
SetEnvIfNoCase User-Agent "^(.*)MojeekBot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Moozilla" bad_bot
SetEnvIfNoCase User-Agent "^(.*)mouse.com" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Mozdex" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Mozilla/3.Mozilla/2.01" bad_bot
Join the above list to the list that follows in my next message which continues this list
.
SetEnvIfNoCase User-Agent "^(.*)Mozilla/8.0" bad_bot
SetEnvIfNoCase User-Agent "^(.*)MS Search" bad_bot
SetEnvIfNoCase User-Agent "^(.*)MSIECrawler" bad_bot
SetEnvIfNoCase User-Agent "^(.*)MSProxy" bad_bot
SetEnvIfNoCase User-Agent "^(.*)muscope" bad_bot
SetEnvIfNoCase User-Agent "^(.*)MyApp" bad_bot
SetEnvIfNoCase User-Agent "^(.*)NASA_Search" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Naver" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Navroad" bad_bot
SetEnvIfNoCase User-Agent "^(.*)NearSite" bad_bot
SetEnvIfNoCase User-Agent "^(.*)NetAnts" bad_bot
SetEnvIfNoCase User-Agent "^(.*)NetCrawler" bad_bot
SetEnvIfNoCase User-Agent "^(.*)NetSpider" bad_bot
SetEnvIfNoCase User-Agent "^(.*)netXplorer" bad_bot
SetEnvIfNoCase User-Agent "^(.*)NetZIP" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Neutrino" bad_bot
SetEnvIfNoCase User-Agent "^(.*)NEWT" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Next" bad_bot
SetEnvIfNoCase User-Agent "^(.*)NICE" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Ninja" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Nintendo" bad_bot
SetEnvIfNoCase User-Agent "^(.*)nogoop" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Nokodo" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Nomad" bad_bot
SetEnvIfNoCase User-Agent "^(.*)NPbot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Nutch" bad_bot
SetEnvIfNoCase User-Agent "^(.*)NWSpider" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Objects" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Octopus" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Offline\ Explorer" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Offline\ Navigator" bad_bot
SetEnvIfNoCase User-Agent "^(.*)OmniExplorer" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Openfind" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Oracle" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Oregano" bad_bot
SetEnvIfNoCase User-Agent "^(.*)osis" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Patwebbot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Pavuk" bad_bot
SetEnvIfNoCase User-Agent "^(.*)PcBrowser" bad_bot
SetEnvIfNoCase User-Agent "^(.*)PEERbot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)PHP/4." bad_bot
SetEnvIfNoCase User-Agent "^(.*)PHP_version_tracker" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Pica" bad_bot
SetEnvIfNoCase User-Agent "^(.*)pics" bad_bot
SetEnvIfNoCase User-Agent "^(.*)pigs" bad_bot
SetEnvIfNoCase User-Agent "^(.*)PipeLine" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Pluck" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Pompos" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Preload" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Privoxy" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Program_Shareware" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Prowl" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Psbot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Python" bad_bot
SetEnvIfNoCase User-Agent "^(.*)QueryN" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Quest" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Qweer" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Reap" bad_bot
SetEnvIfNoCase User-Agent "^(.*)ReGet" bad_bot
SetEnvIfNoCase User-Agent "^(.*)RepoMonkey" bad_bot
SetEnvIfNoCase User-Agent "^(.*)request" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Research" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Ripper" bad_bot
SetEnvIfNoCase User-Agent "^(.*)RMA" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Rogue" bad_bot
SetEnvIfNoCase User-Agent "^(.*)RPT-HTTPClient" bad_bot
SetEnvIfNoCase User-Agent "^(.*)RSSOwl" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Sample" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Sauger" bad_bot
SetEnvIfNoCase User-Agent "^(.*)savvybot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)SCAgent" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Schizozilla" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Schlong" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Schmozilla" bad_bot
SetEnvIfNoCase User-Agent "^(.*)SCrawl" bad_bot
SetEnvIfNoCase User-Agent "^(.*)script_type" bad_bot
SetEnvIfNoCase User-Agent "^(.*)SeznamBot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Shaz" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Sherlock" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Shot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Simple" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Sleipnir" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Sly" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Smurf" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Snag" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Sohu" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Spade" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Span" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Speed" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Spin" bad_bot
SetEnvIfNoCase User-Agent "^(.*)sproose" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Spurl" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Spyder" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Squid" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Sqworm" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Stack" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Statbot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)State" bad_bot
SetEnvIfNoCase User-Agent "^(.*)STEALTH" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Stress" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Strip" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Style" bad_bot
SetEnvIfNoCase User-Agent "^(.*)subot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Suck" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Supervi" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Surf4Me" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Surfbot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)SUS" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Suzuran" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Sweep" bad_bot
SetEnvIfNoCase User-Agent "^(.*)sygol" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Szukacz" bad_bot
SetEnvIfNoCase User-Agent "^(.*)TAke" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Talkro" bad_bot
SetEnvIfNoCase User-Agent "^(.*)tamu" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Tarantula" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Teleport" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Telesoft" bad_bot
SetEnvIfNoCase User-Agent "^(.*)TerrawizBot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Test" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Titan" bad_bot
SetEnvIfNoCase User-Agent "^(.*)ToCrawl" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Todobr" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Track" bad_bot
SetEnvIfNoCase User-Agent "^(.*)True" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Tunnel" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Turing" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Turnitin" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Twat" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Tweak" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Tygo" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Ultraseek" bad_bot
SetEnvIfNoCase User-Agent "^(.*)unknown" bad_bot
SetEnvIfNoCase User-Agent "^(.*)URLy" bad_bot
SetEnvIfNoCase User-Agent "^(.*)User-Agent:" bad_bot
SetEnvIfNoCase User-Agent "^(.*)USyd" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Vagabondo" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Valet" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Valid" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Vamp" bad_bot
SetEnvIfNoCase User-Agent "^(.*)versus" bad_bot
SetEnvIfNoCase User-Agent "^(.*)VCI" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Visual" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Void" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Voila" bad_bot
SetEnvIfNoCase User-Agent "^(.*)W3C" bad_bot
SetEnvIfNoCase User-Agent "^(.*)w3m" bad_bot
SetEnvIfNoCase User-Agent "^(.*)walhello" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Walker" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Watch" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Weather" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Web.Image" bad_bot
SetEnvIfNoCase User-Agent "^(.*)WebCatS" bad_bot
SetEnvIfNoCase User-Agent "^(.*)webcrawl" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Webdup" bad_bot
SetEnvIfNoCase User-Agent "^(.*)WebGo" bad_bot
SetEnvIfNoCase User-Agent "^(.*)WebLeacher" bad_bot
SetEnvIfNoCase User-Agent "^(.*)WebmasterWorld" bad_bot
SetEnvIfNoCase User-Agent "^(.*)WebPictures" bad_bot
SetEnvIfNoCase User-Agent "^(.*)WebPix" bad_bot
SetEnvIfNoCase User-Agent "^(.*)WebRobot" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Webster" bad_bot
SetEnvIfNoCase User-Agent "^(.*)WebZIP" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Wells" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Wget" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Whiz" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Widow" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Win67" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Works" bad_bot
SetEnvIfNoCase User-Agent "^(.*)WWWC" bad_bot
SetEnvIfNoCase User-Agent "^(.*)WWWOFFLE" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Xaldon" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Xenu" bad_bot
SetEnvIfNoCase User-Agent "^(.*)YahooSeeker" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Yamm" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Yotta" bad_bot
SetEnvIfNoCase User-Agent "^(.*)zagre" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Zeal" bad_bot
SetEnvIfNoCase User-Agent "^(.*)Zeus" bad_bot
SetEnvIfNoCase User-Agent "^(.*)ZunZun" bad_bot
<Limit GET POST PUT HEAD>
order allow,deny
allow from all
deny from env=bad_bot
</Limit>
The next message will contain the specific IP's that are banned...
This following part of my .htaccess list nails a lot of the spamsters, mail harvesters, scumbags that like pretending they are real legitimate robots and blatant recurring hack attack originating IP's.
Yes... there are quite a few blocks out against Korea, India, Pakistan, Russia, Bulgaria, Moldavia, Spain and CHINA... but then again these IP's are prime offenders and if your web site does not require such foreign traffic than you will be probably saving yourself a lot of grief by blocking these ranges out. They only made the list after repeated patterns of ABUSE and had to earn their place on this list.
<Files 403.shtml>
order allow,deny
allow from all
deny from env=bad_bot
</Files>
<trim>
List continued in next message ....
.
[edited by: Brett_Tabke at 2:02 am (utc) on April 11, 2005]
[edit reason] sorry... privacy concerns. [/edit]
deny from 212.106.172.
deny from 212.11.184.
deny from 212.129.232.100
deny from 212.138.
deny from 212.158.243.
deny from 212.160.159.60
deny from 212.165.
deny from 212.19.35.
deny from 212.199.252.21
deny from 212.200.
deny from 212.205.245.
deny from 212.21.121.196
deny from 212.217.0.
deny from 212.217.1.
deny from 212.217.2.
deny from 212.219.57.
deny from 212.255.127.193
deny from 212.27.41.
deny from 212.29.32.
deny from 212.31.242.
deny from 212.33.86.47
deny from 212.39.112.
deny from 212.39.113.
deny from 212.42.
deny from 212.47.27.186
deny from 212.50.160.
deny from 212.56.39.
deny from 212.80.76
deny from 212.86.208.
deny from 213.10.
deny from 213.112.195.17
deny from 213.115.111.
deny from 213.115.201.207
deny from 213.128.225.93
deny from 213.130.53.94
deny from 213.131.168.204
deny from 213.131.168.205
deny from 213.131.168.206
deny from 213.131.168.207
deny from 213.131.65.
deny from 213.131.74.
deny from 213.134.168.35
deny from 213.139.192.139
deny from 213.146.148.179
deny from 213.155.143.
deny from 213.156.63.50
deny from 213.16.96.
deny from 213.178.
deny from 213.186.37.104.
deny from 213.193.131.122
deny from 213.194.149.
deny from 213.200.242.18
deny from 213.203.138.51
deny from 213.210.152.
deny from 213.219.84.
deny from 213.219.85.
deny from 213.23.
deny from 213.24.168.
deny from 213.24.169.
deny from 213.249.155.239
deny from 213.25.96.194
deny from 213.250.163.50
deny from 213.27.139.109
deny from 213.42.2.
deny from 213.46.99.
deny from 213.69.149.153
deny from 213.75.12.
deny from 213.77.160.158
deny from 213.8.52.84
deny from 213.80.133.
deny from 213.81.198.
deny from 213.81.198.239
deny from 213.9.220.254
deny from 213.93.204.
deny from 216.11.71.2
deny from 216.19.32.
deny from 216.119.173.251
deny from 216.124.224.123
deny from 216.138.248.86
deny from 216.139.176.
deny from 216.142.221.99
deny from 216.143.84.
deny from 216.148.62.202
deny from 216.157.225.36
deny from 216.160.186.
deny from 216.160.35.130
deny from 216.185.83.
deny from 216.194.66.
deny from 216.224.44.42
deny from 216.226.43.91
deny from 216.23.208.220
deny from 216.244.131.
deny from 216.249.9.118
deny from 216.251.35.
deny from 216.26.148.
deny from 216.26.248.18
deny from 216.31.40.
deny from 216.32.76.
deny from 216.34.209.
deny from 216.43.79.67
deny from 216.60.21.4
deny from 216.63.70.170
deny from 216.66.110.34
deny from 216.68.204.
deny from 216.70.34.17
deny from 216.72.28.100
deny from 216.73.106.
deny from 216.83.96.37
deny from 216.94.44.109
deny from 217-37-230-253.in-addr.btopenworld.com
deny from 217.113.112.17
deny from 217.118.
deny from 217.119.82.
deny from 217.144.
deny from 217.148.186.
deny from 217.149.51.130
deny from 217.153.183.230
deny from 217.157.162.
deny from 217.174.103.
deny from 217.17.18.17
deny from 217.217.
deny from 217.218.
deny from 217.219.
deny from 217.222.94.
deny from 217.227.106.24
deny from 217.248.87.
deny from 217.37.102.
deny from 217.37.119.
deny from 217.43.173.
deny from 217.6.171.36
deny from 217.68.109.5
deny from 217.73.171.
deny from 217.73.172.
deny from 217.73.173.
deny from 217.81.20.
deny from 218.12.
deny from 218.13.
deny from 218.14.
deny from 218.15.
deny from 218.16.
deny from 218.162.
deny from 218.17.
deny from 218.18.
deny from 218.188.
deny from 218.2.
deny from 218.200.
deny from 218.232.
deny from 218.248.
deny from 218.3.
deny from 218.4.
deny from 218.5.
deny from 218.50.
deny from 218.51.
deny from 218.52.
deny from 218.53.
deny from 218.54.
deny from 218.55.
deny from 218.56.
deny from 218.57.
deny from 218.58.
deny from 218.6.
deny from 218.78.
deny from 218.79.
deny from 218.80.
deny from 218.81.
deny from 218.82.
deny from 218.83.
deny from 218.88.
deny from 218.89.
deny from 218.90.
deny from 218.91.
deny from 218.92.
deny from 218.93.
deny from 218.94.
deny from 219.101.
deny from 219.101.
deny from 219.128.
deny from 219.129.
deny from 219.130.
deny from 219.131.
deny from 219.132.
deny from 219.133.
deny from 219.134.
deny from 219.135.
deny from 219.136.
deny from 219.137.
deny from 219.141.
deny from 219.142.
deny from 219.143.
deny from 219.144.
deny from 219.148.
deny from 219.149.
deny from 219.163.
deny from 219.213.
deny from 219.64.
deny from 219.76.
deny from 219.93.
deny from 219.94.
deny from 219.95.
deny from 220.95.
deny from 220.107.
deny from 220.141.
deny from 220.144.
deny from 220.169.
deny from 220.192.
deny from 220.193.
deny from 220.194.
deny from 220.195.
deny from 220.196.
deny from 220.197.
deny from 220.198.
deny from 220.199.
deny from 220.200.
deny from 220.201.
deny from 220.202.
deny from 220.203.
deny from 220.204.
deny from 220.205.
deny from 220.206.
deny from 220.207.
deny from 220.224.
deny from 220.225.
deny from 220.226.
deny from 220.227.
deny from 220.228.
deny from 220.229.
deny from 220.234.
deny from 220.244.
deny from 220.247.244.
deny from 220.65.103.124
deny from 220.76.248.
deny from 221.13.
deny from 221.137.
deny from 221.14.
deny from 221.148.
deny from 221.15.
deny from 221.169.
deny from 221.186.130.115
deny from 221.186.23.
deny from 221.192.
deny from 221.193.
deny from 221.194.
deny from 221.195.
deny from 221.224.33.227
deny from 221.24.17.
deny from 222.115.
deny from 222.168.
deny from 222.150.
deny from 222.33.
deny from 222.88.
deny from 222.88.173.
deny from 222.89.
This list continues... for one more group of IP's in the banned list in the next message...
.
# Robots.txt begins here
# Bandwidth thieves to keep out
User-agent: MJ12bot
Its up to you to decide who gets access to your site, but its a bit harsh to label everyone as thieves :(
You serve so many pages, does your server support gzip'ing of HTML on the fly for browsers and bots that support it? If its not the case then this costs you a great deal of wasted bandwidth.
<snip>
That's the end of it. So if you add all the parts of the .htaccess file together you should end up with about 48k of .htaccess. And surprisingly it really doesn't slow down the server one blip for it to screen through all these offenders and block them from even accessing your site.
The other way of looking at it is this... each of these offenders has the ability to clog, burden, slowdown or inconvenience ALL of your regular visitors to the point where their experience on your web site is seriously impaired.
So by nailing these turkeys before they can even access your cgi scripts, forums, search engines, etc, you are saving yourself wasted bandwidth ( because these idiots are blooked right at the door before they can even get in ) and it saves your server valuable CPU power by not wasting time serving their frivilous request.
I am sharing this .htaccess file with everyone in the hopes that the kind of traffic originating from these leeches of the internet is not the kind of traffic that benefits any of our sites or our regular visitors.
If you are not able to get this to all work together... pass the info along to your web hosting company and ask them to go over it to make any modifications that might be required in order to make it function on your server.
These are in use on a Linux server running a fairly vanilla version of Apache. So I believe it should work properly on most boxes out there. And if it doesn't work the first time... your web hosting company should be able to make the minor changes to make it more in harmony with their server configurations.
.
[edited by: Brett_Tabke at 2:03 am (utc) on April 11, 2005]
[edit reason] sorry - privacy concerns. [/edit]
I believe I fully explained why your personal BOT had made the list. It is not so much that I have anything against your bot pillaging my site... but the fact that SCUMBAGS also mascarade around pretending to be Majestic, Yahoo, MSN, Google and just about any other USER AGENT name that they think will help them slip their mail harvest or web copier pass an .htaccess blockade.
So yes... Majestic may not be a THIEF... or it may... it just depends on WHO has put that tag line as the USER AGENT. And since ANYONE can fake that they are using a Majestic User Agent, then I err on the side of blocking both.
Bottom line is that the traffic coming to my site comes primarily from the big three search engines... and if I let every single home brewed bot and search engine 'in development' on to my site... well there would be no bandwidth available to service all the LEGITIMATE visitors to my web site.
Saddest part is that many webmasters have no idea how many of these leeches and parasite bots are constantly crawling their sites. In cases where the webmaster is having to pay for bandwidth usage overages... these parasites can actually put quite a dent in the financial pocket book of many web sites.
So at some point somone has to simply make a decision... are they running a web site to cater to the leeches and parasites that are costing them bandwidth money.... or does their web site exist to service the legitimate users. I side on that of the legimate users and they are my first priority.
As for using gzip... yes... portions of my site do make use of on the fly compression to reduce bandwidth consumption. I also make extensive use of html compression and html coding tricks to minimize the physical size of the pages being served to my legitimate users. As for my graphics... when used, they are run through a jpeg compression program to reduce them down to the absolute minimal size possible ( without impairing their visual look ). So yes... I think I have that aspect of webmastering well in hand.
But I can assure you that the bandwidth leeches of the Internet can do more damage at running up your bandwidth usage then the sloppiest of web page code.
And if you just happened to have a malicious user who took a disliking to you... well he just has to turn on his web site copier and keep leeching copies of your entire web site for days and days and day... and by the end of the month he can easily consume 100 gigabytes of your bandwidth.
If you happen to be paying for overages of your contracted bandwidth quota.. this leech can impact your pocket book when the hosting company sends you the bandwidth bill. And if you happen to be on a 'free hosting service'... well they simply might kick your site off their servers rather than continue to permit your site to consume such a great amount of bandwidth.
.
I have already identified that the majority of these IP's trace back to Korea, China, Phillipines, Russia, Bulgaria, Spain, several South American countries and various other nest of improper traffic.
Yes... if your site handles legitimate traffic from any of those countries it might have 'some' impact. But the sites it blocks will be originating from identified hotbed IP ranges of scumbags, mail harvesters, etc. etc.
But possibly the ony way to satisfy yourself is to plug them into your .htaccess file and then simply monitor your ERROR LOG and see what traffic ( if any ) is being bounced or denied access to your site.
You will be surprised how few bounces you will have in your logs. That is because there is NOT much legitimate traffic originating from those IP's.
So your own ERROR LOG will be all the proof you need as to the value or effectiveness of their inclusion in your .htaccess file.
As for me... my site is based on US web servers... my traffic is GLOBAL but with a significant majority coming from the US, Canada, South America, Europe, England, Australia, etc. etc. etc. I think my site demographics pretty well match up against the kind of traffic that most other web sites enjoy.
So as I said... even if your site was operating and serving traffic from Korea, China, Phillipines, Russia, Bulgaria, Spain, several South American countries, I don't think it will bounce any of the LEGITIMATE traffic originating from these countries... as these IP's are specific to scumbag nest of abusers...
Try it... if you don't like it... you simply take your text editor and remove it... BUT if it works... well the benefits will prove themselves over the long run...
Ciao
.
are they running a web site to cater to the leeches and parasites that are costing them bandwidth money.... or does their web site exist to service the legitimate users.
The site is yours and its up to you to decide who gains access, but all I am saying is that I find your choice of words towards legitimate bots in your list a bit too harsh. Your list primarily bans bots that care to obey robots.txt -- just since when the bad guys bothered to tell you who they are and follow good practice of checking robots.txt?
If every website in the world followed your lead then the Internet would have become a boring place.... and value of robots.txt would have dropped to zero.
I hope you don't consider people who came to your website but never clicked on any ads that you (I presume here) have parasites and leeches.
As I read it, first SV gave a robots.txt file, then an htaccess file for those who ignore robots.txt.
I'm new to this, so my question is: Is it possible to ignore an .htaccess file, or is your server in total control of that?
Sidenote: as a writer, and someone who appreciates dramatic language usage, I object to your objection of SlothVader's word choice ;-)
So it only serves to be obeyed by the legit companies that have morals and corporate ethics... something that spam harvesters and site rippers are totally lacking.
You take offense because you have a bot. Your bot may be the most honest bot in the world. Isn't that simply enough reason for a 'bad guy' to set his 'user agent' to represent itself on the Internet as your bot to trick sites into letting it in?
Sorry... but the harsh reality of the Internet world is that your 'experimental' search engine, using bots is a parasite. You do not to put coin in my pocket and you hope to reap a profit by mining the data on my site. In a way this is no different then the spam harvesters that rape through a site, downloading every page, simply to look for e-mail addresses embedded in the page.
But look on this on a MUCH LARGER scale than the little pond that you presently envisage. There are tens of thousands of other 'developers' out there that are doing exactly the same thing you are. And who gets raped when each of those 10,000 bots descend on to a web site? Whose bandwidth is getting chewed up and server bogged down because of these commercial 'for profit' downloading of pages? It certainly isn't the webmaster who is profiting from your visit and that of the other 10,000 similar hopeful bots that are experimenting by grabbing pages from sites in their data mining spree.
But I digress. Personally I would love to see all mail harvesters and site rippers made illegal. They are the true parasites on the Internet. And as long as they exist, they will run around mascarading as the legitimate bots. That is simply a fact and I am sure at some point YOU will ( or have been accused ) of web copying someone's web site ( or putting it under a denial of service attack ) simply because its user agent represented itself as one of your BOTS. That is simply the risk you take by putting bots on to the Internet with your branding on it. And if it hasn't happen... well believe me it will as nothing is sacred to the datamining thieves of the Internet.
Your honesty, your integrity will ultimately be damaged by the 'bad bots' - who like wolves like to dress up like sheep in order to better gain access to the sheep herd.
Now as to the question from the other user regarding the power of the .htaccess file. YES it is that powerful and there is nothing a spam harvester or site ripper can do except make sure that they don't have something in their IP, user agent, URL request etc that can be signatured as being particular to them. Once you can id something unique to the objectionable 'people' of this world, you are able to apply a rule to the .htaccess that will permaently block, bar or redirect them elsewhere. You can serve them up 403 error pages or you can 301 them off to somewhere. In my particular case I like to redirect them to THEIR OWN SERVER... ( although most attacking bots don't usually take the time to listen to what your server is telling them to do... ie go off to 127.0.0.1 ( the attackers computer/server) etc.
Now... as to why have a robots.txt file if the .htaccess file is so powerful... the answer is simple... not all webmasters have access to the .htaccess file and therefor ONLY HAVE the ability to throw a robots.txt file against the roving bands of pillaging bots. So it may not be great... but it is one of the few steps a web master with limited power on the server can take against the thieves.
Yes... I have a pretty harsh opinion of all the bandwidth leeches on the Internet. But then again I guess I have continued to watch the continued growth in the number of thieves roaming the Internet and unfortunately a reluctance or questionable cooperation of ISP's or hosting companies to turn a blind eye to the activities of some of their clients.
Why do people not like 'other web sites' using direct links to graphics on their sites? Because the other lazy fellow is stealing YOUR bandwidth to provide graphics on his site, with no additional cost in bandwidth usage to him or her self. That is a thief...
And there are so many variations of activity on the Internet that fit this same scenario.... leeches trying to find a way by pillaging someone elses data for THEIR OWN PROFIT, that it becomes ridiculous when their numbers and bandwidth consumption can accumulate to the point where it is in greater amounts than the regular visitors.
Case in point... A spam harvester may chew down 500 megs of my datafiles... whereas a regular visitor may only download and cache 50k of data. Put your math cap on and figure out how many 50k users that one 500 meg thief equals.
And unfortunately, data thieves tend to operate on 'big pipes' which give them the maximum ability to suck down data off your site as fast as the server and bandwidth can provide it. As most of them can overwhelm a web site with their gulping down of a web site, this impacts the 'regular' visitors who have to try to squeeze in, on to your site, grabbing a place in the stream of traffic being requested by the 'bad bots'.
No... Majestic may take offense at his bot being lumped into the 'bad category'... but ultimately in the end his project is designed to line his own pocket at the data mining expense of others. And so no matter how honorable his intentions he probably takes the greatest offense at seeing that his life's efforts are seen by some folks as little or no different than a spam harvester.
And believe it or not... I think the majority of webmasters who have control of their servers and have witnessed the nonsense being commited against their servers are as fed up as I am....
-:
Isn't that simply enough reason for a 'bad guy' to set his 'user agent' to represent itself on the Internet as your bot to trick sites into letting it in?
No - because bad guys:
a) won't bother supporting robots.txt in the first place
b) fake user agent to be IE6 or even google
c) use throw away botnet controlled DSL/cable boxes so IPs will change
Banning all bots apart from top 3 is a kneejerk reaction to situation and while it may provide some false sense of satisfaction it won't solve your problem.
No... Majestic may take offense at his bot being lumped into the 'bad category'... but ultimately in the end his project is designed to line his own pocket at the data mining expense of others.
Tell that to Google, MSN and Yahoo -- they are lining up their pockets while we argue here from data mining of your site, and since you let them crawl whole of your site they get the most money out of you. Next time they change their algos it may affect your site, so take your time to think how wiser it would have been to have more search engines without handful of them dominating market :)
If every public web site owner was doing what you propose to do then there would have been no Web as we know it. I type this and imagine how Google founders must have been suprised to see that attitude at the time when they started :)
Finally, just for the record -- my bot is not parasite, and its not a leech, it was designed to obey robots.txt, including Crawl-Delay parameter that allows to spread number of requests over time, search engine design also attempts to minimise number of pages sucked from any site. So, as far as I am concerned inclusion of my bot in your list is totally unjustifiable: I'd hazard to guess that it would have "sucked" 150-200 pages from your site, what is it comparing to your 1 mln page views a day + images that will be retrieved by people who browse site.
You are not alone... there are thousand of 'legit' bots out there doing the same grabbing of pages from sites. You may see 100 to 200 pages ( although my site has closer a far greater number of pages than that )... but multiply that by the 10,000 other bots that are surfing around and hopefully you can see the culmulative effect of a web site simply having an open door and letting every bot that knocks in.
So at this point it is like having a 'popular' nightclub - where you have line ups of folks trying to get in. Sure... its free and the crowds are overwhelming your facilities... so you start to put in some controls to try to restrict the number of 'guest' entering your nightclub.
It is not that you are being a bad guy as a night club owner... it is simply that you want the patrons in your night club to have a little elbow room, where they can move about freely without being squashed in in a building that is operating beyond its capacity.
So the nightclub owner may start choking back the number of patrons admitted to the club and he puts two big bouncers on the door to make sure that no one trys to ignore the clubs capacity limit... I guess that would be the .htaccess file of the night club.
Unfortunately there are always going to be weasles who are going to try to trick the bouncers into letting them in by lying about who they are. This is where the legitimate user's identity ( or a legit bot ) is taken over by the club crashers ( or bad bots ).
Bottom line is that controls on entry are simply an accepted way of life, whether in night clubs... movie theatres or on web sites.
And although it may put a minor damper on the lesser known or unknown developing search bots... it is a small price to pay in the overall scheme of things to provide a better experience for the patrons who are frequenting your nightclub or web site. AFter all... if the patrons who do get in your nightclub don't have a pleasant experience ( prompt and responsive server and page displays ) they aren't going to be coming back and will move on and begin frequenting another nightclub ( web site ) that takes better care of its visitors.
I see the quandary that you face... trying to become one of the patrons the nightclub WANTS to let in to its premises. And it will prove more and more frustrating as other webmasters resort to tactics to maintain a good level of service to their present clients.
But just as life is for nightclubs, the same will apply to search engines... and if you provide a quality product, with consistent and repeatable levels of good service, then ultimately folks will be coming to your search site voluntarily and offering to you their URL to crawl. After all, that is the ultimate aim in the end... to have folks want to submit to your crawl - as that will be the signal that you have gone from the unknown quantity to a known and sought after quantity.
The boys from Google and Yahoo also faced similar qualms when their activities were first debated. In the end they have nightclubs that everyone wants to be in... and yes... they are reaping the rewards for their efforts.
But we have to be realistic in that for every Google and Yahoo there are tens of thousands of contenders who would love to reach that pinnacle in the search engine world. But reality tells us that like a pyramid, the majority are destined to be at the botton and only a few make it to the peak.
Can any web site sustain supporting an open door policy to the tens of thousands of beta and developmental search engines being developed around the world? I think not... So like any good user I have to place my bets with the 'going concerns' as they already hold marketshare and it is not likely that they will be usurped from their thrones any time in the near future. They have the momentum to keep growing and we as webmasters have to go with the flow just because the ROI ( return on investment - I let them in to rape me but I gain the greatest number of potential of new visitors )makes it worthwhile in the short and long term.
SlothV
.
but multiply that by the 10,000 other bots that are surfing around and hopefully you can see the culmulative effect of a web site simply having an open door and letting every bot that knocks in.
Your list has far far less than claimed 10,000 other bots, and most of the things in your list are marginal dev search engines that can't possibly hit your site big time.
I see the quandary that you face... trying to become one of the patrons the nightclub WANTS to let in to its premises.
No quandary on my face -- my bot politely asks at the doors and goes away when told off, fair play, but very short sighted -- if everyone decided to follow your policy then it would kill the Net as it is now.
Can any web site sustain supporting an open door policy to the tens of thousands of beta and developmental search engines being developed around the world? I think not...
What thousands search engines? Just how many outdated pages those have? A handful, they can't possibly account for significant chunk of already highly trafficed website.
I challenge you to provide stats showing in percentage based traffic usage by:
1) your "patrons" (include images in that)
2) Google/Yahoo/MSN
3) all other bots
If #3 accounts for 3-5% of your overall traffic then in my view your position is totally unjustifiable from strategic point of view, which is to make sure that incoming traffic is diversified and you are not depending on big boys whose games with algos could cause major fluctuations in traffic.
Now as to a challenge against my stats? Why? I have nothing to prove to you or anyone. The robots.txt and .htaccess file speak far louder than some kind of graphical stat of the viewership of my sites. Suffice to say that if I built this list and have spent time tweaking it to this degree, then you can pretty safely assume that I have volumnous logs to wade through in order to id the attackers and the bad bots.
Now as to questions about traffic... My logs and stats package are pretty precise at giving me my traffic referal stats... and the bottom line is that the pyramid looks something like this so far this month ( April 1 to 10th ) :
For every
100,000 referals from Google, there is
20,000 referals from Inktomi, there is
10,000 referals from MSN, there is
10,000 referals from AskJeeves, there is
1,000 referals from unknown search sites, there is
100 referals from Alexa, there is
10 referals from AltaVista, there is
1 referal from Echo!
That is the pyramid that we see consistently EACH AND EVERY MONTH. Google is seriously kicking all the other search engines combined. Inktomi has made great strides and risen to the #2 spot over the last six month period.
MSN and AskJeeves have also been slowly making headway but they have the most OBNOXIOUS search bots that I have ever seen. They basically overstay their welcome and keep coming back day after day after day to the point where they are more of a pest then a benefit. Hopefully they will get their bots better trained to not make as many frequent appearances.
Even as we speak, there are MSN and AskJeeves bots roaming all over my server. In the case of AskJeeves I noticed an increase in their activity with their recent sale. It almost seems like their new owner is trying to expand the number of sites or pages contained in their database.
Alexa is frankly a pain in the ass. But for some reason there is a large number of users who make use of the Alexa bar and it has given us a fairly high ranking with Alexa. Sadly there are some advertisers who place some merit in those stats and so we grudgingly haven't kicked the Alexa bot off our server.
The AltaVista stat is probably the saddest and reflects how they went from a position of supreme power as 'THE' search engine and have now been relegated to the scrap heap. But their bot is not too intrusive or a frequent visitor to our site, so we just leave it alone in honor and recognition for the great start it gave us when we first put our sites online back in 1997.
With regards to the majority of those 'unknown' search directories... well I believe ( from years of watching our servers and logs ) that at least 80% of them result from DMOZ clones around the Internet. Yeah... it seems that just about any body and their dog can put a DMOZ clone online nowadays. But even with the great number of them online, it really doesn't add up to a hill of beans in the way of traffic. But even on the DMOZ database we have about 20 entries in their directory - and they ALL contain the keyword placements that we wanted.
Now... as to the base of my viewership... well I don't survive solely on the introduction of new visitors via the search engines. My sites have been online since 1997... and so I have built up a steady and loyal cliental over the years by word of mouth and recommendations between friends. Hence the reason I have always given first priority to providing a consistent and dependable service to our visitors. It's sort of like a McDonalds in that no matter where you go the product is always going to be the same... people like that consistency factor.
But the bottom line to quantify our viewership numbers is to simply think logically. There is no way in the world that I would have spent so much time tweaking the robots.txt and .htaccess files if it was not highly beneficial to the operation of our web sites.
The comprehensiveness of these two list is not something a person with a mere 100k of visitors per month or 2 gigabyte of traffic would have involved themselves in producing. And so hopefully that alone should put most quizical minds to rest because the logic of it speaks louder than any stat that I could place here for you to review.
I am extremely cognizant of where my 'good' traffic originates from and where the 'bad' traffic comes from. Ultimately ( after doing this since 1997 and having survived the dot.com bust and armies of competitor sites that have come and gone ) we have developed a bit of an expertise in this area that we wanted to share by putting our robots.txt and .htaccess into the public domain so others could benefit from what we have learned.
Our longevity and success is also probably one of the reasons why so many thieves with web copiers have attempted to steal and plunder the web sites. They operate under the delusion that if they steal it, alter it and put it online, it will bring them the level of success that we enjoy. So you can see by my robots.txt and .htacces there is a strong leaning towards the quashing of these thieves and the warez they use to rip off sites.
So in a way the developmental search engines only make up a small percentage of the IP's, or keywords used to kill off the plunderers and non-valuable traffic roaming through our site.
I am sorry if your bot made it on my list. But if it made it on this list than it is obvious that it or a fraudster using its identity came to my site ( possibly multiple times), saw originally that it wasn't blocked and did something to bring itself to our attention ( on one or more occassions ).
Prior to adding any bot to our list, we do a little research to figure out what the bot is... and to hunt down its originating IP to see if that gives us a clue about its purpose, its intent, who is using it and from what country it originates from.
In cases where we encounter conflicting information about the given signature of a bot, ( like where sometimes it is a good bot and sometimes it has been reported as a potentially bad bot ) we err on the side of caution and respect towards our viewership and simply ban it.
So yes... innocent bots have faced the door when visiting our site... but just as importantly a far greater number of bad bots have likewise seen the door.
This is the luxury that we enjoy compared to many other webmaster, that being that we have full access to minute by minute reports of activities on our servers. We can spot the problems in midstream - and not after they have caused a problem or pillaged our site. As such we can kill them off in mid-gulp as they attempt to download all the content off our sites.
The answer for you is to simply change the user agent identification of your bot so that it does not get caught up in the robots.txt or .htaccess file.
But as I have previously mentioned, there is nothing stopping a bad bot owner from simply taking your new user agent identification and bringing your legitimate bot back into disrepute or onto our list.
Anyhow, I love babbling with you about all this stuff. And in a way the only other folks who will enjoy our conversation are webmasters that are interested in minimizing the amount of non-valuable traffic in order to maximize the available bandwidth of their sites for the 'valuable' traffic - ie the end users.
Because ultimately nothing matters if you don't have a loyal and continually growing core of repeat visitors. And it is our viewership that we give our first priority to...
.
we have developed a bit of an expertise in this area that we wanted to share by putting our robots.txt and .htaccess into the public domain so others could benefit from what we have learned.
You are doing exactly the opposite - you are not helping public domain by having lots of well behaved bots banned for only reason that they currently do not belog to top 3-4 engines that refer to you most of traffic.
If you only limited yourself to bots that do not behave well, but for some reason support robots.txt, then its fair play, however your case is the case of banning everyone apart from Google/MSN/Yahoo. If you are so hell bent on doing that then why not just cloak your robots.txt which would return:
User-agent: *
Disallow: /
to any user-agent that is not Google/Yahoo/MSN.
The answer for you is to simply change the user agent identification of your bot so that it does not get caught up in the robots.txt or .htaccess file.
Like this is the way to do things for well behaved bots that obey robots.txt! The answer for me is to crawl other sites that are bound to have content that is no worse, or better than yours: 10 bln+ pages out there, and chances are your content was copied by some plagiariser who does not ban all bots and as the result will get the traffic :oP
[edited by: Lord_Majestic at 1:41 am (utc) on April 11, 2005]
anyway - this is the robots.txt forum - there is an htaccess topic as part of the apache forum.
There is also this:
[webmasterworld.com...]
