I have trouble in my website, some unknown robots are eating my bandwidth like anything (still 43,000 mb eaten in a week). And i am not familiar in .htaccess and robots.txt creating.
Reports given below: Unknown robot (identified by empty user agent string)
Unknown robot (identified by 'spider')
Unknown robot (identified by 'robot')
Unknown robot (identified by 'bot*')
Unknown robot (identified by 'crawl')
Unknown robot (identified by hit on 'robots.txt') So i added this code in my .htaccess file, will this avoid these bots and spiders eating my bandwidth.
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^Alexibot [OR]
RewriteCond %{HTTP_USER_AGENT} ^Yeti/0.01 [OR]
RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Xenu's [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus
RewriteCond %{HTTP_USER_AGENT} ^BadBot
RewriteCond %{HTTP_USER_AGENT} ^BadBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^EvilScraper [OR]
RewriteCond %{HTTP_USER_AGENT} ^FakeUser
RewriteCond %{HTTP_USER_AGENT} ^LinksManager.com_bot
RewriteCond %{HTTP_USER_AGENT} ""
RewriteCond %{HTTP_USER_AGENT} ^Java
RewriteCond %{HTTP_USER_AGENT} ^Jakarta
RewriteCond %{HTTP_USER_AGENT} User-Agent
RewriteCond %{HTTP_USER_AGENT} compatible
RewriteCond %{HTTP_USER_AGENT} "Mozilla"
RewriteCond %{HTTP_USER_AGENT} panscient.com
RewriteCond %{HTTP_USER_AGENT} libwww
RewriteCond %{HTTP_USER_AGENT} lwp-trivial
RewriteCond %{HTTP_USER_AGENT} curl
RewriteCond %{HTTP_USER_AGENT} PHP/
RewriteCond %{HTTP_USER_AGENT} urllib
RewriteCond %{HTTP_USER_AGENT} GT::WWW
RewriteCond %{HTTP_USER_AGENT} Snoopy
RewriteCond %{HTTP_USER_AGENT} MFC_Tear_Sample
RewriteCond %{HTTP_USER_AGENT} HTTP::Lite
RewriteCond %{HTTP_USER_AGENT} PHPCrawl
RewriteCond %{HTTP_USER_AGENT} URI::Fetch
RewriteCond %{HTTP_USER_AGENT} Zend_Http_Client
RewriteCond %{HTTP_USER_AGENT} PECL::HTTP
RewriteRule ^.* - [F,L]
RewriteCond %{HTTP_REFERER} ^http://.*somebadforum\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*example\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*lastexample\.com [NC]
RewriteCond %{HTTP_REFERER} ^http://.*LinksManager.com\.com [NC]
RewriteRule ^.* - [F,L]
RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*NEWT [OR]
RewriteCond %{HTTP_USER_AGENT} ^Crescent [OR]
RewriteCond %{HTTP_USER_AGENT} ^CherryPicker [OR]
RewriteCond %{HTTP_USER_AGENT} ^[Ww]eb[Bb]andit [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebEMailExtrac.* [OR]
RewriteCond %{HTTP_USER_AGENT} ^NICErsPRO [OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus.*Webster [OR]
RewriteCond %{HTTP_USER_AGENT} ^Microsoft.URL [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^LinkWalker [OR]
RewriteCond %{HTTP_USER_AGENT} ^sitecheck.internetseer.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^ia_archiver [OR]
RewriteCond %{HTTP_USER_AGENT} ^DIIbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^psbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailCollector
RewriteCond %{HTTP_USER_AGENT} ^InternetSeer.com [OR]
RewriteRule ^.* - [F]
RewriteCond %{HTTP_REFERER} ^http://www.iaea.org$
RewriteRule !^http://[^/.]\.your-site.com.* - [F]
RewriteCond %{HTTP_USER_AGENT} ^attach [OR]
RewriteCond %{HTTP_USER_AGENT} ^BackWeb [OR]
RewriteCond %{HTTP_USER_AGENT} ^Bandit [OR]
RewriteCond %{HTTP_USER_AGENT} ^BatchFTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot@yahoo.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^Buddy [OR]
RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]
RewriteCond %{HTTP_USER_AGENT} ^Collector [OR]
RewriteCond %{HTTP_USER_AGENT} ^Copier [OR]
RewriteCond %{HTTP_USER_AGENT} ^DA [OR]
RewriteCond %{HTTP_USER_AGENT} ^DISCo\ Pump [OR]
RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR]
RewriteCond %{HTTP_USER_AGENT} ^Download\ Wonder [OR]
RewriteCond %{HTTP_USER_AGENT} ^Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^Drip [OR]
RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]
RewriteCond %{HTTP_USER_AGENT} ^FileHound [OR]
RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetSmart [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]
RewriteCond %{HTTP_USER_AGENT} ^gotit [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]
RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]
RewriteCond %{HTTP_USER_AGENT} ^HTTrack [OR]
RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]
RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR]
RewriteCond %{HTTP_USER_AGENT} ^Iria [OR]
RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]
RewriteCond %{HTTP_USER_AGENT} ^JOC [OR]
RewriteCond %{HTTP_USER_AGENT} ^JustView [OR]
RewriteCond %{HTTP_USER_AGENT} ^larbin [OR]
RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^lftp [OR]
RewriteCond %{HTTP_USER_AGENT} ^likse [OR]
RewriteCond %{HTTP_USER_AGENT} ^Magnet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mag-Net [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^Memo [OR]
RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mirror [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR]
RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]
RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetZip [OR]
RewriteCond %{HTTP_USER_AGENT} ^Ninja [OR]
RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [OR]
RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR]
RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]
RewriteCond %{HTTP_USER_AGENT} ^Pockey [OR]
RewriteCond %{HTTP_USER_AGENT} ^Pump [OR]
RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^Reaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Recorder [OR]
RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Siphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]
RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^Snake [OR]
RewriteCond %{HTTP_USER_AGENT} ^SpaceBison [OR]
RewriteCond %{HTTP_USER_AGENT} ^Stripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [OR]
RewriteCond %{HTTP_USER_AGENT} ^Vacuum [OR]
RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website [OR]
RewriteCond %{HTTP_USER_AGENT} ^Webster [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^Whacker [OR]
RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]
RewriteCond %{HTTP_USER_AGENT} ^Xaldon
RewriteRule ^.* - [F,L]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/4\.0$ [OR]
RewriteCOnd %{QUERY_STRING} Esystem [OR]
RewriteCond %{HTTP_USER_AGENT} ^-$ [OR]
RewriteCond %{HTTP_REFERER} ^-$
RewriteRule .* - [F]
And currently my robots.txt file details are given below: User-agent: *
Disallow: /cgi-bin/
Disallow: /cron/
Disallow: /admin/
Disallow: /backup/
Disallow: /includes/
Disallow: /images/
Disallow: /lib/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /blog/guestbook/
Disallow: /blog/conf/
Disallow: /blog/getdetail/
Disallow: /blog/getpic/
Disallow: /blog/getthumb/
Disallow: /blog/htsrv/
Disallow: /blog/image/
Disallow: /blog/inc/
Disallow: /blog/locales/
Disallow: /blog/media/
Disallow: /blog/plugins/
Disallow: /blog/rsc/
Disallow: /blog/xmlsrv/
Disallow: /blog/feeds/
Disallow: /blog/a/
Disallow: /forum/admin/
Disallow: /forum/images/
Disallow: /forum/inc/
Disallow: /forum/uploads/
Disallow: /forum/jscripts
Disallow: /forum/cache
Disallow: /forum/captcha.php
Disallow: /forum/editpost.php
Disallow: /forum/misc.php
Disallow: /forum/modcp.php
Disallow: /forum/moderation.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/ratethread.php
Disallow: /forum/report.php
Disallow: /forum/reputation.php
Disallow: /forum/search.php
Disallow: /forum/sendthread.php
Disallow: /forum/task.php
Disallow: /forum/usercp.php
Disallow: /forum/usercp2.php
Disallow: /forum/calendar.php
Disallow: /forum/*action=emailuser*
Disallow: /forum/*action=nextnewest*
Disallow: /forum/*action=nextoldest*
Disallow: /forum/*year=*
Disallow: /forum/*action=weekview*
Disallow: /forum/*action=nextnewest*
Disallow: /forum/*action=nextoldest*
Disallow: /forum/*sort=*
Disallow: /forum/*order=*
Disallow: /forum/*mode=*
Disallow: /forum/*datecut=*
these code's are currently placed in my robots.txt and .htaccess file but after adding this code also i found the same results given below in my awstats.
Unknown robot (identified by empty user agent string)
Unknown robot (identified by 'spider')
Unknown robot (identified by 'robot')
Unknown robot (identified by 'bot*')
Unknown robot (identified by 'crawl')
Unknown robot (identified by hit on 'robots.txt') Please some body fix my issue thanks in advance.
my website address is [
classicsads.com...]
