Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Thousands of Spambot IPs Hitting my Site

spiritualseo - 5:58 pm on Dec 12, 2011 (gmt 0)


Thanks Lucy24, so will this be it:

#Options +FollowSymlinks
RewriteEngine on
RewriteCond %{HTTP_REFERER} ^-?$
RewriteRule ^foldername/page\.php$ - [F,L]

By the way, will blocking blank referrers also blog genuine bots like the Googlebot? I tried fetching my site under Google webmaster central but Googlebot could not fetch the page. It was 403 forbidden. Is that an indication that Google cannot crawl my site?

Thread source:: http://www.webmasterworld.com/search_engine_spiders/4396669.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com