Welcome to WebmasterWorld Guest from 220.127.116.11
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|We can put sitempp.html to robots.txt?
need some advise
|2||etgsgroup||12:00 pm Jul 6, 2007|
|A question about blocking IP from Nigerians
||2||rjr686||10:56 pm Jul 3, 2007|
|Google and cached, obsolete robots.txt
||5||MamaDawg||10:59 am Jul 3, 2007|
IDBOT hammered our site, now banned
|1||JohnCanyon||5:21 pm Jun 29, 2007|
|Disallow a single image?
Is it possible to block the search engines from indexing a single image?
|4||PrattTA1||4:59 pm Jun 26, 2007|
|Blocking Site Server 3.0 Robot
|13||Matt_Probert||5:53 am Jun 23, 2007|
|A list of american search engines.
search engine list. American.
|9||Michel_Samuel||12:49 am Jun 22, 2007|
|Does this trailing slash mean anything?
<meta name="robots" content="noindex" />
|11||bouncybunny||7:10 pm Jun 20, 2007|
|Need help with a specific robots.txt question
||4||Kurgano||9:18 pm Jun 12, 2007|
|about:blank >>>what is this
then the page loads?
|4||twtnyc||11:44 pm Jun 8, 2007|
|Allow rather than Disallow
Is it possible?
|5||Marshall||10:30 pm Jun 7, 2007|
|Why is msnbot ignoring this robot's.txt rule?
Wildcard disallow ignored
|3||bouncybunny||12:17 pm Jun 6, 2007|
|Block all Nutch variants. Possible?
||6||morags||9:02 pm Jun 5, 2007|
|Use FP 2000 and Want to Keep Goo from Indexing It
|7||ember||12:19 pm May 31, 2007|
|Allow google to crawl on only primary domains
How to achieve exclusion on multiple domains using obj.conf & robots.txt
|3||justlonging1||12:35 pm May 30, 2007|
|block all except in robots.txt
||2||visualscope||2:26 pm May 28, 2007|
|What is a bot trap?
||2||fischermx||4:28 pm May 26, 2007|
|Custom rule for google, url args and wordpress feeds
||6||Kurt||2:28 pm May 24, 2007|
|Wordpress - Robots.txt to fix Duplicate Content
I need a robots.txt file to prevent dup content from being crawled
|7||jman123||2:16 pm May 23, 2007|
issue with alexa bot & alexa toolbar users
|6||Michel_Samuel||10:45 am May 20, 2007|
Can you exclude a parameter in robots.txt?
|3||North_Shore||9:22 pm May 18, 2007|
||4||strawbleu||9:12 pm May 18, 2007|
||6||ajitaryan||11:35 am May 16, 2007|
|Is this valid?
||3||aha7||12:34 pm May 9, 2007|
|Disallowing content previously indexed
What happens to indexed content subsequently blocked by robots.txt
|3||daveozzz||11:22 am May 9, 2007|