Welcome to WebmasterWorld Guest from 54.224.102.26

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Does this trailing slash mean anything?
<meta name="robots" content="noindex" />
11 bouncybunny 7:10 pm Jun 20, 2007
Need help with a specific robots.txt question
4 Kurgano 9:18 pm Jun 12, 2007
about:blank >>>what is this
then the page loads?
4 twtnyc 11:44 pm Jun 8, 2007
Allow rather than Disallow
Is it possible?
5 Marshall 10:30 pm Jun 7, 2007
Why is msnbot ignoring this robot's.txt rule?
Wildcard disallow ignored
3 bouncybunny 12:17 pm Jun 6, 2007
Block all Nutch variants. Possible?
6 morags 9:02 pm Jun 5, 2007
Use FP 2000 and Want to Keep Goo from Indexing It
How?
7 ember 12:19 pm May 31, 2007
Allow google to crawl on only primary domains
How to achieve exclusion on multiple domains using obj.conf & robots.txt
3 justlonging1 12:35 pm May 30, 2007
block all except in robots.txt
2 visualscope 2:26 pm May 28, 2007
What is a bot trap?
2 fischermx 4:28 pm May 26, 2007
Custom rule for google, url args and wordpress feeds
6 Kurt 2:28 pm May 24, 2007
Wordpress - Robots.txt to fix Duplicate Content
I need a robots.txt file to prevent dup content from being crawled
7 jman123 2:16 pm May 23, 2007
Alexa robot
issue with alexa bot & alexa toolbar users
6 Michel_Samuel 10:45 am May 20, 2007
Robots.txt
Can you exclude a parameter in robots.txt?
3 North_Shore 9:22 pm May 18, 2007
robots.txt mess-up
4 strawbleu 9:12 pm May 18, 2007
XML Stemap
6 ajitaryan 11:35 am May 16, 2007
Is this valid?
3 aha7 12:34 pm May 9, 2007
Disallowing content previously indexed
What happens to indexed content subsequently blocked by robots.txt
3 daveozzz 11:22 am May 9, 2007
duplicate content indexing
11 coburn 6:34 pm May 7, 2007
Denying one page with multiple variables
page.html?item=1, page.html?item=2, page.html?item=3
7 wfernley 6:21 pm May 7, 2007
Can anyone post a complete robots.txt file here?
4 PowerUp 2:29 pm May 7, 2007
How do I disallow bots from my Wordpress year Archive Folders
I guess this is a common question but I can't find an answer
4 Drum 5:26 pm May 5, 2007
Add Sitemap To robots.txt For Autodiscovery
Ask.com, Google, Microsoft Live Search and Yahoo! [2] ( 1 2 )
32 OldWolf 7:56 am May 4, 2007
How is alexa getting past my stoppers
stop alexa index
1 Michel_Samuel 3:31 pm May 2, 2007
how to disallow all robots except Googlebot, Yahoo Slurp and MSNBot?
8 d6rth7ader 5:48 am May 2, 2007