Welcome to WebmasterWorld Guest from 54.147.44.93

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Use FP 2000 and Want to Keep Goo from Indexing It
How?
7 ember 12:19 pm May 31, 2007
Allow google to crawl on only primary domains
How to achieve exclusion on multiple domains using obj.conf & robots.txt
3 justlonging1 12:35 pm May 30, 2007
block all except in robots.txt
2 visualscope 2:26 pm May 28, 2007
What is a bot trap?
2 fischermx 4:28 pm May 26, 2007
Custom rule for google, url args and wordpress feeds
6 Kurt 2:28 pm May 24, 2007
Wordpress - Robots.txt to fix Duplicate Content
I need a robots.txt file to prevent dup content from being crawled
7 jman123 2:16 pm May 23, 2007
Alexa robot
issue with alexa bot & alexa toolbar users
6 Michel_Samuel 10:45 am May 20, 2007
Robots.txt
Can you exclude a parameter in robots.txt?
3 North_Shore 9:22 pm May 18, 2007
robots.txt mess-up
4 strawbleu 9:12 pm May 18, 2007
XML Stemap
6 ajitaryan 11:35 am May 16, 2007
Is this valid?
3 aha7 12:34 pm May 9, 2007
Disallowing content previously indexed
What happens to indexed content subsequently blocked by robots.txt
3 daveozzz 11:22 am May 9, 2007
duplicate content indexing
11 coburn 6:34 pm May 7, 2007
Denying one page with multiple variables
page.html?item=1, page.html?item=2, page.html?item=3
7 wfernley 6:21 pm May 7, 2007
Can anyone post a complete robots.txt file here?
4 PowerUp 2:29 pm May 7, 2007
How do I disallow bots from my Wordpress year Archive Folders
I guess this is a common question but I can't find an answer
4 Drum 5:26 pm May 5, 2007
Add Sitemap To robots.txt For Autodiscovery
Ask.com, Google, Microsoft Live Search and Yahoo! [2] ( 1 2 )
32 OldWolf 7:56 am May 4, 2007
How is alexa getting past my stoppers
stop alexa index
1 Michel_Samuel 3:31 pm May 2, 2007
how to disallow all robots except Googlebot, Yahoo Slurp and MSNBot?
8 d6rth7ader 5:48 am May 2, 2007
Htaccess
3 harish_s 2:17 pm May 1, 2007
Limiting the depth of Google's crawl?
Can robots.txt be used to limit depth
6 daveozzz 7:21 pm Apr 27, 2007
Disallow: /
How did googlebot index my web server dir structure?
3 thinkweb 5:24 am Apr 26, 2007
URLs restricted by robots.txt (41) AGH!
I'm a bit stumped about what's best to do re my robots.txt
8 wrafter 8:38 pm Apr 25, 2007
Is a robots.txt file necessary?
if not trying to block bots?
3 Jakes_Redding 5:52 am Apr 20, 2007
Googlebot ignoring robots.txt?
Or is my robots.txt file wrong...
3 designaweb 9:28 am Apr 17, 2007

Featured Threads

Hot Threads This Week

Hot Threads This Month