homepage Welcome to WebmasterWorld Guest from 50.16.24.12
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconWhy is msnbot ignoring this robot's.txt rule?
  Wildcard disallow ignored
3bouncybunny 12:17 pm on Jun 6, 2007 (gmt 0)
newpost thread iconBlock all Nutch variants. Possible?
 
6morags 9:02 pm on Jun 5, 2007 (gmt 0)
newpost thread iconUse FP 2000 and Want to Keep Goo from Indexing It
  How?
7ember 12:19 pm on May 31, 2007 (gmt 0)
newpost thread iconAllow google to crawl on only primary domains
  How to achieve exclusion on multiple domains using obj.conf & robots.txt
3justlonging1 12:35 pm on May 30, 2007 (gmt 0)
newpost thread iconblock all except in robots.txt
 
2visualscope 2:26 pm on May 28, 2007 (gmt 0)
newpost thread iconWhat is a bot trap?
 
2fischermx 4:28 pm on May 26, 2007 (gmt 0)
newpost thread iconCustom rule for google, url args and wordpress feeds
 
6Kurt 2:28 pm on May 24, 2007 (gmt 0)
newpost thread iconWordpress - Robots.txt to fix Duplicate Content
  I need a robots.txt file to prevent dup content from being crawled
7jman123 2:16 pm on May 23, 2007 (gmt 0)
newpost thread iconAlexa robot
  issue with alexa bot & alexa toolbar users
6Michel_Samuel 10:45 am on May 20, 2007 (gmt 0)
newpost thread iconRobots.txt
  Can you exclude a parameter in robots.txt?
3North_Shore 9:22 pm on May 18, 2007 (gmt 0)
newpost thread iconrobots.txt mess-up
 
4strawbleu 9:12 pm on May 18, 2007 (gmt 0)
newpost thread iconXML Stemap
 
6ajitaryan 11:35 am on May 16, 2007 (gmt 0)
newpost thread iconIs this valid?
 
3aha7 12:34 pm on May 9, 2007 (gmt 0)
newpost thread iconDisallowing content previously indexed
  What happens to indexed content subsequently blocked by robots.txt
3daveozzz 11:22 am on May 9, 2007 (gmt 0)
newpost thread iconduplicate content indexing
 
11coburn 6:34 pm on May 7, 2007 (gmt 0)
newpost thread iconDenying one page with multiple variables
  page.html?item=1, page.html?item=2, page.html?item=3
7wfernley 6:21 pm on May 7, 2007 (gmt 0)
newpost thread iconCan anyone post a complete robots.txt file here?
 
4PowerUp 2:29 pm on May 7, 2007 (gmt 0)
newpost thread iconHow do I disallow bots from my Wordpress year Archive Folders
  I guess this is a common question but I can't find an answer
4Drum 5:26 pm on May 5, 2007 (gmt 0)
newpost thread iconAdd Sitemap To robots.txt For Autodiscovery[2] ( 1 2 )
  Ask.com, Google, Microsoft Live Search and Yahoo!
32OldWolf 7:56 am on May 4, 2007 (gmt 0)
newpost thread iconHow is alexa getting past my stoppers
  stop alexa index
1Michel_Samuel 3:31 pm on May 2, 2007 (gmt 0)
newpost thread iconhow to disallow all robots except Googlebot, Yahoo Slurp and MSNBot?
 
8d6rth7ader 5:48 am on May 2, 2007 (gmt 0)
newpost thread iconHtaccess
 
3harish_s 2:17 pm on May 1, 2007 (gmt 0)
newpost thread iconLimiting the depth of Google's crawl?
  Can robots.txt be used to limit depth
6daveozzz 7:21 pm on Apr 27, 2007 (gmt 0)
newpost thread iconDisallow: /
  How did googlebot index my web server dir structure?
3thinkweb 5:24 am on Apr 26, 2007 (gmt 0)
newpost thread iconURLs restricted by robots.txt (41) AGH!
  I'm a bit stumped about what's best to do re my robots.txt
8wrafter 8:38 pm on Apr 25, 2007 (gmt 0)
Displaying Page 29 of 72<< Back 1 ... 19 20 21 22 23 24 25 26 27 28 [ 29 ] 30 31 32 33 34 35 36 37 38 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved