homepage Welcome to WebmasterWorld Guest from 174.129.74.186
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconBroken Links on my web site
  I have too many broken links
2erdsah88 12:47 pm on Jul 16, 2007 (gmt 0)
newpost thread icon"Disallow: //" - error or meaningful?
  robots.txt crawler error
5sitetruth 11:58 pm on Jul 11, 2007 (gmt 0)
newpost thread iconBLOX dynamic urls with robots.txt
  BLOX dynamic urls with robots.txt
2bo0oost 11:50 am on Jul 11, 2007 (gmt 0)
newpost thread iconBlock affiliate tracking
  but keep the main page in Google
5chubba 2:26 pm on Jul 9, 2007 (gmt 0)
newpost thread iconWe can put sitempp.html to robots.txt?
  need some advise
2etgsgroup 12:00 pm on Jul 6, 2007 (gmt 0)
newpost thread iconA question about blocking IP from Nigerians
 
2rjr686 10:56 pm on Jul 3, 2007 (gmt 0)
newpost thread iconGoogle and cached, obsolete robots.txt
 
5MamaDawg 10:59 am on Jul 3, 2007 (gmt 0)
newpost thread iconIdbot
  IDBOT hammered our site, now banned
1JohnCanyon 5:21 pm on Jun 29, 2007 (gmt 0)
newpost thread iconDisallow a single image?
  Is it possible to block the search engines from indexing a single image?
4PrattTA1 4:59 pm on Jun 26, 2007 (gmt 0)
newpost thread iconBlocking Site Server 3.0 Robot
  How?
13Matt_Probert 5:53 am on Jun 23, 2007 (gmt 0)
newpost thread iconA list of american search engines.
  search engine list. American.
9Michel_Samuel 12:49 am on Jun 22, 2007 (gmt 0)
newpost thread iconDoes this trailing slash mean anything?
  <meta name="robots" content="noindex" />
11bouncybunny 7:10 pm on Jun 20, 2007 (gmt 0)
newpost thread iconNeed help with a specific robots.txt question
 
4Kurgano 9:18 pm on Jun 12, 2007 (gmt 0)
newpost thread iconabout:blank >>>what is this
  then the page loads?
4twtnyc 11:44 pm on Jun 8, 2007 (gmt 0)
newpost thread iconAllow rather than Disallow
  Is it possible?
5Marshall 10:30 pm on Jun 7, 2007 (gmt 0)
newpost thread iconWhy is msnbot ignoring this robot's.txt rule?
  Wildcard disallow ignored
3bouncybunny 12:17 pm on Jun 6, 2007 (gmt 0)
newpost thread iconBlock all Nutch variants. Possible?
 
6morags 9:02 pm on Jun 5, 2007 (gmt 0)
newpost thread iconUse FP 2000 and Want to Keep Goo from Indexing It
  How?
7ember 12:19 pm on May 31, 2007 (gmt 0)
newpost thread iconAllow google to crawl on only primary domains
  How to achieve exclusion on multiple domains using obj.conf & robots.txt
3justlonging1 12:35 pm on May 30, 2007 (gmt 0)
newpost thread iconblock all except in robots.txt
 
2visualscope 2:26 pm on May 28, 2007 (gmt 0)
newpost thread iconWhat is a bot trap?
 
2fischermx 4:28 pm on May 26, 2007 (gmt 0)
newpost thread iconCustom rule for google, url args and wordpress feeds
 
6Kurt 2:28 pm on May 24, 2007 (gmt 0)
newpost thread iconWordpress - Robots.txt to fix Duplicate Content
  I need a robots.txt file to prevent dup content from being crawled
7jman123 2:16 pm on May 23, 2007 (gmt 0)
newpost thread iconAlexa robot
  issue with alexa bot & alexa toolbar users
6Michel_Samuel 10:45 am on May 20, 2007 (gmt 0)
newpost thread iconRobots.txt
  Can you exclude a parameter in robots.txt?
3North_Shore 9:22 pm on May 18, 2007 (gmt 0)
Displaying Page 28 of 72<< Back 1 ... 18 19 20 21 22 23 24 25 26 27 [ 28 ] 29 30 31 32 33 34 35 36 37 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved