homepage Welcome to WebmasterWorld Guest from 54.226.213.228
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconBlocking Partial Contents of Folder Without Using Robots Tag
  10 files (7 want indexed, 3 want blocked)
5SandySEO 8:49 pm on Mar 27, 2007 (gmt 0)
newpost thread iconAnother bad bot to ban
  this one is using 302 redirects .. bad bad
1Bewenched 3:43 am on Mar 25, 2007 (gmt 0)
newpost thread iconexcluding a subdomain
  Using the robots.txt file to exclude a subdomain
10joebray 9:47 am on Mar 23, 2007 (gmt 0)
newpost thread iconG. ignores robots.txt at other ports, gives false page count
  Google shows in its index pages that it never saw, even prohibited
1OnetSzukaj 2:58 pm on Mar 22, 2007 (gmt 0)
newpost thread iconHow to use Robots.txt as an INCLUSION method
  It's exclusion only if you use it that way
1incrediBILL 12:32 am on Mar 22, 2007 (gmt 0)
newpost thread iconTripAdvisorBot
  Anyone seen this?
3bobbarnes 10:28 pm on Mar 16, 2007 (gmt 0)
newpost thread iconproblem with msnbot and wildcard matching
 
2Achernar 1:36 pm on Mar 16, 2007 (gmt 0)
newpost thread iconHow Can I Make Bots Visit My Web Site?
 
3emadido 2:34 pm on Mar 15, 2007 (gmt 0)
newpost thread iconwhy i can't reaply the topic
  why i can't reaply the topic
3unknown 8:43 am on Mar 14, 2007 (gmt 0)
newpost thread iconrobotsxx.txt not found.
 
2benevolent001 1:16 pm on Mar 13, 2007 (gmt 0)
newpost thread iconSomething a little different
 
2futureX 1:56 pm on Mar 12, 2007 (gmt 0)
newpost thread iconWhy is WebmasterWorld robot validator giving back HTML
 
3Clair 7:11 pm on Mar 10, 2007 (gmt 0)
newpost thread iconWhen Robots.txt And Meta Robots Collide
  What happens?
13Easy_Coder 1:57 am on Mar 10, 2007 (gmt 0)
newpost thread iconOSC, dupe content, robots.txt and 301 redirects
  Need to ensure bots 301 to new SEO URL
5spina45 8:23 pm on Mar 9, 2007 (gmt 0)
newpost thread iconDisallow / access to our directory
  disallow
2drew68 3:47 pm on Mar 6, 2007 (gmt 0)
newpost thread iconBlock the main page of a directory but not its contents?
 
6bresso 7:39 am on Mar 6, 2007 (gmt 0)
newpost thread iconRobots.txt and pointing domains
 
3Shanada 6:58 am on Feb 28, 2007 (gmt 0)
newpost thread iconWill Googlebot be able to crawl these pages of my website?
 
4saqitude 3:40 pm on Feb 25, 2007 (gmt 0)
newpost thread iconrobots.txt question
 
2seomack 8:14 am on Feb 19, 2007 (gmt 0)
newpost thread iconIs correct this robots.txt?
 
2MalastiC 9:44 pm on Feb 18, 2007 (gmt 0)
newpost thread iconSitemap and robot.txt
 
3spuddy 3:53 pm on Feb 15, 2007 (gmt 0)
newpost thread iconHow to exclude from crawling subdomain
  exclude the crawling subdomain
4fendy 1:00 pm on Feb 15, 2007 (gmt 0)
newpost thread iconConfused over Google spider and robots.txt
 
4flowermark 12:56 pm on Feb 15, 2007 (gmt 0)
newpost thread iconWhich bots to block?
  robot.txt
2cyclesolutions 9:51 pm on Feb 12, 2007 (gmt 0)
newpost thread iconAccidentally blocked root access, doh!
 
3ninjaNeen 6:51 pm on Feb 8, 2007 (gmt 0)
Displaying Page 30 of 72<< Back 1 ... 20 21 22 23 24 25 26 27 28 29 [ 30 ] 31 32 33 34 35 36 37 38 39 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved