homepage Welcome to WebmasterWorld Guest from 54.161.146.154
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconWhy is WebmasterWorld robot validator giving back HTML
 
3Clair 7:11 pm on Mar 10, 2007 (gmt 0)
newpost thread iconWhen Robots.txt And Meta Robots Collide
  What happens?
13Easy_Coder 1:57 am on Mar 10, 2007 (gmt 0)
newpost thread iconOSC, dupe content, robots.txt and 301 redirects
  Need to ensure bots 301 to new SEO URL
5spina45 8:23 pm on Mar 9, 2007 (gmt 0)
newpost thread iconDisallow / access to our directory
  disallow
2drew68 3:47 pm on Mar 6, 2007 (gmt 0)
newpost thread iconBlock the main page of a directory but not its contents?
 
6bresso 7:39 am on Mar 6, 2007 (gmt 0)
newpost thread iconRobots.txt and pointing domains
 
3Shanada 6:58 am on Feb 28, 2007 (gmt 0)
newpost thread iconWill Googlebot be able to crawl these pages of my website?
 
4saqitude 3:40 pm on Feb 25, 2007 (gmt 0)
newpost thread iconrobots.txt question
 
2seomack 8:14 am on Feb 19, 2007 (gmt 0)
newpost thread iconIs correct this robots.txt?
 
2MalastiC 9:44 pm on Feb 18, 2007 (gmt 0)
newpost thread iconSitemap and robot.txt
 
3spuddy 3:53 pm on Feb 15, 2007 (gmt 0)
newpost thread iconHow to exclude from crawling subdomain
  exclude the crawling subdomain
4fendy 1:00 pm on Feb 15, 2007 (gmt 0)
newpost thread iconConfused over Google spider and robots.txt
 
4flowermark 12:56 pm on Feb 15, 2007 (gmt 0)
newpost thread iconWhich bots to block?
  robot.txt
2cyclesolutions 9:51 pm on Feb 12, 2007 (gmt 0)
newpost thread iconAccidentally blocked root access, doh!
 
3ninjaNeen 6:51 pm on Feb 8, 2007 (gmt 0)
newpost thread iconrobots.txt usage and examples
  Using robots.txt to improve SEO
3cduke250 3:50 pm on Feb 7, 2007 (gmt 0)
newpost thread icondisallowing subdomain access rule
 
1coburn 1:37 pm on Feb 7, 2007 (gmt 0)
newpost thread iconcan put two robots at one line at robot.txt to ban two spiders?
  robots.txt, ban google, yahoo bots
4johnlim9988 3:54 am on Feb 7, 2007 (gmt 0)
newpost thread iconBlocking subdir, but allowing crawl in deeper dir?
 
7Vulcan315 3:52 pm on Feb 6, 2007 (gmt 0)
newpost thread iconrobots noindex on an iframe page
  What would this do?
2dataguy 1:20 pm on Feb 3, 2007 (gmt 0)
newpost thread iconSlowing down MSN bot?
  Doubling it's bandwidth usage each month.
4bouncybunny 3:33 am on Feb 3, 2007 (gmt 0)
newpost thread iconSlurps and wildcards
 
5Vimes 1:36 am on Feb 3, 2007 (gmt 0)
newpost thread iconrobots.txt blocking my index page?
  Hello
4hola2 2:08 pm on Feb 2, 2007 (gmt 0)
newpost thread iconRobots.txt command? "do not search anything outside this."
 
5OWizeOne 1:08 pm on Feb 1, 2007 (gmt 0)
newpost thread iconHow do I block yahoo's SE spider and allow their image spider?
  Does Yahoo send a different spider for the image search?
4Clark 5:18 pm on Jan 27, 2007 (gmt 0)
newpost thread iconHow robot.txt works
  Robot.txt
3walmslei 4:10 am on Jan 27, 2007 (gmt 0)
Displaying Page 31 of 72<< Back 1 ... 21 22 23 24 25 26 27 28 29 30 [ 31 ] 32 33 34 35 36 37 38 39 40 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved