homepage Welcome to WebmasterWorld Guest from 54.146.143.102
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconrobots.txt question
 
2seomack 8:14 am on Feb 19, 2007 (gmt 0)
newpost thread iconIs correct this robots.txt?
 
2MalastiC 9:44 pm on Feb 18, 2007 (gmt 0)
newpost thread iconSitemap and robot.txt
 
3spuddy 3:53 pm on Feb 15, 2007 (gmt 0)
newpost thread iconHow to exclude from crawling subdomain
  exclude the crawling subdomain
4fendy 1:00 pm on Feb 15, 2007 (gmt 0)
newpost thread iconConfused over Google spider and robots.txt
 
4flowermark 12:56 pm on Feb 15, 2007 (gmt 0)
newpost thread iconWhich bots to block?
  robot.txt
2cyclesolutions 9:51 pm on Feb 12, 2007 (gmt 0)
newpost thread iconAccidentally blocked root access, doh!
 
3ninjaNeen 6:51 pm on Feb 8, 2007 (gmt 0)
newpost thread iconrobots.txt usage and examples
  Using robots.txt to improve SEO
3cduke250 3:50 pm on Feb 7, 2007 (gmt 0)
newpost thread icondisallowing subdomain access rule
 
1coburn 1:37 pm on Feb 7, 2007 (gmt 0)
newpost thread iconcan put two robots at one line at robot.txt to ban two spiders?
  robots.txt, ban google, yahoo bots
4johnlim9988 3:54 am on Feb 7, 2007 (gmt 0)
newpost thread iconBlocking subdir, but allowing crawl in deeper dir?
 
7Vulcan315 3:52 pm on Feb 6, 2007 (gmt 0)
newpost thread iconrobots noindex on an iframe page
  What would this do?
2dataguy 1:20 pm on Feb 3, 2007 (gmt 0)
newpost thread iconSlowing down MSN bot?
  Doubling it's bandwidth usage each month.
4bouncybunny 3:33 am on Feb 3, 2007 (gmt 0)
newpost thread iconSlurps and wildcards
 
5Vimes 1:36 am on Feb 3, 2007 (gmt 0)
newpost thread iconrobots.txt blocking my index page?
  Hello
4hola2 2:08 pm on Feb 2, 2007 (gmt 0)
newpost thread iconRobots.txt command? "do not search anything outside this."
 
5OWizeOne 1:08 pm on Feb 1, 2007 (gmt 0)
newpost thread iconHow do I block yahoo's SE spider and allow their image spider?
  Does Yahoo send a different spider for the image search?
4Clark 5:18 pm on Jan 27, 2007 (gmt 0)
newpost thread iconHow robot.txt works
  Robot.txt
3walmslei 4:10 am on Jan 27, 2007 (gmt 0)
newpost thread iconWhy is google indexing this banned directory?
 
5ichthyous 4:55 pm on Jan 26, 2007 (gmt 0)
newpost thread iconhttps on google and robots.txt
 
2concepts99 1:54 pm on Jan 24, 2007 (gmt 0)
newpost thread icondirectory structure question
 
5union_jack 7:43 pm on Jan 22, 2007 (gmt 0)
newpost thread iconblock a subdirectory
 
2concepts99 2:05 pm on Jan 22, 2007 (gmt 0)
newpost thread iconGoogle not respecting robots.txt
  Found >500 pages in Google index that are forbiden by robots.txt
5roots 5:54 am on Jan 22, 2007 (gmt 0)
newpost thread icondomo arigato
  mr. roboto
3ricnut 11:52 pm on Jan 20, 2007 (gmt 0)
newpost thread iconDisallow Slurp from all files except two
  Yahoo likes only one page, so I'd like to just 'give' it that.
2tigertom 3:00 am on Jan 20, 2007 (gmt 0)
Displaying Page 31 of 72<< Back 1 ... 21 22 23 24 25 26 27 28 29 30 [ 31 ] 32 33 34 35 36 37 38 39 40 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved