homepage Welcome to WebmasterWorld Guest from 54.145.209.80
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconHow to exclude from crawling subdomain
  exclude the crawling subdomain
4fendy 1:00 pm on Feb 15, 2007 (gmt 0)
newpost thread iconConfused over Google spider and robots.txt
 
4flowermark 12:56 pm on Feb 15, 2007 (gmt 0)
newpost thread iconWhich bots to block?
  robot.txt
2cyclesolutions 9:51 pm on Feb 12, 2007 (gmt 0)
newpost thread iconAccidentally blocked root access, doh!
 
3ninjaNeen 6:51 pm on Feb 8, 2007 (gmt 0)
newpost thread iconrobots.txt usage and examples
  Using robots.txt to improve SEO
3cduke250 3:50 pm on Feb 7, 2007 (gmt 0)
newpost thread icondisallowing subdomain access rule
 
1coburn 1:37 pm on Feb 7, 2007 (gmt 0)
newpost thread iconcan put two robots at one line at robot.txt to ban two spiders?
  robots.txt, ban google, yahoo bots
4johnlim9988 3:54 am on Feb 7, 2007 (gmt 0)
newpost thread iconBlocking subdir, but allowing crawl in deeper dir?
 
7Vulcan315 3:52 pm on Feb 6, 2007 (gmt 0)
newpost thread iconrobots noindex on an iframe page
  What would this do?
2dataguy 1:20 pm on Feb 3, 2007 (gmt 0)
newpost thread iconSlowing down MSN bot?
  Doubling it's bandwidth usage each month.
4bouncybunny 3:33 am on Feb 3, 2007 (gmt 0)
newpost thread iconSlurps and wildcards
 
5Vimes 1:36 am on Feb 3, 2007 (gmt 0)
newpost thread iconrobots.txt blocking my index page?
  Hello
4hola2 2:08 pm on Feb 2, 2007 (gmt 0)
newpost thread iconRobots.txt command? "do not search anything outside this."
 
5OWizeOne 1:08 pm on Feb 1, 2007 (gmt 0)
newpost thread iconHow do I block yahoo's SE spider and allow their image spider?
  Does Yahoo send a different spider for the image search?
4Clark 5:18 pm on Jan 27, 2007 (gmt 0)
newpost thread iconHow robot.txt works
  Robot.txt
3walmslei 4:10 am on Jan 27, 2007 (gmt 0)
newpost thread iconWhy is google indexing this banned directory?
 
5ichthyous 4:55 pm on Jan 26, 2007 (gmt 0)
newpost thread iconhttps on google and robots.txt
 
2concepts99 1:54 pm on Jan 24, 2007 (gmt 0)
newpost thread icondirectory structure question
 
5union_jack 7:43 pm on Jan 22, 2007 (gmt 0)
newpost thread iconblock a subdirectory
 
2concepts99 2:05 pm on Jan 22, 2007 (gmt 0)
newpost thread iconGoogle not respecting robots.txt
  Found >500 pages in Google index that are forbiden by robots.txt
5roots 5:54 am on Jan 22, 2007 (gmt 0)
newpost thread icondomo arigato
  mr. roboto
3ricnut 11:52 pm on Jan 20, 2007 (gmt 0)
newpost thread iconDisallow Slurp from all files except two
  Yahoo likes only one page, so I'd like to just 'give' it that.
2tigertom 3:00 am on Jan 20, 2007 (gmt 0)
newpost thread iconYahoo Slurp Bot misbehaving
  .. or is it someone masquerading?
4tigertom 9:27 am on Jan 17, 2007 (gmt 0)
newpost thread iconBlocking Site in some countries
 
3eldinboy 5:55 pm on Jan 15, 2007 (gmt 0)
newpost thread iconWhat to do with duplicate pages
 
3olba 2:12 am on Jan 12, 2007 (gmt 0)
Displaying Page 31 of 72<< Back 1 ... 21 22 23 24 25 26 27 28 29 30 [ 31 ] 32 33 34 35 36 37 38 39 40 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved