homepage Welcome to WebmasterWorld Guest from 54.226.191.80
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconSpiders finding you against all odds
 
3Lyndsay 6:49 pm on Jan 23, 2006 (gmt 0)
newpost thread iconyahoo/slurp spidering
  We don't want these pages indexed by yahoo
9latimer 8:28 am on Jan 23, 2006 (gmt 0)
newpost thread iconCorrect way to ban a directory
 
5palmpal 3:18 am on Jan 23, 2006 (gmt 0)
newpost thread iconWWW vs. Non WWW
  Site indexed twice need robot.txt help
3carjocky 4:10 pm on Jan 20, 2006 (gmt 0)
newpost thread iconUnknowingly restricting some directories
 
2estaquieto 4:40 am on Jan 20, 2006 (gmt 0)
newpost thread iconMSNBOT and robots.txt
 
2Impala 4:31 am on Jan 20, 2006 (gmt 0)
newpost thread iconHow-to: robots.txt white-listing with PHP/Apache
 
2encyclo 2:00 pm on Jan 16, 2006 (gmt 0)
newpost thread iconProhibiting folders by name regardless of root relation
  Is this possible or not?
6Jeremy_H 2:38 pm on Jan 15, 2006 (gmt 0)
newpost thread iconquestion about robots.txt, am I banning a directory by accident
 
17arbitrary 3:32 am on Jan 15, 2006 (gmt 0)
newpost thread iconQuick 'Sanity Check'
 
3markd 10:20 am on Jan 11, 2006 (gmt 0)
newpost thread icondirectory
 
4sylvieg 4:38 am on Jan 11, 2006 (gmt 0)
newpost thread iconRemoving pages from Google's cache
  Is it best to allow a "robots" "noarchive" crawl first?
1Phil_Payne 2:05 am on Jan 10, 2006 (gmt 0)
newpost thread iconDisallow ALL folders?
  but allow top level html
6Reno 9:14 pm on Jan 9, 2006 (gmt 0)
newpost thread iconHow to disallow widgets from being indexed
  Order of User-agents
2albatros 8:10 pm on Jan 9, 2006 (gmt 0)
newpost thread iconEmpty line after User-agent:
  Could this do any harm?
1taps 5:31 pm on Jan 7, 2006 (gmt 0)
newpost thread iconRobot Validator Problem
 
7bunky6767 1:13 pm on Jan 6, 2006 (gmt 0)
newpost thread iconBlocking all URLS to a website using the robots.txt file?
 
2scubajared 10:05 am on Jan 6, 2006 (gmt 0)
newpost thread iconTuring keys
 
2Ejap 5:57 pm on Jan 5, 2006 (gmt 0)
newpost thread iconABC of robots.txt
  Now that will be the best tutorial!
8Monzilla 2:19 pm on Jan 4, 2006 (gmt 0)
newpost thread iconDefine allowable spider times[2] ( 1 2 )
  MSN is going nutz!
31Reno 1:44 pm on Jan 4, 2006 (gmt 0)
newpost thread iconNew empty site beeing spidered
 
2sren 3:08 am on Dec 29, 2005 (gmt 0)
newpost thread iconremove PHP from Google.com
  remove PHP from Google.com
2maimax 7:32 am on Dec 28, 2005 (gmt 0)
newpost thread iconpaging google! robots.txt being ignored!
 
14elvey 7:25 am on Dec 27, 2005 (gmt 0)
newpost thread iconHow to block robots spidering certain pages?
 
4zoltan 6:42 pm on Dec 23, 2005 (gmt 0)
newpost thread iconCorrect Syntax for Blogs in robots.txt
 
3vincevincevince 4:27 pm on Dec 22, 2005 (gmt 0)
Displaying Page 40 of 72<< Back 1 ... 30 31 32 33 34 35 36 37 38 39 [ 40 ] 41 42 43 44 45 46 47 48 49 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved