homepage Welcome to WebmasterWorld Guest from 54.211.201.65
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconHow do I disallow bots from my Wordpress year Archive Folders
  I guess this is a common question but I can't find an answer
4Drum 5:26 pm on May 5, 2007 (gmt 0)
newpost thread iconAdd Sitemap To robots.txt For Autodiscovery[2] ( 1 2 )
  Ask.com, Google, Microsoft Live Search and Yahoo!
32OldWolf 7:56 am on May 4, 2007 (gmt 0)
newpost thread iconHow is alexa getting past my stoppers
  stop alexa index
1Michel_Samuel 3:31 pm on May 2, 2007 (gmt 0)
newpost thread iconhow to disallow all robots except Googlebot, Yahoo Slurp and MSNBot?
 
8d6rth7ader 5:48 am on May 2, 2007 (gmt 0)
newpost thread iconHtaccess
 
3harish_s 2:17 pm on May 1, 2007 (gmt 0)
newpost thread iconLimiting the depth of Google's crawl?
  Can robots.txt be used to limit depth
6daveozzz 7:21 pm on Apr 27, 2007 (gmt 0)
newpost thread iconDisallow: /
  How did googlebot index my web server dir structure?
3thinkweb 5:24 am on Apr 26, 2007 (gmt 0)
newpost thread iconURLs restricted by robots.txt (41) AGH!
  I'm a bit stumped about what's best to do re my robots.txt
8wrafter 8:38 pm on Apr 25, 2007 (gmt 0)
newpost thread iconIs a robots.txt file necessary?
  if not trying to block bots?
3Jakes_Redding 5:52 am on Apr 20, 2007 (gmt 0)
newpost thread iconGooglebot ignoring robots.txt?
  Or is my robots.txt file wrong...
3designaweb 9:28 am on Apr 17, 2007 (gmt 0)
newpost thread iconBlocking Google Only
  From certain sub domains
5Moncao 7:25 pm on Apr 13, 2007 (gmt 0)
newpost thread iconunknown robot
  robot
4kool002 1:59 pm on Apr 11, 2007 (gmt 0)
newpost thread iconNot being found by the search engines
  noindex,nofollow
4gtate 6:24 pm on Apr 6, 2007 (gmt 0)
newpost thread iconEfficient way to block multiple bots?
 
11JAB_Creations 2:33 pm on Apr 3, 2007 (gmt 0)
newpost thread iconBlocking Google
  I need to block google from my index page
4loner 4:25 pm on Apr 2, 2007 (gmt 0)
newpost thread iconhelp needed
 
1hugh 4:03 pm on Apr 2, 2007 (gmt 0)
newpost thread iconWhat's the Proper Way to Block Sub Folders
  Looking for the best way to block 2nd level sub folders
5graywolf 7:43 pm on Mar 28, 2007 (gmt 0)
newpost thread iconROBOT and PR rank?
  how does the robot effect your pr rank?
7moppy56 1:48 pm on Mar 28, 2007 (gmt 0)
newpost thread iconBlocking Partial Contents of Folder Without Using Robots Tag
  10 files (7 want indexed, 3 want blocked)
5SandySEO 8:49 pm on Mar 27, 2007 (gmt 0)
newpost thread iconAnother bad bot to ban
  this one is using 302 redirects .. bad bad
1Bewenched 3:43 am on Mar 25, 2007 (gmt 0)
newpost thread iconexcluding a subdomain
  Using the robots.txt file to exclude a subdomain
10joebray 9:47 am on Mar 23, 2007 (gmt 0)
newpost thread iconG. ignores robots.txt at other ports, gives false page count
  Google shows in its index pages that it never saw, even prohibited
1OnetSzukaj 2:58 pm on Mar 22, 2007 (gmt 0)
newpost thread iconHow to use Robots.txt as an INCLUSION method
  It's exclusion only if you use it that way
1incrediBILL 12:32 am on Mar 22, 2007 (gmt 0)
newpost thread iconTripAdvisorBot
  Anyone seen this?
3bobbarnes 10:28 pm on Mar 16, 2007 (gmt 0)
newpost thread iconproblem with msnbot and wildcard matching
 
2Achernar 1:36 pm on Mar 16, 2007 (gmt 0)
Displaying Page 29 of 71<< Back 1 ... 19 20 21 22 23 24 25 26 27 28 [ 29 ] 30 31 32 33 34 35 36 37 38 ... 71 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved