homepage Welcome to WebmasterWorld Guest from 54.163.190.170
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconhow to disallow all robots except Googlebot, Yahoo Slurp and MSNBot?
 
8d6rth7ader 5:48 am on May 2, 2007 (gmt 0)
newpost thread iconHtaccess
 
3harish_s 2:17 pm on May 1, 2007 (gmt 0)
newpost thread iconLimiting the depth of Google's crawl?
  Can robots.txt be used to limit depth
6daveozzz 7:21 pm on Apr 27, 2007 (gmt 0)
newpost thread iconDisallow: /
  How did googlebot index my web server dir structure?
3thinkweb 5:24 am on Apr 26, 2007 (gmt 0)
newpost thread iconURLs restricted by robots.txt (41) AGH!
  I'm a bit stumped about what's best to do re my robots.txt
8wrafter 8:38 pm on Apr 25, 2007 (gmt 0)
newpost thread iconIs a robots.txt file necessary?
  if not trying to block bots?
3Jakes_Redding 5:52 am on Apr 20, 2007 (gmt 0)
newpost thread iconGooglebot ignoring robots.txt?
  Or is my robots.txt file wrong...
3designaweb 9:28 am on Apr 17, 2007 (gmt 0)
newpost thread iconBlocking Google Only
  From certain sub domains
5Moncao 7:25 pm on Apr 13, 2007 (gmt 0)
newpost thread iconunknown robot
  robot
4kool002 1:59 pm on Apr 11, 2007 (gmt 0)
newpost thread iconNot being found by the search engines
  noindex,nofollow
4gtate 6:24 pm on Apr 6, 2007 (gmt 0)
newpost thread iconEfficient way to block multiple bots?
 
11JAB_Creations 2:33 pm on Apr 3, 2007 (gmt 0)
newpost thread iconBlocking Google
  I need to block google from my index page
4loner 4:25 pm on Apr 2, 2007 (gmt 0)
newpost thread iconhelp needed
 
1hugh 4:03 pm on Apr 2, 2007 (gmt 0)
newpost thread iconWhat's the Proper Way to Block Sub Folders
  Looking for the best way to block 2nd level sub folders
5graywolf 7:43 pm on Mar 28, 2007 (gmt 0)
newpost thread iconROBOT and PR rank?
  how does the robot effect your pr rank?
7moppy56 1:48 pm on Mar 28, 2007 (gmt 0)
newpost thread iconBlocking Partial Contents of Folder Without Using Robots Tag
  10 files (7 want indexed, 3 want blocked)
5SandySEO 8:49 pm on Mar 27, 2007 (gmt 0)
newpost thread iconAnother bad bot to ban
  this one is using 302 redirects .. bad bad
1Bewenched 3:43 am on Mar 25, 2007 (gmt 0)
newpost thread iconexcluding a subdomain
  Using the robots.txt file to exclude a subdomain
10joebray 9:47 am on Mar 23, 2007 (gmt 0)
newpost thread iconG. ignores robots.txt at other ports, gives false page count
  Google shows in its index pages that it never saw, even prohibited
1OnetSzukaj 2:58 pm on Mar 22, 2007 (gmt 0)
newpost thread iconHow to use Robots.txt as an INCLUSION method
  It's exclusion only if you use it that way
1incrediBILL 12:32 am on Mar 22, 2007 (gmt 0)
newpost thread iconTripAdvisorBot
  Anyone seen this?
3bobbarnes 10:28 pm on Mar 16, 2007 (gmt 0)
newpost thread iconproblem with msnbot and wildcard matching
 
2Achernar 1:36 pm on Mar 16, 2007 (gmt 0)
newpost thread iconHow Can I Make Bots Visit My Web Site?
 
3emadido 2:34 pm on Mar 15, 2007 (gmt 0)
newpost thread iconwhy i can't reaply the topic
  why i can't reaply the topic
3unknown 8:43 am on Mar 14, 2007 (gmt 0)
newpost thread iconrobotsxx.txt not found.
 
2benevolent001 1:16 pm on Mar 13, 2007 (gmt 0)
Displaying Page 30 of 72<< Back 1 ... 20 21 22 23 24 25 26 27 28 29 [ 30 ] 31 32 33 34 35 36 37 38 39 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved