homepage Welcome to WebmasterWorld Guest from 54.205.193.39
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconWhy are my .swf files still being indexed by Googlebot?
 
3ichthyous 6:26 pm on Jul 13, 2004 (gmt 0)
newpost thread iconexcluding dynamic pages in all search engines
  excluding dynamic pages in robots.txt
3toddmc04 2:49 pm on Jul 13, 2004 (gmt 0)
newpost thread iconGeneral Guide to Robots.txt
  is there one here :)
4rj87uk 7:51 pm on Jul 7, 2004 (gmt 0)
newpost thread icondomain.com vs www.domain.com
  different sites
9HitProf 4:01 pm on Jul 7, 2004 (gmt 0)
newpost thread iconDo i need the txt file?
  if i want to allow everyone
4markbiz 12:59 am on Jul 5, 2004 (gmt 0)
newpost thread iconI'm not understanding the different makes of bots
  Can I stop some, while allowing others?
2tomequin 10:11 pm on Jul 1, 2004 (gmt 0)
newpost thread iconYahoo! Slurp - robots.txt
  Not obeyng robots.txt
5EarWig 1:59 pm on Jun 29, 2004 (gmt 0)
newpost thread iconMaking robots?
 
4lifedeuce 1:22 pm on Jun 29, 2004 (gmt 0)
newpost thread iconDisallow pages in a subdirectory
  Basic help please
3pjuk 8:40 pm on Jun 24, 2004 (gmt 0)
newpost thread iconsite setup for spider?
 
5Gemelli_no1 12:51 pm on Jun 23, 2004 (gmt 0)
newpost thread iconneeded?
  is it completely neccisary
3natty 1:34 pm on Jun 15, 2004 (gmt 0)
newpost thread iconRobots dot text for all user files?
  robots.txt all public_html
4kj6loh 7:21 pm on Jun 11, 2004 (gmt 0)
newpost thread iconwildcards in the Disallow field,
  non standard?
3pmac 4:19 pm on Jun 11, 2004 (gmt 0)
newpost thread iconwhere to place robots.txt?
  robots newbie alert
6Duke_of_Url 10:17 pm on Jun 10, 2004 (gmt 0)
newpost thread iconHow should I use a robots.txt file on a virtual account?
  I want to ban spiders from one site only
3soccer_star 2:01 pm on Jun 10, 2004 (gmt 0)
newpost thread iconRobots.txt for yahoo and google.com
  google and yahoo robots.txt
3adisakmai 4:47 am on Jun 10, 2004 (gmt 0)
newpost thread iconPrevent Google's "cached" link from appearing
  but still stay in the search results
4digitalv 4:51 pm on Jun 8, 2004 (gmt 0)
newpost thread iconDisallowing specific query strings
 
3duke33 3:05 pm on Jun 8, 2004 (gmt 0)
newpost thread iconIs it a spider?
 
2topher 5:38 am on Jun 8, 2004 (gmt 0)
newpost thread iconPreventing robot access to all subdirectories within a directory
 
2Leonardi 12:16 am on Jun 7, 2004 (gmt 0)
newpost thread iconRobots.txt triggers 404 error
  Robots.txt triggers 404 error
2Nout 11:10 am on May 29, 2004 (gmt 0)
newpost thread iconIs this robots.txt ok?
 
5esllou 12:32 pm on May 26, 2004 (gmt 0)
newpost thread iconRemoving "query-stringed" pages from Google
  And other search engines using robotx.txt
3MatthewHSE 2:23 pm on May 25, 2004 (gmt 0)
newpost thread iconStop Spiders Crawling Message Board Edit
  Cgi Factory Message Board
7mn1dbp 12:36 pm on May 25, 2004 (gmt 0)
newpost thread iconGoogle still lists disallowed pages
  How long do disallowed pages stay in the index?
5jam13 10:56 am on May 23, 2004 (gmt 0)
Displaying Page 56 of 71<< Back 1 ... 46 47 48 49 50 51 52 53 54 55 [ 56 ] 57 58 59 60 61 62 63 64 65 ... 71 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved