homepage Welcome to WebmasterWorld Guest from 54.196.159.11
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconHow do I prevent the SEs from crawling any links to my site w/HTTPS
  I don't want the crawlers to index anything w/ HTTPS protocol
3ZydoSEO 8:37 pm on Dec 9, 2007 (gmt 0)
newpost thread iconyahoo google msn etc.
  other searches
1ComputerHermit 5:56 pm on Dec 7, 2007 (gmt 0)
newpost thread iconProposal for robots.txt To Have Greater Flexibility
 
12engine 4:58 pm on Nov 30, 2007 (gmt 0)
newpost thread iconset crawl rate permanently to faster
 
1agushalim 5:39 pm on Nov 28, 2007 (gmt 0)
newpost thread iconWildcards on robots.txt disallows
 
11MindTwist 5:59 pm on Nov 27, 2007 (gmt 0)
newpost thread iconSite title and description missing in search results
  I had made some changes in robots.txt file
2ecmedia 2:12 pm on Nov 27, 2007 (gmt 0)
newpost thread iconperfect robot text files
  good robot text files
2likwidholdings 12:57 am on Nov 27, 2007 (gmt 0)
newpost thread iconGoogle showing site despite blocking in robots.txt
  The page is not cached though
3ecmedia 4:28 pm on Nov 20, 2007 (gmt 0)
newpost thread iconDisallow all except adsense
 
6thegreatpretender 4:40 pm on Nov 19, 2007 (gmt 0)
newpost thread iconHow to avoid double indexing
 
3mavi 4:43 pm on Nov 13, 2007 (gmt 0)
newpost thread iconRobots.txt Clarification
 
5smartcard 10:44 am on Nov 13, 2007 (gmt 0)
newpost thread iconRobot from colo is on behalf of Google?!
  Colo ISP claims robot crawling my site is from 3rd party Google contractor
5yodokame 4:03 am on Nov 12, 2007 (gmt 0)
newpost thread iconrobots.txt file
  how it affects the page rank of any website
2monikagaba 6:21 pm on Nov 1, 2007 (gmt 0)
newpost thread iconExclude ALL bots but major search engines
  Trying to make the perfect robots.txt
5serpmaster 12:03 pm on Oct 25, 2007 (gmt 0)
newpost thread iconblocking all except root (index.php)
  would disallow ALL , allow index work?
1bkode 7:56 am on Oct 24, 2007 (gmt 0)
newpost thread iconhttp code 200 mystery
  is the page served or not!?
3dailyclicker 5:14 pm on Oct 13, 2007 (gmt 0)
newpost thread iconDo robots get cookies?
  robots cookies possible dumb question
4Michel_Samuel 10:55 am on Oct 11, 2007 (gmt 0)
newpost thread iconmultiple disallows
  excluding multiple root directories.
5Michel_Samuel 10:05 pm on Oct 10, 2007 (gmt 0)
newpost thread iconMultiple robots.txt files in different directories -will it work?
  placing a robots.txt file in sub directories.
3Michel_Samuel 7:44 pm on Oct 9, 2007 (gmt 0)
newpost thread iconKeep physical hosting domain from being indexed?
 
5php4U 2:21 am on Oct 9, 2007 (gmt 0)
newpost thread iconHow do I not allow google
 
2pizzaguy 10:31 pm on Oct 3, 2007 (gmt 0)
newpost thread iconShould this file be "Public"
  Is it good practice to have the robots.txt file.......
4Propools 11:58 am on Sep 26, 2007 (gmt 0)
newpost thread iconProper URL & File Structure
  Fighting with Robots. Txt
3srkmodi 12:34 pm on Sep 24, 2007 (gmt 0)
newpost thread iconUse & or & amp; in robots.txt
  Which to use?
3g1smd 1:53 pm on Sep 21, 2007 (gmt 0)
newpost thread iconUsing allow to index search-only pages
 
3timmbbo 12:31 pm on Sep 18, 2007 (gmt 0)
Displaying Page 26 of 72<< Back 1 ... 16 17 18 19 20 21 22 23 24 25 [ 26 ] 27 28 29 30 31 32 33 34 35 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved