homepage Welcome to WebmasterWorld Guest from 54.196.57.4
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Accredited PayPal World Seller

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconGoogle Hammering My site (1500+ hits already today)
  How to edit robots.txt to set delay
5wfernley 12:05 pm on Aug 31, 2005 (gmt 0)
newpost thread iconWhy do you need robots.txt?
 
5gordele 6:38 pm on Aug 30, 2005 (gmt 0)
newpost thread iconSpecifying pages for robots to crawl
  Worth it or not?
1sixdeep 6:14 pm on Aug 29, 2005 (gmt 0)
newpost thread iconIncluding specific robots necessary or not?
 
7silverbytes 8:27 am on Aug 29, 2005 (gmt 0)
newpost thread iconhow do i ban image robots
 
2spivei 11:04 pm on Aug 28, 2005 (gmt 0)
newpost thread iconbasic idea of robots.txt
 
8moi_britphile 10:38 am on Aug 28, 2005 (gmt 0)
newpost thread iconBanning AskJeeves Bot
  Need to ban askJeeves
2Marcus_Aurelius 8:31 pm on Aug 27, 2005 (gmt 0)
newpost thread iconproblem with robots.txt file
  problem with robots.txt file
5raybo75 5:36 pm on Aug 27, 2005 (gmt 0)
newpost thread iconHow Often Should a Spider Come
  What is the right amount of access a spider should have
4sirjeffer 11:00 pm on Aug 22, 2005 (gmt 0)
newpost thread iconBot cn exclude this directory?
 
2tintin74 10:42 pm on Aug 22, 2005 (gmt 0)
newpost thread icondisallow all : including host
 
4moua 3:57 pm on Aug 22, 2005 (gmt 0)
newpost thread iconGoogle Sitemap file name?
  What should I call my Google Sitemap?
2jetnovo 3:21 pm on Aug 22, 2005 (gmt 0)
newpost thread iconSpiders and redirects to disallowed directories
  Are the disallowed pages indexed?
2cigjonser 7:18 pm on Aug 19, 2005 (gmt 0)
newpost thread iconallow some and exclude the rest in robots.txt
 
9jcmiras 5:24 am on Aug 19, 2005 (gmt 0)
newpost thread iconcan you have 2 robots.txt files on site?
  one in main and one in subdomain
8stiksandstones 12:10 am on Aug 19, 2005 (gmt 0)
newpost thread iconWhat bots ignore robots.txt?
 
8Jon12345 10:56 pm on Aug 18, 2005 (gmt 0)
newpost thread iconValidator Questions
  Need Help Deciphering Validator Output
7Jioffri 6:07 am on Aug 18, 2005 (gmt 0)
newpost thread iconQuick question on robots.txt
  Can you separate the agent and the allows/disallows?
2cmendla 11:19 pm on Aug 17, 2005 (gmt 0)
newpost thread iconGoogle's Nonstandard robots.txt is Not Valid
  Googles robots.txt Does not validate
6forak 1:59 pm on Aug 17, 2005 (gmt 0)
newpost thread iconSearch engines picking up our backend?
  Developement site picked up
3financialhost 12:46 pm on Aug 11, 2005 (gmt 0)
newpost thread iconStill no PR for site
 
3gnuman 8:39 pm on Aug 10, 2005 (gmt 0)
newpost thread iconCan excluding with one word also exclude phrases?
 
3Marcia 10:47 pm on Aug 7, 2005 (gmt 0)
newpost thread iconNew algo about Robots.txt
  Is there any new algo introduced regarding robots.txt?
4Graham_Hans 2:31 am on Aug 7, 2005 (gmt 0)
newpost thread iconDmoz robots.txt
  Is the dmoz.org/world disallowed?
4angiolo 8:05 am on Aug 6, 2005 (gmt 0)
newpost thread iconHow to exclude Oracle Portal Help pages using robots.txt
  Need to exclude Oracle Portal Help Pages from search engines
10atul_iiit 9:12 am on Aug 5, 2005 (gmt 0)
Displaying Page 44 of 71<< Back 1 ... 34 35 36 37 38 39 40 41 42 43 [ 44 ] 45 46 47 48 49 50 51 52 53 ... 71 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved