homepage Welcome to WebmasterWorld Guest from 54.227.20.250
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconHow to prevent hotlinking but allow one site using htaccess?
 
2hersheyyu 11:38 pm on Sep 15, 2005 (gmt 0)
newpost thread iconHelp with disallow
  Help with disallow
4SlugKing 10:21 pm on Sep 14, 2005 (gmt 0)
newpost thread iconCan Spiders see my dynamic "includes" pages?
 
3zepper 11:57 pm on Sep 13, 2005 (gmt 0)
newpost thread iconpicking up
 
5stevelibby 7:04 am on Sep 13, 2005 (gmt 0)
newpost thread iconNeed Help
  Google spiders crawl 2 sites same content
1milindnaik 6:54 am on Sep 12, 2005 (gmt 0)
newpost thread iconAllow search spider - error 503 socket.gaierror: (7, 'getaddrinfo fail
  Allow search spider - error
5shahindastur 5:39 am on Sep 12, 2005 (gmt 0)
newpost thread iconQuick Question
  When something is disallowed, do bots ignore it completely?
5jlander 2:43 am on Sep 12, 2005 (gmt 0)
newpost thread iconAm I correctly blocking diamondbot?
 
2JAB_Creations 11:25 am on Sep 11, 2005 (gmt 0)
newpost thread iconDoes excluding a cgi page automatically disable same page with paramet
  paramters?
4Clark 3:47 am on Sep 11, 2005 (gmt 0)
newpost thread iconIs it harmful if google indexes my /feed/ directory?
  how to dissallow safely
3brokenbricks 2:42 pm on Sep 9, 2005 (gmt 0)
newpost thread iconrobots.txt wildcard
  Will this work
7gosman 1:59 pm on Sep 6, 2005 (gmt 0)
newpost thread iconRobots.txt and frontpage
 
14sunflower12 1:32 pm on Sep 4, 2005 (gmt 0)
newpost thread iconThe codes required to call the robots
 
2benifer 10:04 pm on Aug 31, 2005 (gmt 0)
newpost thread iconGoogle Hammering My site (1500+ hits already today)
  How to edit robots.txt to set delay
5wfernley 12:05 pm on Aug 31, 2005 (gmt 0)
newpost thread iconWhy do you need robots.txt?
 
5gordele 6:38 pm on Aug 30, 2005 (gmt 0)
newpost thread iconSpecifying pages for robots to crawl
  Worth it or not?
1sixdeep 6:14 pm on Aug 29, 2005 (gmt 0)
newpost thread iconIncluding specific robots necessary or not?
 
7silverbytes 8:27 am on Aug 29, 2005 (gmt 0)
newpost thread iconhow do i ban image robots
 
2spivei 11:04 pm on Aug 28, 2005 (gmt 0)
newpost thread iconbasic idea of robots.txt
 
8moi_britphile 10:38 am on Aug 28, 2005 (gmt 0)
newpost thread iconBanning AskJeeves Bot
  Need to ban askJeeves
2Marcus_Aurelius 8:31 pm on Aug 27, 2005 (gmt 0)
newpost thread iconproblem with robots.txt file
  problem with robots.txt file
5raybo75 5:36 pm on Aug 27, 2005 (gmt 0)
newpost thread iconHow Often Should a Spider Come
  What is the right amount of access a spider should have
4sirjeffer 11:00 pm on Aug 22, 2005 (gmt 0)
newpost thread iconBot cn exclude this directory?
 
2tintin74 10:42 pm on Aug 22, 2005 (gmt 0)
newpost thread icondisallow all : including host
 
4moua 3:57 pm on Aug 22, 2005 (gmt 0)
newpost thread iconGoogle Sitemap file name?
  What should I call my Google Sitemap?
2jetnovo 3:21 pm on Aug 22, 2005 (gmt 0)
Displaying Page 44 of 72<< Back 1 ... 34 35 36 37 38 39 40 41 42 43 [ 44 ] 45 46 47 48 49 50 51 52 53 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved