homepage Welcome to WebmasterWorld Guest from 54.163.70.249
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt

Sitemaps, Meta Data, and robots.txt

Forum Charter, Library, Moderated by: goodroi Forum Options: Reset Last Read Pointers

Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
new Subject Messages Started by Last Message
newpost thread iconpicking up
 
5stevelibby 7:04 am on Sep 13, 2005 (gmt 0)
newpost thread iconNeed Help
  Google spiders crawl 2 sites same content
1milindnaik 6:54 am on Sep 12, 2005 (gmt 0)
newpost thread iconAllow search spider - error 503 socket.gaierror: (7, 'getaddrinfo fail
  Allow search spider - error
5shahindastur 5:39 am on Sep 12, 2005 (gmt 0)
newpost thread iconQuick Question
  When something is disallowed, do bots ignore it completely?
5jlander 2:43 am on Sep 12, 2005 (gmt 0)
newpost thread iconAm I correctly blocking diamondbot?
 
2JAB_Creations 11:25 am on Sep 11, 2005 (gmt 0)
newpost thread iconDoes excluding a cgi page automatically disable same page with paramet
  paramters?
4Clark 3:47 am on Sep 11, 2005 (gmt 0)
newpost thread iconIs it harmful if google indexes my /feed/ directory?
  how to dissallow safely
3brokenbricks 2:42 pm on Sep 9, 2005 (gmt 0)
newpost thread iconrobots.txt wildcard
  Will this work
7gosman 1:59 pm on Sep 6, 2005 (gmt 0)
newpost thread iconRobots.txt and frontpage
 
14sunflower12 1:32 pm on Sep 4, 2005 (gmt 0)
newpost thread iconThe codes required to call the robots
 
2benifer 10:04 pm on Aug 31, 2005 (gmt 0)
newpost thread iconGoogle Hammering My site (1500+ hits already today)
  How to edit robots.txt to set delay
5wfernley 12:05 pm on Aug 31, 2005 (gmt 0)
newpost thread iconWhy do you need robots.txt?
 
5gordele 6:38 pm on Aug 30, 2005 (gmt 0)
newpost thread iconSpecifying pages for robots to crawl
  Worth it or not?
1sixdeep 6:14 pm on Aug 29, 2005 (gmt 0)
newpost thread iconIncluding specific robots necessary or not?
 
7silverbytes 8:27 am on Aug 29, 2005 (gmt 0)
newpost thread iconhow do i ban image robots
 
2spivei 11:04 pm on Aug 28, 2005 (gmt 0)
newpost thread iconbasic idea of robots.txt
 
8moi_britphile 10:38 am on Aug 28, 2005 (gmt 0)
newpost thread iconBanning AskJeeves Bot
  Need to ban askJeeves
2Marcus_Aurelius 8:31 pm on Aug 27, 2005 (gmt 0)
newpost thread iconproblem with robots.txt file
  problem with robots.txt file
5raybo75 5:36 pm on Aug 27, 2005 (gmt 0)
newpost thread iconHow Often Should a Spider Come
  What is the right amount of access a spider should have
4sirjeffer 11:00 pm on Aug 22, 2005 (gmt 0)
newpost thread iconBot cn exclude this directory?
 
2tintin74 10:42 pm on Aug 22, 2005 (gmt 0)
newpost thread icondisallow all : including host
 
4moua 3:57 pm on Aug 22, 2005 (gmt 0)
newpost thread iconGoogle Sitemap file name?
  What should I call my Google Sitemap?
2jetnovo 3:21 pm on Aug 22, 2005 (gmt 0)
newpost thread iconSpiders and redirects to disallowed directories
  Are the disallowed pages indexed?
2cigjonser 7:18 pm on Aug 19, 2005 (gmt 0)
newpost thread iconallow some and exclude the rest in robots.txt
 
9jcmiras 5:24 am on Aug 19, 2005 (gmt 0)
newpost thread iconcan you have 2 robots.txt files on site?
  one in main and one in subdomain
8stiksandstones 12:10 am on Aug 19, 2005 (gmt 0)
Displaying Page 44 of 72<< Back 1 ... 34 35 36 37 38 39 40 41 42 43 [ 44 ] 45 46 47 48 49 50 51 52 53 ... 72 Next >>
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved