Welcome to WebmasterWorld Guest from 54.226.130.194

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Google "URL exclusion" not following our robots.txt!
robots.txt in subdirectory is not getting followed
9 joeduck 9:00 pm Sep 21, 2005
newbie question
About robots.txt
5 dicken288 7:37 am Sep 21, 2005
Keeping engines away from some files.
Keeping some image files away from the prying seahc engines....
3 ScoobyFlew 6:05 am Sep 20, 2005
Robot.txt
newbie question
2 playadonna 2:51 pm Sep 18, 2005
how long till search bot checks again?
2 editordude 11:47 am Sep 17, 2005
How to prevent hotlinking but allow one site using htaccess?
2 hersheyyu 11:38 pm Sep 15, 2005
Help with disallow
Help with disallow
4 SlugKing 10:21 pm Sep 14, 2005
Can Spiders see my dynamic "includes" pages?
3 zepper 11:57 pm Sep 13, 2005
picking up
5 stevelibby 7:04 am Sep 13, 2005
Need Help
Google spiders crawl 2 sites same content
1 milindnaik 6:54 am Sep 12, 2005
Allow search spider - error 503 socket.gaierror: (7, 'getaddrinfo fail
Allow search spider - error
5 shahindastur 5:39 am Sep 12, 2005
Quick Question
When something is disallowed, do bots ignore it completely?
5 jlander 2:43 am Sep 12, 2005
Am I correctly blocking diamondbot?
2 JAB_Creations 11:25 am Sep 11, 2005
Does excluding a cgi page automatically disable same page with paramet
paramters?
4 Clark 3:47 am Sep 11, 2005
Is it harmful if google indexes my /feed/ directory?
how to dissallow safely
3 brokenbricks 2:42 pm Sep 9, 2005
robots.txt wildcard
Will this work
7 gosman 1:59 pm Sep 6, 2005
Robots.txt and frontpage
14 sunflower12 1:32 pm Sep 4, 2005
The codes required to call the robots
2 benifer 10:04 pm Aug 31, 2005
Google Hammering My site (1500+ hits already today)
How to edit robots.txt to set delay
5 wfernley 12:05 pm Aug 31, 2005
Why do you need robots.txt?
5 gordele 6:38 pm Aug 30, 2005
Specifying pages for robots to crawl
Worth it or not?
1 sixdeep 6:14 pm Aug 29, 2005
Including specific robots necessary or not?
7 silverbytes 8:27 am Aug 29, 2005
how do i ban image robots
2 spivei 11:04 pm Aug 28, 2005
basic idea of robots.txt
8 moi_britphile 10:38 am Aug 28, 2005
Banning AskJeeves Bot
Need to ban askJeeves
2 Marcus_Aurelius 8:31 pm Aug 27, 2005