Welcome to WebmasterWorld Guest from 54.163.115.193

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Robot.txt and Sitemap
3 sierra11b 1:26 am Oct 5, 2005
Newbie Robots.txt file Q
5 CoffeeMan 12:34 pm Oct 3, 2005
SygolBot, from sygol.com: anyone seen this before?
just visited my site, and did NOT respect robots.txt
3 stapel 12:37 am Oct 3, 2005
Can I get a page crawled daily?
disallow commands all I can find
5 innocbystr 11:33 pm Oct 2, 2005
Need to double check my Entry
Q - regarding ( / ) (Forward Slash) in Entry
2 Tutti 3:59 pm Oct 2, 2005
Trying to add robots.txt to invisionfree forum
can't get it to work
5 Damaris 1:34 am Sep 29, 2005
What does robots.txt help to do?
3 flashcination 10:40 pm Sep 28, 2005
How to block URL's of other sites link on my site?
How to block URL's of other sites link on my site?
3 sweetu 9:09 pm Sep 25, 2005
How to make a robots.txt file?
What are the steps?
2 steelcutman1975 8:37 pm Sep 23, 2005
Google "URL exclusion" not following our robots.txt!
robots.txt in subdirectory is not getting followed
9 joeduck 9:00 pm Sep 21, 2005
newbie question
About robots.txt
5 dicken288 7:37 am Sep 21, 2005
Keeping engines away from some files.
Keeping some image files away from the prying seahc engines....
3 ScoobyFlew 6:05 am Sep 20, 2005
Robot.txt
newbie question
2 playadonna 2:51 pm Sep 18, 2005
how long till search bot checks again?
2 editordude 11:47 am Sep 17, 2005
How to prevent hotlinking but allow one site using htaccess?
2 hersheyyu 11:38 pm Sep 15, 2005
Help with disallow
Help with disallow
4 SlugKing 10:21 pm Sep 14, 2005
Can Spiders see my dynamic "includes" pages?
3 zepper 11:57 pm Sep 13, 2005
picking up
5 stevelibby 7:04 am Sep 13, 2005
Need Help
Google spiders crawl 2 sites same content
1 milindnaik 6:54 am Sep 12, 2005
Allow search spider - error 503 socket.gaierror: (7, 'getaddrinfo fail
Allow search spider - error
5 shahindastur 5:39 am Sep 12, 2005
Quick Question
When something is disallowed, do bots ignore it completely?
5 jlander 2:43 am Sep 12, 2005
Am I correctly blocking diamondbot?
2 JAB_Creations 11:25 am Sep 11, 2005
Does excluding a cgi page automatically disable same page with paramet
paramters?
4 Clark 3:47 am Sep 11, 2005
Is it harmful if google indexes my /feed/ directory?
how to dissallow safely
3 brokenbricks 2:42 pm Sep 9, 2005
robots.txt wildcard
Will this work
7 gosman 1:59 pm Sep 6, 2005

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month