Welcome to WebmasterWorld Guest from 54.224.143.233

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Robots.txt File to Block from Dynamic URLs
Trying to eliminate duplicate content...
3 BillyS 2:07 am Oct 28, 2005
disallowing a folder with its own domain name
subdomain being spidered although it has its own domain name
2 chandrika 8:52 pm Oct 26, 2005
robots.txt in a folder not root
will this work?
3 fatpeter 6:27 am Oct 26, 2005
Robots.txt and Query Strings
Can they block non directory paths?
3 soquinn 4:45 am Oct 26, 2005
Spider and registred access to web
5 andreabg 8:07 am Oct 19, 2005
google accessing blocked image files
why does it appear that google is not abiding robots.txt?
3 latimer 8:01 am Oct 19, 2005
Disallow all Directories
2 Omega1Megs 8:48 pm Oct 14, 2005
Disallow: line in robots.txt
2 stefan009 5:33 pm Oct 14, 2005
Can a robots.txt be too long?
3 gingerspice 3:32 am Oct 13, 2005
Log file is robot.txt file?
Log file is robot.txt file?
5 i_am_dhaval 9:20 pm Oct 12, 2005
Excluding domain in robots.txt
10 triumph 7:25 pm Oct 11, 2005
Disallow dynamic content with mod_rewrite
Disallow dynamic content with mod_rewrite
2 crglmb 3:54 pm Oct 10, 2005
Robots.txt in Ascii
How-to?
2 funnycloud 6:20 pm Oct 7, 2005
is there a reason to use a robots.txt
5 thorthedestroyer 2:50 am Oct 7, 2005
Robot.txt and Sitemap
3 sierra11b 1:26 am Oct 5, 2005
Newbie Robots.txt file Q
5 CoffeeMan 12:34 pm Oct 3, 2005
SygolBot, from sygol.com: anyone seen this before?
just visited my site, and did NOT respect robots.txt
3 stapel 12:37 am Oct 3, 2005
Can I get a page crawled daily?
disallow commands all I can find
5 innocbystr 11:33 pm Oct 2, 2005
Need to double check my Entry
Q - regarding ( / ) (Forward Slash) in Entry
2 Tutti 3:59 pm Oct 2, 2005
Trying to add robots.txt to invisionfree forum
can't get it to work
5 Damaris 1:34 am Sep 29, 2005
What does robots.txt help to do?
3 flashcination 10:40 pm Sep 28, 2005
How to block URL's of other sites link on my site?
How to block URL's of other sites link on my site?
3 sweetu 9:09 pm Sep 25, 2005
How to make a robots.txt file?
What are the steps?
2 steelcutman1975 8:37 pm Sep 23, 2005
Google "URL exclusion" not following our robots.txt!
robots.txt in subdirectory is not getting followed
9 joeduck 9:00 pm Sep 21, 2005
newbie question
About robots.txt
5 dicken288 7:37 am Sep 21, 2005