Welcome to WebmasterWorld Guest from 220.127.116.11
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|disallowing a folder with its own domain name
subdomain being spidered although it has its own domain name
|2||chandrika||8:52 pm Oct 26, 2005|
|robots.txt in a folder not root
will this work?
|3||fatpeter||6:27 am Oct 26, 2005|
|Robots.txt and Query Strings
Can they block non directory paths?
|3||soquinn||4:45 am Oct 26, 2005|
|Spider and registred access to web
||5||andreabg||8:07 am Oct 19, 2005|
|google accessing blocked image files
why does it appear that google is not abiding robots.txt?
|3||latimer||8:01 am Oct 19, 2005|
|Disallow all Directories
||2||Omega1Megs||8:48 pm Oct 14, 2005|
|Disallow: line in robots.txt
||2||stefan009||5:33 pm Oct 14, 2005|
|Can a robots.txt be too long?
||3||gingerspice||3:32 am Oct 13, 2005|
|Log file is robot.txt file?
Log file is robot.txt file?
|5||i_am_dhaval||9:20 pm Oct 12, 2005|
|Excluding domain in robots.txt
||10||triumph||7:25 pm Oct 11, 2005|
|Disallow dynamic content with mod_rewrite
Disallow dynamic content with mod_rewrite
|2||crglmb||3:54 pm Oct 10, 2005|
|Robots.txt in Ascii
|2||funnycloud||6:20 pm Oct 7, 2005|
|is there a reason to use a robots.txt
||5||thorthedestroyer||2:50 am Oct 7, 2005|
|Robot.txt and Sitemap
||3||sierra11b||1:26 am Oct 5, 2005|
|Newbie Robots.txt file Q
||5||CoffeeMan||12:34 pm Oct 3, 2005|
|SygolBot, from sygol.com: anyone seen this before?
just visited my site, and did NOT respect robots.txt
|3||stapel||12:37 am Oct 3, 2005|
|Can I get a page crawled daily?
disallow commands all I can find
|5||innocbystr||11:33 pm Oct 2, 2005|
|Need to double check my Entry
Q - regarding ( / ) (Forward Slash) in Entry
|2||Tutti||3:59 pm Oct 2, 2005|
|Trying to add robots.txt to invisionfree forum
can't get it to work
|5||Damaris||1:34 am Sep 29, 2005|
|What does robots.txt help to do?
||3||flashcination||10:40 pm Sep 28, 2005|
|How to block URL's of other sites link on my site?
How to block URL's of other sites link on my site?
|3||sweetu||9:09 pm Sep 25, 2005|
|How to make a robots.txt file?
What are the steps?
|2||steelcutman1975||8:37 pm Sep 23, 2005|
|Google "URL exclusion" not following our robots.txt!
robots.txt in subdirectory is not getting followed
|9||joeduck||9:00 pm Sep 21, 2005|
|5||dicken288||7:37 am Sep 21, 2005|
|Keeping engines away from some files.
Keeping some image files away from the prying seahc engines....
|3||ScoobyFlew||6:05 am Sep 20, 2005|