Welcome to WebmasterWorld Guest from 126.96.36.199
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Need help making robots to block all spiders
||9||budbiss||7:47 pm Feb 16, 2004|
|Selective bot blocking
|2||fabfurs||11:43 am Feb 16, 2004|
|Do I need to
||2||ronhollin||6:46 am Feb 16, 2004|
|How important is having a properly configured robots?
||4||budbiss||3:45 pm Feb 14, 2004|
|DOS Line Enders
How to download sample robots.txt files
|3||webman2||10:30 am Feb 14, 2004|
|robots.txt and template pages
Does blocking templates prevent spiders from following links?
|2||butterfi||12:59 am Feb 12, 2004|
|disallow is allowing
Is this done correctly?
|1||DavidAtWork||6:25 pm Feb 11, 2004|
|I don't have a robots.txt file..error 404
Can't get good Google or Yahoo placement
|6||jagbitsllc||2:17 am Feb 11, 2004|
what is this?
|2||Trisha||10:33 pm Feb 10, 2004|
|How to stop robots on .php files
||3||djtaverner||6:40 pm Feb 10, 2004|
|What in the world is a robot.txt?
I need this to be stated simple for me
|5||Claudia_T||6:35 pm Feb 10, 2004|
|Help! I need to disallow a specific query string
||1||Hissingsid||11:26 am Feb 10, 2004|
||2||mikemcs||4:25 am Feb 10, 2004|
|Googlebot chasing .php tracking script
is robots.txt enough?
|2||Edouard_H||12:51 am Feb 9, 2004|
|301 redirects & robots.txt
What is the implication of a 301 redirect on robots.txt
|2||Jerry_Pannu||10:17 pm Feb 7, 2004|
Why won't my site get spidered
|2||DavidAtWork||5:56 pm Feb 5, 2004|
|I've been looking at the WW robots.txt ...
...but why are these spiders disallowed?
|7||elgumbo||1:32 pm Feb 4, 2004|
Will using robots exclusion and meta exclusion hurt me?
|1||mercury||9:12 pm Feb 3, 2004|
|Spiders that ignore or skip robots.txt
Can bad spiders be identified?
|4||DavidAtWork||2:05 pm Feb 3, 2004|
|Robots Meta and .txt
|4||admin_amoroustoys||1:34 pm Feb 3, 2004|
|robots.txt used in court case
||2||transactiongeek||1:27 pm Feb 3, 2004|
|what is the "KAIST AITrc Crawler" bot?
it ignored my "robots.txt" file and got caught in my bot trap
|4||stapel||12:01 pm Feb 3, 2004|
|Is it illegal to ignore a robots.txt?
Could I sue google?
|2||transactiongeek||6:44 am Feb 3, 2004|
I have noindex,follow but the page is indexed
|2||nevetS||4:41 am Feb 3, 2004|
|save and upload robots.txt
||4||helenp||8:27 pm Feb 2, 2004|