Welcome to WebmasterWorld Guest from 126.96.36.199
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|How to stop Yahoo Slurp?
Slurp ingnores my robots.txt
|16||atlrus||9:47 pm Apr 26, 2005|
|How to remove ezboard forum from Wayback?
||3||wmw99||12:04 pm Apr 26, 2005|
|Is there a definitive robots.txt to disallow bad bots
bad bots and bad bots only
|6||Clark||11:40 am Apr 24, 2005|
|Disallow keyword Ignored?
Who guarantees me that disallow keyword isn't ignored
|5||bkausbk||9:02 am Apr 24, 2005|
|Allowing and disallowing spiders
Easy one - can somebody confirm my understanding?
|3||Zeberdy||9:19 am Apr 21, 2005|
|How to disallow getlastpost
||2||annelisabeth||9:19 am Apr 21, 2005|
|Robots.txt Help request.
I need help with stopping spyder robots.
|10||westmore||6:22 am Apr 21, 2005|
How many can I have?
|8||Andromeda||8:07 pm Apr 20, 2005|
I need a confirmation for robots.txt from you experts
|11||RaniaD||11:51 am Apr 20, 2005|
|I want to disallow extra for Slurp
is this syntax right?
|2||baze22||12:55 pm Apr 19, 2005|
|Robots.txt vs Robots META tag
Does one over-rule the other?
|3||obsos||7:14 am Apr 18, 2005|
|Validation of robots.txt
||3||designei||10:50 am Apr 17, 2005|
|Help w/ robots.txt Google won't toch it
|2||JackieB||11:04 am Apr 16, 2005|
|What was wrong with this robots.txt if anything
||4||montenegro||9:25 am Apr 16, 2005|
|block cgi bin on a virtual host
cgi bin is not in root directory
|3||Reid||6:26 am Apr 16, 2005|
|Block specific words from being indexed
||13||Jer1024||7:34 pm Apr 15, 2005|
|Allowing robots to index secure directories in root
||13||naomi||2:46 pm Apr 14, 2005|
|Block certain links
||2||liviu||1:13 pm Apr 14, 2005|
Do I NEED robots.txt?
|3||TheRookie||11:47 pm Apr 12, 2005|
Where does it go?
|1||mark1||10:55 pm Apr 10, 2005|
is googlebot coming and going
|4||Steve_w||9:00 pm Apr 10, 2005|
Can this be done via robots.txt?
|2||yosmc||8:45 pm Apr 9, 2005|
|mod_rewrite and robots.txt
General discussion on use mod_rewrite with robots.txt
|7||Vaddy||4:34 pm Apr 7, 2005|
|New moderator for robots.txt forum
||4||engine||7:15 pm Apr 5, 2005|
|Subdomains and restricted access Web pages
Search Engine Optimization
|2||anshul||7:06 pm Apr 5, 2005|