Welcome to WebmasterWorld Guest from 22.214.171.124
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Can't find the robot.txt
where is it?
|27||mmmwowmmm||10:56 am May 21, 2005|
|SonicWALL Firewall Blocks Spiders
Settings will block spiders from indexing site.
|26||EliteWeb||3:16 pm May 15, 2005|
Will this work?
|4||kbba04527||6:37 am May 6, 2005|
|googlebot, robots.txt and spidering
Perplexed by lack or spidering of the site
|8||partnermine||5:12 am May 6, 2005|
|3||bonkph||8:16 pm May 5, 2005|
|Baiduspider - not checking robots.txt & how to block?
||6||TreeShare||8:07 am May 5, 2005|
|robots.txt and subdomain
Can I use this file to block a specific subdomain
|4||Pat1975||8:57 am May 4, 2005|
|google hits robots.txt but does not index anything?
am i doing something wrong?
|19||mbatta||5:38 am May 4, 2005|
|Can someone tell me how to ........
||3||Lex_Luther||9:00 pm May 3, 2005|
|Google - User Agent: * Disallow: /
||4||wchan07||4:42 pm May 3, 2005|
|mod_rewrite and robots.txt
||2||jewels||3:03 pm May 3, 2005|
|Blocking Yahoo from a specific folder
How do I do this?
|2||Rollo||2:55 pm May 3, 2005|
|How to Validate robots.txt and do I even have one?
||13||neophitemoneymaker||11:58 pm May 1, 2005|
|Page indexed in Google despite robots.txt entry
How did they find it in the first place?
|23||pmkpmk||9:17 pm May 1, 2005|
|The Great SearchEngineWorld robots.txt Survey #2 is Under Way
||9||Brett_Tabke||2:35 pm Apr 29, 2005|
|No-Follow, allow indexing of a file but not it's links?
||2||JAB_Creations||8:32 am Apr 27, 2005|
|How to stop Yahoo Slurp?
Slurp ingnores my robots.txt
|16||atlrus||9:47 pm Apr 26, 2005|
|How to remove ezboard forum from Wayback?
||3||wmw99||12:04 pm Apr 26, 2005|
|Is there a definitive robots.txt to disallow bad bots
bad bots and bad bots only
|6||Clark||11:40 am Apr 24, 2005|
|Disallow keyword Ignored?
Who guarantees me that disallow keyword isn't ignored
|5||bkausbk||9:02 am Apr 24, 2005|
|Allowing and disallowing spiders
Easy one - can somebody confirm my understanding?
|3||Zeberdy||9:19 am Apr 21, 2005|
|How to disallow getlastpost
||2||annelisabeth||9:19 am Apr 21, 2005|
|Robots.txt Help request.
I need help with stopping spyder robots.
|10||westmore||6:22 am Apr 21, 2005|
How many can I have?
|8||Andromeda||8:07 pm Apr 20, 2005|
I need a confirmation for robots.txt from you experts
|11||RaniaD||11:51 am Apr 20, 2005|