Welcome to WebmasterWorld Guest from 188.8.131.52
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
from Competitors Analysis Tools...
|4||ambiorix_rb||1:21 pm Jun 1, 2005|
|SEO / robots.txt question
unintentional cross-linking penalty
|1||wolfgang||6:27 pm May 31, 2005|
|Blocking a Sub Domain Directory
||3||is300||8:25 am May 30, 2005|
|Order allow,deny or deny,allow
Not sure which order is best...
|3||whoisgregg||9:16 pm May 27, 2005|
|Search Engine Crawlers
||2||dayo||8:39 pm May 26, 2005|
|Spidering of my code pages
Wrong pages being spidered and losing referals
|5||djfelip||11:18 pm May 25, 2005|
|robots4.txt -- I'm confused
Confused about what is allowed in this file
|3||Goose_68||8:46 pm May 24, 2005|
|2||webmz||3:10 am May 22, 2005|
||18||Dexie||4:14 pm May 21, 2005|
|Can't find the robot.txt
where is it?
|27||mmmwowmmm||10:56 am May 21, 2005|
|SonicWALL Firewall Blocks Spiders
Settings will block spiders from indexing site.
|26||EliteWeb||3:16 pm May 15, 2005|
Will this work?
|4||kbba04527||6:37 am May 6, 2005|
|googlebot, robots.txt and spidering
Perplexed by lack or spidering of the site
|8||partnermine||5:12 am May 6, 2005|
|3||bonkph||8:16 pm May 5, 2005|
|Baiduspider - not checking robots.txt & how to block?
||6||TreeShare||8:07 am May 5, 2005|
|robots.txt and subdomain
Can I use this file to block a specific subdomain
|4||Pat1975||8:57 am May 4, 2005|
|google hits robots.txt but does not index anything?
am i doing something wrong?
|19||mbatta||5:38 am May 4, 2005|
|Can someone tell me how to ........
||3||Lex_Luther||9:00 pm May 3, 2005|
|Google - User Agent: * Disallow: /
||4||wchan07||4:42 pm May 3, 2005|
|mod_rewrite and robots.txt
||2||jewels||3:03 pm May 3, 2005|
|Blocking Yahoo from a specific folder
How do I do this?
|2||Rollo||2:55 pm May 3, 2005|
|How to Validate robots.txt and do I even have one?
||13||neophitemoneymaker||11:58 pm May 1, 2005|
|Page indexed in Google despite robots.txt entry
How did they find it in the first place?
|23||pmkpmk||9:17 pm May 1, 2005|
|The Great SearchEngineWorld robots.txt Survey #2 is Under Way
||9||Brett_Tabke||2:35 pm Apr 29, 2005|
|No-Follow, allow indexing of a file but not it's links?
||2||JAB_Creations||8:32 am Apr 27, 2005|