Welcome to WebmasterWorld Guest from 188.8.131.52
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|disallowing subdomain access rule
||1||coburn||1:37 pm Feb 7, 2007|
|can put two robots at one line at robot.txt to ban two spiders?
robots.txt, ban google, yahoo bots
|4||johnlim9988||3:54 am Feb 7, 2007|
|Blocking subdir, but allowing crawl in deeper dir?
||7||Vulcan315||3:52 pm Feb 6, 2007|
|robots noindex on an iframe page
What would this do?
|2||dataguy||1:20 pm Feb 3, 2007|
|Slowing down MSN bot?
Doubling it's bandwidth usage each month.
|4||bouncybunny||3:33 am Feb 3, 2007|
|Slurps and wildcards
||5||Vimes||1:36 am Feb 3, 2007|
|robots.txt blocking my index page?
|4||hola2||2:08 pm Feb 2, 2007|
|Robots.txt command? "do not search anything outside this."
||5||OWizeOne||1:08 pm Feb 1, 2007|
|How do I block yahoo's SE spider and allow their image spider?
Does Yahoo send a different spider for the image search?
|4||Clark||5:18 pm Jan 27, 2007|
|How robot.txt works
|3||walmslei||4:10 am Jan 27, 2007|
|Why is google indexing this banned directory?
||5||ichthyous||4:55 pm Jan 26, 2007|
|https on google and robots.txt
||2||concepts99||1:54 pm Jan 24, 2007|
|directory structure question
||5||union_jack||7:43 pm Jan 22, 2007|
|block a subdirectory
||2||concepts99||2:05 pm Jan 22, 2007|
|Google not respecting robots.txt
Found >500 pages in Google index that are forbiden by robots.txt
|5||roots||5:54 am Jan 22, 2007|
|3||ricnut||11:52 pm Jan 20, 2007|
|Disallow Slurp from all files except two
Yahoo likes only one page, so I'd like to just 'give' it that.
|2||tigertom||3:00 am Jan 20, 2007|
|Yahoo Slurp Bot misbehaving
.. or is it someone masquerading?
|4||tigertom||9:27 am Jan 17, 2007|
|Blocking Site in some countries
||3||eldinboy||5:55 pm Jan 15, 2007|
|What to do with duplicate pages
||3||olba||2:12 am Jan 12, 2007|
|Why should I have a robots.txt file?
Isn't it OK if I don't?
|23||beren||8:45 am Jan 11, 2007|
|Google Site Diagnostics - robots.txt blocks urls
||4||akreider||7:44 pm Jan 5, 2007|
|Robots txt to solve gallery duplicates
can someone confirm this is correct
|3||caines||10:54 am Jan 5, 2007|
|How many search engine understand crawl delay?
||1||foxfox||3:38 pm Jan 4, 2007|
|Multiple Domains Issue
||3||kdobson99||4:58 am Jan 4, 2007|