Welcome to WebmasterWorld Guest from 220.127.116.11
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
robot.txt, subdirectories, protected directories
|1||Ansuz||1:47 pm Feb 28, 2005|
|Robot terms defined
The differences and similarities between nofollow, noindex and robots.txt
|1||u_to_the_sixth||7:29 am Feb 26, 2005|
|Maximum size for robots.txt
Does any one know what it is?
|6||blend27||9:40 pm Feb 25, 2005|
|4||indodpendo||8:59 pm Feb 25, 2005|
|How can I get my whole site out of search engines and Google cache?
||2||mig1234||5:19 pm Feb 23, 2005|
|Banning Affiliate Links from being Indexed
||1||bears5122||11:57 pm Feb 22, 2005|
|Advanced robots.txt question
About blocking subdomain on different server.
|4||Buddha||12:57 am Feb 21, 2005|
|robots.txt wildcard matching
How wild is it?
|8||kwasher||12:51 am Feb 21, 2005|
|Creating A Robots.Txt File
How Do I Create A Correct robots.txt File?
|3||Jackal||4:27 pm Feb 14, 2005|
|ban all except index.htm
||2||rebelo||3:08 pm Feb 12, 2005|
|Is it possible to block SSI for certain spiders?
Is it possible to block SSI for certain spiders?
|1||mikeseo||3:19 pm Feb 10, 2005|
|Alexa (IA Archiver)
How do I excluse it
|2||Rollo||3:21 am Feb 10, 2005|
|Disallow index.html file
|1||yowza||7:26 pm Feb 9, 2005|
||6||textex||5:50 pm Feb 8, 2005|
|Please help me with robots.txt
||8||karantiq||1:18 pm Feb 8, 2005|
e-SocietyRobot new IP...
|2||beauzero||5:47 pm Feb 4, 2005|
<meta name="Robots" content="index,follow">
|4||Trojanfx||11:37 pm Feb 3, 2005|
|Can I Disallow Domain Hosts?
I want to disallow spiders from hosts, like foo.mydomain.com
|1||SoCalSam||7:24 pm Jan 30, 2005|
||4||energylevel||6:36 pm Jan 30, 2005|
|Help I'm trying to get my site on search engines such as google
Help on how to get your site on search engines such as google
|9||Mako3||8:32 am Jan 30, 2005|
|Correct way to exclude dynamic pages
question about differing methods of excluding dynamic pages
|8||heuristick||6:50 pm Jan 28, 2005|
|are the robots.txt examples still valid from WebmasterWorld?
||1||keywordguru||2:59 pm Jan 28, 2005|
|Suggestions On 'Low Value' Bots
Which bots provide little to no value to the web site.
|2||netcent||6:24 pm Jan 24, 2005|
|Blocking a file in the main folder?
||4||Ept103||3:33 am Jan 23, 2005|
|robot.txt needed to be crawled by google?
bot leaves after not finding one
|8||thegreatpretender||4:25 pm Jan 22, 2005|