Welcome to WebmasterWorld Guest from 126.96.36.199
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Can I Disallow Domain Hosts?
I want to disallow spiders from hosts, like foo.mydomain.com
|1||SoCalSam||7:24 pm Jan 30, 2005|
||4||energylevel||6:36 pm Jan 30, 2005|
|Help I'm trying to get my site on search engines such as google
Help on how to get your site on search engines such as google
|9||Mako3||8:32 am Jan 30, 2005|
|Correct way to exclude dynamic pages
question about differing methods of excluding dynamic pages
|8||heuristick||6:50 pm Jan 28, 2005|
|are the robots.txt examples still valid from WebmasterWorld?
||1||keywordguru||2:59 pm Jan 28, 2005|
|Suggestions On 'Low Value' Bots
Which bots provide little to no value to the web site.
|2||netcent||6:24 pm Jan 24, 2005|
|Blocking a file in the main folder?
||4||Ept103||3:33 am Jan 23, 2005|
|robot.txt needed to be crawled by google?
bot leaves after not finding one
|8||thegreatpretender||4:25 pm Jan 22, 2005|
quick question to confirm format
|11||Reno||2:03 am Jan 20, 2005|
|Search Engines doesn't work with my robots.txt file
||2||nirpet||10:38 pm Jan 18, 2005|
|google follows links disallowed
Have I done something wrong?
|5||shady||4:54 pm Jan 18, 2005|
|Please clarify this for the last time
Subfolder, particualr bots exclusion
|1||damlag||10:12 pm Jan 11, 2005|
||3||seoerer||11:08 pm Jan 7, 2005|
should i put the results on my website?
|3||rj87uk||10:54 am Jan 6, 2005|
|Not sure how to handle space character
||5||Jon_King||1:48 am Jan 6, 2005|
|Problem of syntax and ordering
||1||esllou||11:50 am Jan 4, 2005|
|Will robots.txt affect IE or other browsers?
A little confuse...
|2||iProgram||4:34 am Jan 4, 2005|
|what are the names of the image bots?
I can't ban them if I don't know their names
|8||stapel||5:17 pm Jan 2, 2005|
|Blocking images folder from spiders
Good or bad idea?
|2||twist||10:17 pm Jan 1, 2005|
|Sudden Increase in robots.txt Hits
||3||Rob_Moore||10:04 pm Dec 26, 2004|
|Keeping spiders off co-branded sites
how to keep spiders from indexing our partner / co-branded sites
|4||BillingOrchard||9:41 pm Dec 23, 2004|
|About robots.txt file
Confusion in robots.txt file...i need hep..
|7||venuc||5:32 am Dec 21, 2004|
How do you tell the real googlebot from spoofers?
|3||teaperson||6:29 pm Dec 20, 2004|
|Tricky Robots Txt Help,
Two sites one robot
|3||badboybart||2:50 pm Dec 20, 2004|
|redirect without confusing spiders
||1||houseofsecrets||4:55 pm Dec 13, 2004|