Welcome to WebmasterWorld Guest from 184.108.40.206
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Creating robots.txt file.
||6||kabanero||8:34 pm Aug 18, 2004|
|Blocking just one parameter via robots.txt
Is this even possible?
|2||werty||7:31 pm Aug 18, 2004|
|Problem with spiderbots scrounging for robot.txt
||3||Mr_N||6:49 am Aug 10, 2004|
|i have a subdomain I dont want indexed
is robots.txt the answer?
|4||chrisandsarah||1:21 pm Aug 8, 2004|
|robots.txt appearing in my log files
But I have n't got one there!
|5||uk_webber||2:59 pm Aug 5, 2004|
|To Robots.txt or not
Do we use both?
|4||obsos||11:20 pm Aug 1, 2004|
|msn search violating robots.txt!
`disallowed` images appear in preview on result page
|26||DoppyNL||4:15 pm Aug 1, 2004|
|Robots text file or noindex,nofollow
|4||mthorpe||4:18 am Jul 31, 2004|
|google allow - disallow all others
||6||jimmy19||7:20 am Jul 30, 2004|
|MSN and Yahoo both missing robots.txt
||4||bppilot||3:26 pm Jul 29, 2004|
|Yahoo robots.txt problem
|4||yowza||9:53 pm Jul 27, 2004|
|Google listing disallowed pages
|2||sonik||6:44 pm Jul 22, 2004|
|robots.txt for different language sites
can use charset tag to allow/disallow?
|4||a2ztranslate||11:51 pm Jul 20, 2004|
|about subdomain and robots
should i copy robots to all subdomains?
|2||webon||3:33 am Jul 18, 2004|
||3||motorcycle||5:30 pm Jul 17, 2004|
|Disallowing Robot's Effect on Sub-Domain
Will it disallow the sub domains as well
|2||bppilot||5:53 am Jul 14, 2004|
|Why is Google following my exclusions?
Robots.txt file says dont go and Google does!
|5||anxvariety||5:49 am Jul 14, 2004|
|Why are my .swf files still being indexed by Googlebot?
||3||ichthyous||6:26 pm Jul 13, 2004|
|excluding dynamic pages in all search engines
excluding dynamic pages in robots.txt
|3||toddmc04||2:49 pm Jul 13, 2004|
|General Guide to Robots.txt
is there one here :)
|4||rj87uk||7:51 pm Jul 7, 2004|
|domain.com vs www.domain.com
|9||HitProf||4:01 pm Jul 7, 2004|
|Do i need the txt file?
if i want to allow everyone
|4||markbiz||12:59 am Jul 5, 2004|
|I'm not understanding the different makes of bots
Can I stop some, while allowing others?
|2||tomequin||10:11 pm Jul 1, 2004|
|Yahoo! Slurp - robots.txt
Not obeyng robots.txt
|5||EarWig||1:59 pm Jun 29, 2004|
||4||lifedeuce||1:22 pm Jun 29, 2004|