Validator Questions

Need Help Deciphering Validator Output

[Jioffri]

After using the validator the following output below the hashed line was generated. Can anyone please tell me within the robots.txt file where I am duplicating effort or basically wasting my effort and should combine the syntax so this works optimally? Any feedback would be appreciated.

Jio

---------------------------------------
It validates, but has some bad style.

8 warning Multiple wildcard User-agent values. These should be combined under one User-agent/Disallow pair.

User-agent: *
8 warning Duplicate User-agent value seen: *. It is unknown how spiders will react to duplicates.

User-agent: *
12 warning Multiple wildcard User-agent values. These should be combined under one User-agent/Disallow pair.

User-agent: *
12 warning Duplicate User-agent value seen: *. It is unknown how spiders will react to duplicates.

User-agent: *
16 warning Multiple wildcard User-agent values. These should be combined under one User-agent/Disallow pair.

User-agent: *
16 warning Duplicate User-agent value seen: *. It is unknown how spiders will react to duplicates.

User-agent: *
20 warning Multiple wildcard User-agent values. These should be combined under one User-agent/Disallow pair.

User-agent: *
20 warning Duplicate User-agent value seen: *. It is unknown how spiders will react to duplicates.

User-agent: *
24 warning Multiple wildcard User-agent values. These should be combined under one User-agent/Disallow pair.

User-agent: *
24 warning Duplicate User-agent value seen: *. It is unknown how spiders will react to duplicates.

User-agent: *
28 warning Multiple wildcard User-agent values. These should be combined under one User-agent/Disallow pair.

User-agent: *
28 warning Duplicate User-agent value seen: *. It is unknown how spiders will react to duplicates.

User-agent: *
32 warning Multiple wildcard User-agent values. These should be combined under one User-agent/Disallow pair.

User-agent: *
32 warning Duplicate User-agent value seen: *. It is unknown how spiders will react to duplicates.

User-agent: *
36 warning Multiple wildcard User-agent values. These should be combined under one User-agent/Disallow pair.

User-agent: *
36 warning Duplicate User-agent value seen: *. It is unknown how spiders will react to duplicates.

User-agent: *
40 warning Multiple wildcard User-agent values. These should be combined under one User-agent/Disallow pair.

User-agent: *
40 warning Duplicate User-agent value seen: *. It is unknown how spiders will react to duplicates.

User-agent: *
241 warning Duplicate User-agent value seen: no. It is unknown how spiders will react to duplicates.

User-agent: no
415 warning Comments were detected after a valid field name. Although ok, this is bad style.
User-agent: Senrigan/#*$!#*$!
448 warning Duplicate User-agent value seen: no. It is unknown how spiders will react to duplicates.

User-agent: no
514 warning Duplicate User-agent value seen: no. It is unknown how spiders will react to duplicates.

User-agent: no
544 warning Duplicate User-agent value seen: no. It is unknown how spiders will react to duplicates.

User-agent: no
550 warning Duplicate User-agent value seen: none. It is unknown how spiders will react to duplicates.

User-agent: none
601 warning Duplicate User-agent value seen: w3index. It is unknown how spiders will react to duplicates.

User-agent: w3index

robots.txt source code for [anywebsite.com...]
Line Code
1 User-agent: searchpreview
2 Disallow: /
3
4 User-agent: *
5 Disallow: /cgi-bin/
6 Disallow: /images/
7
8 User-agent: *
9 Disallow: /cgi/
10 Disallow: /_vti_inf.html/
11
12 User-agent: *
13 Disallow: /_private/
14 Disallow: /_borders/
15
16 User-agent: *
17 Disallow: /_derived/
18 Disallow: /_fpclass/
19
20 User-agent: *
21 Disallow: /_overlay/
22 Disallow: /_themes/
23
24 User-agent: *
25 Disallow: /_media/
26 Disallow: /_pdf/
27
28 User-agent: *
29 Disallow: /stats/
30 Disallow: /statshistory/
31
32 User-agent: *
33 Disallow: /timetracker/
34 Disallow: /_vti_bin/
35
36 User-agent: *
37 Disallow: /_vti_cnf/
38 Disallow: /_vti_log/
39
40 User-agent: *
41 Disallow: /_vti_pvt/
42 Disallow: /_vti_txt/
43
44 User-agent: Website Stripper
45 Disallow: /
46 User-agent: Konqueror/3.0
47 Disallow: /
48

[...]

646 User-agent: Googlebot
647 Disallow:
648
649 User-agent: msnbot
650 Disallow:
651
652 User-agent: METAGOPHER
653 Disallow:

[edited by: ThomasB at 3:07 am (utc) on Aug. 15, 2005]
[edit reason] shortened [/edit]

[larryhatch]

Validators, W3C especially, have a built in obfuscator designed to prevent humans from decyphering the output.

Example: You forgot your car keys. Output = "Engine start prevented by switch contact failure."
.. and THAT is a really easy one. W3C might say something like "tachometer low limit timeout fault."
but again that's way too easy.

Old pros have found that its best to look for clues in the very very first error message.
W3C does not indicate where YOU went wrong. W3C
tries to teach itself where W3C stumbled.
This helps W3C people design even darker messages.

Look at the very first error message. Anything after that likely results from the real problem.
Don't expect too much help. Everyone is too busy with error messages.

-Larry.

[Jioffri]

Let me ask this way.

If I were to use one User-Agent: a thru z

with a being an agent all the way through to z being the last agent and giving it Allow then this would eliminate duplication? Likewise, with the Deny? I am not expecting any help other than what duplicate syntax areas are being referred to since I am under the assumption that I need separate entries for each agent. Yes or no? I need an example of a good syntax or links for where to locate good exmamples. Thanks.

Jio

[Dijkgraaf]

Well the "Multiple wildcard User-agent values" are rather obvious.
Delete lines 7,8,11,12,15,16,19,20 etc.

User-agent: * should only appear ONCE, and then list all the disallows under that entry.

[Jioffri]

Thanks for clarifying that for me. I had looked up a link from this website that had all of the agents listed as allow or deny but they were all listed separately the way I had them. Like I said, I am trying to learn as much as possible and thanks for teaching me something about this. Have a good one!

Jio

[Dijkgraaf]

You can list all the agents seperately as allow or deny, but you can't repeat the same agent multiple times, nor the wildcard user-agent multiple times, which does seem to be the case in the sample you provided.

[Jioffri]

D,

Thanks again! However, I think it is very amusing that a computer can tell when I have "bad style"! Hahaha