[e-mail]

[web_t]

I don't know, but what does
[e-mail] do? I found it on my log!

Please help me.

[BoneHeadicus]

Do your part for freedom and democracy. Put this in your robots.txt

User-agent: DIIbot

Disallow: /

[web_t]

thank you, but I guess, I will like to have little bit help over here.

How to make robots.txt in addition, if I want a logging tool that will log *all* the SE Spiders hits - where can I get such a tool (it would be great if some free tool is available).

Can anyone please walk/guide me.....

Thank you.

[BoneHeadicus]

User-agent: *
Disallow: /css/
Disallow: /js/
Disallow: /logs/
User-agent: asterias
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: DIIbot
Disallow: /

User-agent: LinkWalker
Disallow: /

Take this and copy and paste it into Notepad or any text editor and save it as "robots.txt". Then ftp it into the root of your web.

This is a good start, however you'll want to change the paths for js and css etc to correlate to the structure of your own site.