Welcome to WebmasterWorld Guest from 50.17.74.162

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

robot@digital-integrity.com

     

web_t

11:02 pm on Mar 23, 2001 (gmt 0)

10+ Year Member



I don't know, but what does
robot@digital-integrity.com do? I found it on my log!

Please help me.

BoneHeadicus

11:23 pm on Mar 23, 2001 (gmt 0)

10+ Year Member



Do your part for freedom and democracy. Put this in your robots.txt

User-agent: DIIbot

Disallow: /

web_t

2:10 am on Mar 24, 2001 (gmt 0)

10+ Year Member



thank you, but I guess, I will like to have little bit help over here.

How to make robots.txt in addition, if I want a logging tool that will log *all* the SE Spiders hits - where can I get such a tool (it would be great if some free tool is available).

Can anyone please walk/guide me.....

Thank you.

BoneHeadicus

3:10 am on Mar 24, 2001 (gmt 0)

10+ Year Member



User-agent: *
Disallow: /css/
Disallow: /js/
Disallow: /logs/
User-agent: asterias
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: DIIbot
Disallow: /

User-agent: LinkWalker
Disallow: /

Take this and copy and paste it into Notepad or any text editor and save it as "robots.txt". Then ftp it into the root of your web.

This is a good start, however you'll want to change the paths for js and css etc to correlate to the structure of your own site.