Welcome to WebmasterWorld Guest from 54.226.246.160

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

robot@digital-integrity.com

     

web_t

11:02 pm on Mar 23, 2001 (gmt 0)



I don't know, but what does
robot@digital-integrity.com do? I found it on my log!

Please help me.

BoneHeadicus

11:23 pm on Mar 23, 2001 (gmt 0)

10+ Year Member



Do your part for freedom and democracy. Put this in your robots.txt

User-agent: DIIbot

Disallow: /

web_t

2:10 am on Mar 24, 2001 (gmt 0)



thank you, but I guess, I will like to have little bit help over here.

How to make robots.txt in addition, if I want a logging tool that will log *all* the SE Spiders hits - where can I get such a tool (it would be great if some free tool is available).

Can anyone please walk/guide me.....

Thank you.

BoneHeadicus

3:10 am on Mar 24, 2001 (gmt 0)

10+ Year Member



User-agent: *
Disallow: /css/
Disallow: /js/
Disallow: /logs/
User-agent: asterias
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: DIIbot
Disallow: /

User-agent: LinkWalker
Disallow: /

Take this and copy and paste it into Notepad or any text editor and save it as "robots.txt". Then ftp it into the root of your web.

This is a good start, however you'll want to change the paths for js and css etc to correlate to the structure of your own site.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month