homepage Welcome to WebmasterWorld Guest from 204.236.255.69
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
robot@digital-integrity.com
web_t




msg:403637
 11:02 pm on Mar 23, 2001 (gmt 0)

I don't know, but what does
robot@digital-integrity.com do? I found it on my log!

Please help me.

 

BoneHeadicus




msg:403638
 11:23 pm on Mar 23, 2001 (gmt 0)

Do your part for freedom and democracy. Put this in your robots.txt

User-agent: DIIbot

Disallow: /

web_t




msg:403639
 2:10 am on Mar 24, 2001 (gmt 0)

thank you, but I guess, I will like to have little bit help over here.

How to make robots.txt in addition, if I want a logging tool that will log *all* the SE Spiders hits - where can I get such a tool (it would be great if some free tool is available).

Can anyone please walk/guide me.....

Thank you.

BoneHeadicus




msg:403640
 3:10 am on Mar 24, 2001 (gmt 0)

User-agent: *
Disallow: /css/
Disallow: /js/
Disallow: /logs/
User-agent: asterias
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: DIIbot
Disallow: /

User-agent: LinkWalker
Disallow: /

Take this and copy and paste it into Notepad or any text editor and save it as "robots.txt". Then ftp it into the root of your web.

This is a good start, however you'll want to change the paths for js and css etc to correlate to the structure of your own site.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved