homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

Newbie - what is cr001.digital-integrity.com?
Are we beeing checked for cloaking?

 2:21 pm on Jan 4, 2001 (gmt 0)

Hi all,

My mother tongue is French, i will try to make sense, OK?

I recently recieved the visit of sneaky robot. It seems to provoke a 404 to test the server.

Here is a clip from the log. - - [04/Jan/2001:02:34:16 -0500] "GET /robots.txt HTTP/1.0" 200 354 - - [04/Jan/2001:02:34:22 -0500] "GET / HTTP/1.0" 200 14788 - - [04/Jan/2001:02:34:20 -0500] "GET /test404response462450495.html HTTP/1.0"

The site uses java script redirect to the index.htm file, only because I use frames.
A couple of real content pages where optimised for targeted engines, but I neither use entry pages nor cloaking. I visited digital integrity web site to discover that they sell robots.

Can some one teach me about this robot, please?

Edited by: Macguru



 2:36 pm on Jan 4, 2001 (gmt 0)

Digital Integrity is a company that searches the web looking for information that has been stolen then posted on other web sites.

Like if you copied news report from cnn.com and posted it on your site without their approval. Digital Integrity is one of those companies that searches around the web looking to see if this news source is posted on any other sites.

Or if you post photographs on your site that you didn't license from the photographer.

These are the types of things they spider for. If you didn't nick anything from other peoples' sites without their consent then you have nothing to worry about.


 2:40 pm on Jan 4, 2001 (gmt 0)

Thanks msgraph,

I have nothing to worry about, then.

But can someone tell me more about the purpose of testing a 404 response?

I am very new to SEO... a lot to learn


 2:43 pm on Jan 4, 2001 (gmt 0)

I will usually ban these types of companies from viewing my pages. Not that I have anything to hide from them, just that I get really annoyed by their foolish snooping tactics.


 8:24 am on Jan 6, 2001 (gmt 0)

i ended up with them at my site too. does it mean someone has paid to have my site looked at? or do they just look at sites randomly for some companies who are randomly searching for thieves? how do they know if you are using someone else's stuff? i

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved