http://www.webmasterworld.com Welcome to WebmasterWorld Guest from 38.107.191.101
register, login, search, glossary, subscribe, help, library, PubCon, announcements , recent posts, unanswered posts
Subscribe and Support WebmasterWorld
Home / Forums Index / The Search Engine World / Search Engine Spider and User Agent Identification
Forum Library : Charter : Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification

  
user agent: www.zatka.com
Appears to be a new Indian search engine that doesn't read robots.txt
GaryK


#:401372
 1:49 pm on Aug. 14, 2005 (utc 0)

I saw the www.zatka.com user agent in my logs this past week.

On the surface it appears to be the crawler for an Indian search engine called Zatka.

There were two things that struck me as odd about this user agent:

1) It did not request robots.txt, and

2) The referrer (http://www.rndinfo.com/) is for an India-based software development company called Radix.

I could not find anything on WW about Zatka or Radix other than WW has a member named Radix.

If it's really a search engine it's already signed its death warrant for not reading robots.txt.

Is it possible this search engine is using technology from Radix to do its crawling? Again, if so then not reading robots.txt is a serious mistake on the part of Radix's software developers.

Thanks for any and all comments.

mcneely


#:401373
 4:24 pm on Aug. 15, 2005 (utc 0)

zatka.com is a search engine from india

Haven't seen them around, so I'm not at all familiar with them.

I've got so many ip's denied that come from india as a result of spam, that I might even have these guys denied as well.

rndinfo.com are their nameservers

GaryK


#:401374
 12:26 am on Aug. 16, 2005 (utc 0)

You're right that ns.rndinfo.com is their nameserver. Good catch. I wonder why they're including a link to the nameserver host's website as their referrer?

 

Home / Forums Index / The Search Engine World / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
WebmasterWorld ® and PubCon ® are a Registered Trademarks of WebmasterWorld Inc.
© WebmasterWorld Inc. / SearchEngineWorld 1996-2009 all rights reserved