[e-mail]

[Son_House]

Made 8 requests for pages and also got robots.txt a number of times.

www.24sevennonstoppower.co.uk - - [27/Jun/2001:22:23:47 -0400] "GET /robots.txt HTTP/1.1" 200 1030 "-" "[e-mail]"

Now for the strange part. When I went to this url: http:*//www.24sevennonstoppower.co.uk/ I ended up at a page from http:*//www.ripe.net/ letting me know what my ip was.

So I went here http:*//www.relevare.com/
Here is the no flash version: http:*//www.relevare.com/htmlsite/

I didn't read a lot but they say they create vertical portals and the index is updated 24 hours a day, 7 days a week.

With that ua they use, can @ be put in the robots.txt file with out upsetting other bots?

[jimmykav]

I have had 21 requests in the last 2 days. No idea how i got onto their menu....

[Jaf]

> With that ua they use, can @ be put in the robots.txt file with out upsetting other bots?

I'd say no. It's quite common for well behaved bots to put a contact email address
in the UA, and "@" would be far to general. You'd end up barring FAST, Euroseek
and Inktomi to name but three.

I'd use the domain name "relevare.com" if you want to stop just this bot.

[Son_House]

> jimmykav - No idea how I got onto their menu....

I don't either as I have not submitted to an se in over a year and never heard of this bot till it hit our site. They are probably just following links.

> Jaf

Thanks for the info. I was think having @ in the robots.txt might mess things up. Maybe I should send them an email asking them to change the ua.

This bot is still hitting our site but so far it seems well behaved.

<update>Well I sent the email asking them nicely to change the ua, just have to wait and see if they do.</update>

[Jaf]

>Thanks for the info. I was think having @ in the robots.txt might mess things up. Maybe I
>should send them an email asking them to change the ua.

I'm no expert on barring robots (generally I welcome the traffic :-), but if you can do
a partial match, then I would use "relevare.com", "@" would match too many
others. If this isn't possible, then why not match the whole thing.

I don't think a "@" in the middle of a match string is going to cause problems with
other agents that contain an "@".

But like I say, I'm no expert, I just know that lots of UA's have "@"s in them.

[engine]

I've seen this in my logs this week.
Who knows what they are up to - it could be beneficial?

Anyone had contact with them?

[Son_House]

I sent them an email on June 30 but so far they have not responded.

They are still requesting pages from our site. Which is ok with me so far.

[grnidone]

Have you heard anything? I've had nothing but polite spiderings from this ... thing.

=G

[Son_House]

Still have not heard anything from them. They have also been polite at our site. I just wanted to see if they would change the ua. But now I agree with Jaf and don't think having @ in the robots.txt would matter.

engine > Who knows what they are up to - it could be beneficial?

Well I'm not sure what they are up to. I still think it is strange when I go to [24sevennonstoppower.co.uk...] I end up at ripe.net telling me what my ip is. I'm I the only one this happens to and or finds it strange?

[Marcia]

Doing a view-source: on that url, there's a link to www.iana.org within the source code, which is the Internet Assigned Numbers Authority.

Captioned at the top of the iana.org page:

"Dedicated to preserving the central coordinating functions of the global Internet for the public good."

Notation at the bottom of the iana page:

"This site is mirrored at [iana.netnod.se...] with the generous assistance of the Royal Institute of Technology's Network Operations Centre, and Netnod AB, operators of the D-GIX, Stockholm, Sweden."

[aztech_amys]

So is this an indexing spider, non-indexing, not a spider? Somehow I got the great job of making these decisions at my work and I am just a spider novice. Please help!

[geosom]

They do follow the robots protocol so
in your robots.txt file, paste the following:

User-agent: [e-mail]
Disallow: /