homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Alexa robot
issue with alexa bot & alexa toolbar users
Michel Samuel

5+ Year Member

Msg#: 3337848 posted 5:54 am on May 12, 2007 (gmt 0)

This is the letter I got form Alexa. (WITHIN 24 HOURS OF SENDING MY COMPLIANT!)

I basically told them that their recording my site's traffic was the same as sitting outside someone's home or business and cataloging who goes in and out.

Under United States Law that is called "Be sitting and watching" and that was ilegal. Well they still claim to have a right to my traffic stats. But I have a partial victory. Read below

<snip please no reposting of emails, title restated to avoid confusion>

[edited by: goodroi at 3:12 pm (utc) on May 14, 2007]



5+ Year Member

Msg#: 3337848 posted 12:16 pm on May 12, 2007 (gmt 0)

How could alexa possibly record your site's traffic?
If all you wanted to do was to prevent their bot from spidering your site (is that what you call recording your traffic?), you could have saved yourself time by adding a line to your robots.txt. ;)


5+ Year Member

Msg#: 3337848 posted 12:27 pm on May 12, 2007 (gmt 0)

Alexa has a toolbar similar to other browser toolbars. When a user visits a site and the Alexa toolbar is running, they record the hit and other data...

Michel Samuel

5+ Year Member

Msg#: 3337848 posted 2:11 pm on May 12, 2007 (gmt 0)

Sometimes I make mistakes with the language. English is not my first language.

The bot was being bad.
I had a line in robots.txt to tell it to go away.

I configured my .htaccess file to prevent refers from the alexa site... AND I even found the Alexa IP addresses and excluded them from accessing my site.

Like the song...
But the BOT came back the very next day!

And now having locked them out.
Got a letter saying the have taken care of the bot.

Now they tell me they can't do anything about the Toolbar users. Alexa is like a disease that will not disappear.


WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 3337848 posted 3:20 pm on May 14, 2007 (gmt 0)

Hi Michel,

It is rare but it does happen from time to time that a major robot will not properly handle the robots.txt. It appears that Alexa did handle the situation as well as possible. You complained the bot misbehaved and indexed your site. Alexa removed your site from the index and requested snippets of your log file so they could further investigate and fix the bot.

You should also double check your code. In my experience 9 out of 10 misbehaving robots are actually improper website code.


Michel Samuel

5+ Year Member

Msg#: 3337848 posted 10:45 am on May 20, 2007 (gmt 0)

I will not get into the particulars legale because you have requested that I not do so. But this is what I have done.

I had the my host society look at my setup. It is working.
It is that 1 time from the 10 that the bot did not listen.

But with that stated...
My site is still listed in their cache.
And the bot has returned.

My robots.txt file is now with every possible never come to my home again command against the bot. My .htaccess file has been set to ban the alexa ip addresses, toolbar users, etc. All of my pages have been set to tell the bot to go away.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved