Forum Moderators: open
Grabbed about 1800 pages over a period of two hours.
From their site:
Exava is a new company created by the founders of mySimon and Wisenut.
Exava, Inc. is a venture-funded start-up that is building the next generation search engine for shopping. We are developing an innovative new search engine technology that will significantly improve the online shopping experience.
(Doesn't seem to be related to the other "exabot" out of Paris, exabot.com, that's been talked about before.)
Jim
64.124.25.77 -- around 01:00 27th
69.111.170.194 -- around 21:00 27th
64.124.25.78 -- around 16:00 28th
(times/dates UTC)
For each domain, it does request my (empty) robots.txt first.
With one exception. It can't follow 301 redirects properly - it follows them but doesn't realise the domain has changed, so it requests all subsequent pages (links on the home page of the second domain) from the original (before redirect) domain.
I have a domain where every request is 301 redirected to the home page of a second domain, so Exabot is getting plenty of copies of the home page of that second domain, and no further.
I am pissed!
I noticed this because I'm writing a web log analyzer tool to make reports like Google's Zeitgeist. Despite its only redeeming quality being its improvement of my software, it is now disallowed.
If there are other issues, please feel free to email exabot@exava.com and we'll try to respond in a timely manner.
Here is our webmaster info page.
[exava.com...]
Hubert
exabot@exava.com
The spider is supposed to be for "shopping". I don't sell anything on my website and exabot crawled my entire site pretty darn quickly. For these reasons I have also disallowed it.
Allowing spiders out into the wild when they don't obey robots.txt is a rather good way to give your service a bad name, but I can forgive that when "testing" (but you should test on a fake web first!).
Forgetting to encode a space in a URL, however, is simply a sign of laziness, sloppiness or ignorance in your development and QA process. Properly encoding a URL is a few lines of code, and there are numerous examples of how to do this correctly on the web.
In general, we handle URL encoding, but this may be a new, unknown bug. Also some of our URL encoding fixes were not in earlier crawls(those before 8/28 I believe). What was the date on that "/foo/bar/<A HREF" request? Can you send email with that log entry to exabot@exava.com
Thanks for your feedback, and sorry again.
Hubert
Our robot does obey robots.txt, but it might have accidentally not obeyed robots.txt if your site was found through a redirect, which is a problem we are fixing
It's crazy even leaving this thing out crawling while it still can't follow a redirect (i.e. use the new domain for the links on the page it was redirected to). It fills your database with failed requests for non-existant pages.
For example I have some domains I redirect to a parking service. Those pages at the parking service have links which are redirections to ads. So, many of the 800+ requests Exabot sent to my server yesterday are failed requests for pages within the ad server, two redirects away...
1) Crawler is told to grab URL A.
2) Crawler downloads robots.txt from domain name in URL A
3) URL A passes robots.txt validation and gets requested
4) URL A is actually a redirect but its not known until it actually gets requested
Many standard libraries will follow redirected URLs automatically (its a typical default option) and therefore will fail to get robots.txt for URLs at all stages, so it is necessary to turn off automatic redirects following and process robots.txt files for each of possibly many redirects. Pain in ass, but appears to be necessary :(
Issue for webmasters is to make sure you don't exclude your tracking/redirecting URLs from robots.txt, or they won't be followed.
1) Crawler decides to download URL U from site S.
2) Crawler looks in robots.txt cache for S
2a) Crawler finds robots.txt entry expired/nonexistent
2b) Crawler downloads and parses robots.txt for S
3) Crawler finds U is allowed for S
4) Crawler attempts to download URL U and gets a redirect to R
At this point you should go to step 2, like this:
2) Crawler looks in robots.txt cache for S
3) Crawler finds R is disallowed for S and stops
2) Crawler looks in robots.txt cache for S
3) Crawler finds R is disallowed for S and stops
Not totally correct me thinking - when U is redirected, a new URL is given, and this new URL belongs to R, which is (at least in some cases) the other domain name than S, then the check for robots.txt should be done on appropriate new domain which is R, but not S:
[example.com...] is first checked against robots.txt on www.example.com, when it redirects to [example.net...] then that new URL - "somenewlink" should be checked against robots.txt on www.example.net.
Unless you see any flaws in that I will sleep well in knowledge that my algorithm is correct ;)
(You'd better checking against a the full URL!)
You'd better checking against a the full URL!
Who says I am writing a crawler? btw hope you did not confuse me with Exabot people ;)
Having said that generally downloaded pages are stored in data barrels with all headers, so any commands that instructing client to set cookies will be there, and analysis software can pick them up. I don't think top bots do that as I (currently) see no reason why.
I have reviewed my code and every page in my administration are checks for two authentication cookies before allowing the page to be loaded. Now, I have come to realize that I should have also been checking for the value of the cookie and not just the fact that one exists - that's a lesson learned. But I am still baffled at the fact that I cannot find an explanation for the bots ability to gan access other than the bot creating a cookie to do so.
When I review my server log files, I can see the bot accessing the page that clears the cookie more than 5 times in a row - making me think that that was when it was figuring out what cookies to set.
In any case, it seems that you are confirming that it is possible and maybe somewhat likely. I will put something together for the folks at exava and see how they reply.
