Is it Worth Crawled/Spammed by Teoma

Being Continously crawlled by Teoma since March 7th but No Listing

[Roger_M]

Hi There,

Has anyone out there has any experience with Teoma/AskJeeves Crawler.I have been continously crawled since March 7 and it is still crawling as I am writing this message.

Teoma crawler (65.214.36.115) is building up major Traffic on our website I mean March-April Traffic from Teoma was 1607.71MB and May-June Traffic is 3114.91MB.

I would have no problems with Teoma if I would had a Single click from Teoma/AskJeeves.

Googlebot Built up 1532.94MB Traffic In March but at least we got 4099 Pages Listed on google and are getting 60% traffic from google search.

My Question to other Webmasters is:

1. Is there a way of checking how many Pages are Indexed in Teoma. Something Like " Link: foobar.com"

2. Is it worth Paying for the Traffic from Teoma to your Hosting company which is like .07 cents per MB.

My decision to block (65.214.36.115) Teoma Totaly Depend on the feedback I get from Webmasters World as I don't Want Teoma Spamming our Website.

[jdMorgan]

Roger_M,

Welcome to WebmasterWorld [webmasterworld.com]!

Why not use robots.txt to limit the scope of the pages that Teoma can crawl? Or use the meta robots tag to do the same thing?

It never hurts to be in AJ/Teoma's index, but you don't need to let them crawl your whole site if it's costing you.

I'm trying to get back into AJ/Teoma - they dropped a free listing I had when I made major changes to the site. Obviously, the value of AJ/Teoma depends on your site's demographics, but it never hurts to be listed in any reputable engine. Send 'em over here if you don't want 'em... ;)

Jim

[Roger_M]

You are right on that: Only if there was a Listing it woudn't Hurt.

Blocking certain parts of the site is not as option as we sell Products online. And the big Problem is Teoma crawler is Going over the same Pages 10-15 Times is a row.

Is there any way of checking links to our website in Teoma/AskJeeves Search.

[carfac]

I have LONG had a problem with Ask Jeeves. I do not mind it.... but the damn bot keeps ignoring robots.txt and getting itself banned. I used to save it. I notified Ask 2 or 3 times, and never once got the courtesy of a reply. (Would be nice if we had a JeeveGuy here!) In any case, I just got tired of fixing Jeeve's own stupid crawling mistakes, and I no longer "unban" it automatically. At this point in time, I could not tell you if they are allowed on mys ite or not- but I would not be surprised if Jeeves was banned again! Bottom line: Jeeves: NOT worth the hassle, especially if they do not respond to problems!

dave

[jdMorgan]

> Going over the same Pages 10-15 times

If you are using session IDs, you may have created a spider trap, since the crawler will get a new session each time it crawls, and it will think it's on a different page. In these cases, it's a good idea to always give each spider a fixed session ID - or none at all.

Where you draw the line on allowing crawling is a trade-off between depth of listings and bandwidth. You could allow it to spider category pages, but not individual product pages, for example.

carfac's problem is a common one; We can only hope they fix their crawler. I like to plan and build for the future, so I'll support AJ/Teoma in the hope that they will get it right eventually.

I do agree that their responsiveness leaves a lot to be desired.

Jim

[Roger_M]

> Going over the same Pages 10-15 times
> using session IDs

As or website is designed keeping Bot's in mind, there is so session ID's in the url unless the customer is "Logged On" or has Something in there Shopping Cart.

Otherwise we won't be having much Luck with Googlebot either and Googlebot is kind enough to give is 4099 Links in there Index with a PR 2 Site.

;