homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Ask - Teoma
Forum Library, Charter, Moderator: open

Ask - Teoma Forum

Is it Worth Crawled/Spammed by Teoma
Being Continously crawlled by Teoma since March 7th but No Listing

 3:54 am on May 17, 2003 (gmt 0)

Hi There,

Has anyone out there has any experience with Teoma/AskJeeves Crawler.I have been continously crawled since March 7 and it is still crawling as I am writing this message.

Teoma crawler ( is building up major Traffic on our website I mean March-April Traffic from Teoma was 1607.71MB and May-June Traffic is 3114.91MB.

I would have no problems with Teoma if I would had a Single click from Teoma/AskJeeves.

Googlebot Built up 1532.94MB Traffic In March but at least we got 4099 Pages Listed on google and are getting 60% traffic from google search.

My Question to other Webmasters is:

1. Is there a way of checking how many Pages are Indexed in Teoma. Something Like " Link: foobar.com"

2. Is it worth Paying for the Traffic from Teoma to your Hosting company which is like .07 cents per MB.

My decision to block ( Teoma Totaly Depend on the feedback I get from Webmasters World as I don't Want Teoma Spamming our Website.



 4:05 am on May 17, 2003 (gmt 0)


Welcome to WebmasterWorld [webmasterworld.com]!

Why not use robots.txt to limit the scope of the pages that Teoma can crawl? Or use the meta robots tag to do the same thing?

It never hurts to be in AJ/Teoma's index, but you don't need to let them crawl your whole site if it's costing you.

I'm trying to get back into AJ/Teoma - they dropped a free listing I had when I made major changes to the site. Obviously, the value of AJ/Teoma depends on your site's demographics, but it never hurts to be listed in any reputable engine. Send 'em over here if you don't want 'em... ;)



 4:24 am on May 17, 2003 (gmt 0)

You are right on that: Only if there was a Listing it woudn't Hurt.

Blocking certain parts of the site is not as option as we sell Products online. And the big Problem is Teoma crawler is Going over the same Pages 10-15 Times is a row.

Is there any way of checking links to our website in Teoma/AskJeeves Search.


 4:30 am on May 17, 2003 (gmt 0)

I have LONG had a problem with Ask Jeeves. I do not mind it.... but the damn bot keeps ignoring robots.txt and getting itself banned. I used to save it. I notified Ask 2 or 3 times, and never once got the courtesy of a reply. (Would be nice if we had a JeeveGuy here!) In any case, I just got tired of fixing Jeeve's own stupid crawling mistakes, and I no longer "unban" it automatically. At this point in time, I could not tell you if they are allowed on mys ite or not- but I would not be surprised if Jeeves was banned again! Bottom line: Jeeves: NOT worth the hassle, especially if they do not respond to problems!



 5:00 am on May 17, 2003 (gmt 0)

> Going over the same Pages 10-15 times

If you are using session IDs, you may have created a spider trap, since the crawler will get a new session each time it crawls, and it will think it's on a different page. In these cases, it's a good idea to always give each spider a fixed session ID - or none at all.

Where you draw the line on allowing crawling is a trade-off between depth of listings and bandwidth. You could allow it to spider category pages, but not individual product pages, for example.

carfac's problem is a common one; We can only hope they fix their crawler. I like to plan and build for the future, so I'll support AJ/Teoma in the hope that they will get it right eventually.

I do agree that their responsiveness leaves a lot to be desired.



 5:12 am on May 17, 2003 (gmt 0)

> Going over the same Pages 10-15 times
> using session IDs

As or website is designed keeping Bot's in mind, there is so session ID's in the url unless the customer is "Logged On" or has Something in there Shopping Cart.

Otherwise we won't be having much Luck with Googlebot either and Googlebot is kind enough to give is 4099 Links in there Index with a PR 2 Site.


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Ask - Teoma
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved