Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: open
Has anyone out there has any experience with Teoma/AskJeeves Crawler.I have been continously crawled since March 7 and it is still crawling as I am writing this message.
Teoma crawler (18.104.22.168) is building up major Traffic on our website I mean March-April Traffic from Teoma was 1607.71MB and May-June Traffic is 3114.91MB.
I would have no problems with Teoma if I would had a Single click from Teoma/AskJeeves.
Googlebot Built up 1532.94MB Traffic In March but at least we got 4099 Pages Listed on google and are getting 60% traffic from google search.
My Question to other Webmasters is:
1. Is there a way of checking how many Pages are Indexed in Teoma. Something Like " Link: foobar.com"
2. Is it worth Paying for the Traffic from Teoma to your Hosting company which is like .07 cents per MB.
My decision to block (22.214.171.124) Teoma Totaly Depend on the feedback I get from Webmasters World as I don't Want Teoma Spamming our Website.
Welcome to WebmasterWorld [webmasterworld.com]!
Why not use robots.txt to limit the scope of the pages that Teoma can crawl? Or use the meta robots tag to do the same thing?
It never hurts to be in AJ/Teoma's index, but you don't need to let them crawl your whole site if it's costing you.
I'm trying to get back into AJ/Teoma - they dropped a free listing I had when I made major changes to the site. Obviously, the value of AJ/Teoma depends on your site's demographics, but it never hurts to be listed in any reputable engine. Send 'em over here if you don't want 'em... ;)
Blocking certain parts of the site is not as option as we sell Products online. And the big Problem is Teoma crawler is Going over the same Pages 10-15 Times is a row.
Is there any way of checking links to our website in Teoma/AskJeeves Search.
If you are using session IDs, you may have created a spider trap, since the crawler will get a new session each time it crawls, and it will think it's on a different page. In these cases, it's a good idea to always give each spider a fixed session ID - or none at all.
Where you draw the line on allowing crawling is a trade-off between depth of listings and bandwidth. You could allow it to spider category pages, but not individual product pages, for example.
carfac's problem is a common one; We can only hope they fix their crawler. I like to plan and build for the future, so I'll support AJ/Teoma in the hope that they will get it right eventually.
I do agree that their responsiveness leaves a lot to be desired.
As or website is designed keeping Bot's in mind, there is so session ID's in the url unless the customer is "Logged On" or has Something in there Shopping Cart.
Otherwise we won't be having much Luck with Googlebot either and Googlebot is kind enough to give is 4099 Links in there Index with a PR 2 Site.