Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: open
The entire index appears to be updated about once a month, though I remember just reading an interview with someone from Ink, and they stated that they were trying to get it down to every 18 days or so. (The article discussed a strategy of aiming for frequency of updates rather than sheer number of sites in the index....)
Not sure how it finds the pages, either, as it's not soley DMOZ listings - though those tend to get crawled pretty regularly. There are other odd pages that I think it must find one on my site that links to it one day, then adds it to a queue, and then gets around to it several days later? I dunno.
O.K., lets say your site has ... 3000 pages. Are you saying that INK will probably never crawl more than a few a day? That would mean that content is not king. A webmaster would then need to pack either the index page, or the INK included page, with all of his/her primary and secondary keywords. Hmmm...
It's a mystery to me. I can say I've never seen it enter on a page and start crawling links it comes across like Googlebot does.
I guess I'll know more in a few more updates (or another month).
about 15 hits from Inktomi Slurp on a daily basis. Not a huge amount, but more than any other spider.
I was under the impression that inclusion takes 48hrs or so i was lead to believe.
Looks to me like Inktomi need to sort things out, or at least have a word with ineedhits, it would be nice if ineedhits answered the email that I sent.
Anyone had similar experiences ?
By the way the URLs are not dodgy in the slightest, no cloaking, no hidden text, in fact they are hardly optimised.
I have PositionTech for indexing. I get "Disallowed by robots.txt" - problem is, I don't have a robots.txt (and when I did it used to work just fine, since it's the Webmaster World list).
Is it possible that Slurp! is being blocked by .htaccess? And if so, are you using a new agent which identifies itself as something other than Slurp!? None of the IPs I have blocked appear to match or even come close to yours.