Forum Moderators: open
When I first launched my site, I didn't have a robots.txt file or meta tags that said index, follow. As a result, Inktomi came, went to my home page, and then went away. It did this on an almost daily basis.
I then put in a robots.txt file and changed all meta tags to index,follow. I know I've done it right since none of the other spiders are having any problems with it (thankfully!). Google, AV, Looksmart, FAST and others all continue to spider my site ok. Yet, Inktomi still hits my first page and then goes elsewhere. It even gives a return code of 200 in the logs for my robots.txt file, so it must like it.
Main question is - why is Inktomi still stopping at my first page and going no further? Is there anything I can do? I'm listed in DMOZ and have many backlinks - 9 of which show in the backlink check at MSN.
Moreover, I'm not even sure that Inktomi is actually taking in the first page. I've only ever gotten traffic from MSN due to my listing in the Zeal directory. I've never seen my site listed anywhere by Inktomi in the "web pages" area of MSN.
Any advice? Or am I just out of luck as far as Inktomi goes?
Thanks
Jim
You may want to look at paying for a couple of pages if you want to get in quickly.
My robots.txt file is pretty simple. It is:
User-agent: *
Disallow:
I did the usual check with the robots.txt checker at webmaster world and it seemed to like it ok.
I just submitted my site to Goo a few days ago - after reading the posts here about it.
Guess I'll just have to sit back and wait. Can't afford to pay since my site is a content site.
Jim
