Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: open
All the other SE's have indexed the whole site.
Could there be something wrong with my robots.txt file that somehow gives Slurp a headache and tells it to only get the index page. Here it is, a direct cut and paste:
# bad bots, you are not welcome here so get lost
It looks OK.
Just for insurance, try running your robots.txt file through the robots.txt validator [searchengineworld.com].
Ink is very slow unless you've paid them for inclusion and frequent spidering. I think it took them almost three months to pick up one of my non-paid sites... So, you can pay or wait. :(
Other people have reported this same behavior, but it seems no one has an explanation. Yet others are reporting that Slurp crawls all over their site.
It's not my policy to disclose URLs, but this site is an experiment with a serious dose of high quality content to see if content is really king. Google says "yes". Fast says "yes". Ask Jeeves says "yes". Alta Vista says "yes". But Inktomi is saying "no".
I actually built it from one of the robots.txt's I found here at webmasterworld.
I've had no problem with slurp deepcrawling my site over the past one or two months, although I did not submit anywhere or pay anyone for this. However for the first four months or so after they found me they only crawled my index page and sometimes one other main page. I don't know why slurp suddenly decided they liked me but I assumed it was either the passage of time or an increase in backlinks. Slurp has given my files a couple of good going overs since the deepcrawling started anyrate. The robots.txt was the same all this time so I don't think it made any difference.
Unless your file containing:
includes a more specific override for individual bots like Slurp, namely:
I would think you are <gasp> excluding all the bots
[edited by: mayor at 9:36 am (utc) on Dec. 4, 2002]
Heh, I think it's time I went to bed :)