INK Problems

What's Up With Their Spider?

[dstanovic]

I cannot understand this. This makes the second time they claim have there is a inclusion error in the robots.txt file. I removed my disallow for /cgi-bin/ and they had me put in the silly
User-agent: slurp
Allow: /
and it still gets a error 7000, which means an error in the robots text file. The funny thing is that I have the same robots.txt file on all of my sites that they spider. Fast can spider along with Teoma and everyone else, why does INK seem to have this problem? I had the same problem a month ago or so with another site and after a week the problem mysteriously corrected itself.

[jdMorgan]

*They* had you put in "Allow:" ???
There is no such directive in the robots exclusion standard!

In order to allow an agent, you want:
User-agent: agent_name
Disallow:

(The argument on the Disallow line is blank)

In order to disallow that agent entirely, you
want:

User-agent: agent-name
Disallow: /

But there is no "Allow" directive in the standard, so I'd get rid of that line.

Try validating your robots.txt with the validator on the Search Engine World part of this web site. If it validates, then the problem is on Ink's end.

[searchengineworld.com...]

Hope this helps,

Jim

[mayor]

According to this URL, the the 1996 proposed Robot Exlusion Standard accomodates the Allow directive ....

[robotstxt.org...]

And according to this URL Inktomi is using the 1996 proposed standard in situations where the 1994 standard is ambiguous.

[inktomi.com...]

[dstanovic]

They finally was able to spider the page so I re-vamped the robots.txt.