Forum Moderators: open
Didn't request robots.txt x2 visits.
Site just shows two links on the front page, "About Octora" and Contact," both of which link to -- the front page. (Sheesh.)
Ban-worthy in my book.
I'm the lead designer for Octora, and I would ask you not to ban Octora - Beta version. It is an extremely new bot that will become very useful with time. We have not enabled the feature of checking robots.txt because we're just running simple tests right now with the engine.
If you do not see information on www.octora.com by March 17th, or if we continue to visit your webpage without checking through your robot.txt after March 17th, by all means ban us.
But give us a break until then please.
Thank you,
Regards
Octora.com
It is an extremely new bot that will become very useful with time
Useful for who? :)
Welcome to WebmasterWorld!
[...] the feature of checking robots.txt
It's not a feature. It is a function essential for the survival of your robot. I suggest you test that 'feature' first.
Spidering without fetching and complying with robots.txt is like going into all the houses in your town without knocking, and then telling the residents, "Oh sorry, just checking out the neighborhood." Don't be surprised if your are reported to the police, thrown out on your hindquarters, or shot by some of those residents.
There's a right way and a wrong way to deploy a spider, and I encourage you to do it the right way.
Jim
