after i stole ..uhmm... borrowed a template of a robots.txt file from pageoneresults, i came across a new problem.
our website search tool is Verity. spider is called "vspider".
realized, i was blocking it. ok, so thought i could add:
and be all good....
nope, its still being blocked.
anyone had an instance like this before?
if i cant allow this spider, and have to remove the lines:
am i going to have to manually add every other possible spider or bot?