superclown2 - 8:14 pm on Oct 11, 2007 (gmt 0)
One of my female staff complained this afternoon about a particularly offensive search term in the logs of one of our sites - that term about increasing a certain size. I gave instructions for several hundred robots.txt files to be altered over the next weekend to block the spider. Now I've just read on other threads that their bot sometimes ignores robots.txt. Quality control? What has that term got to do with a mortgage site? I cannot express my disgust.