voices - 7:45 pm on Jul 18, 2007 (gmt 0)
I know about robots.txt and have rarely used it. I dont want to look through all my servers with 10 years worth of crap on them. I should, but I really have better things to do with my time. The point is, in the past things weren't spidered if they weren't linked to. So where and why is google picking this stuff up from?