Welcome to WebmasterWorld Guest from 18.104.22.168
Anyone know why? .. and how to invite it back?
Check this thread:
It seems they are always 6 months behind at least.
Alexa donates its crawl to the Internet archive after 6 months, so the Wayback Machine will never have the most recent stuff...
My robots.txt contains:
User-agent: *Disallow: /cgi-bin/
Thanks for the log tip... I'll write a script to check it on a daily basis.
I see it crawling my site :)
Become a Pro Member