Yesrterday night, Alexa's crawler crawled my grub.txt. The file contains only a numeric ID. Anyone has any idea why would Alexa want my grub id?
jeremy goodrich
4:44 pm on Aug 29, 2003 (gmt 0)
Alexa will crawl anything & everything they can find online...so, they may have found a link to the file someplace else & then that triggered their spider.
sdani
4:49 pm on Aug 29, 2003 (gmt 0)
I know exactly from where they found a link to it .. I installed Alexa tool bar. When I was trying to install grub crawler.. since it was giving me errors (grub crawler) that grub.txt is not found.. I entered in my browser's address bar - "my site * com / grub . txt " .. and I guess Alexa remembered that URL. Thanks for the hint. I got the answer.
Does google also follow the same stuff? If I install google toolbar and visit my site several times a day, will google also start crawling my web site?
jeremy goodrich
5:07 pm on Aug 29, 2003 (gmt 0)
From all the tests, yes, Google spidering will be triggered by a visit from the Google Toolbar.
However, their rep here (GoogleGuy) has denied it.