Forum Moderators: bakedjake

Message Too Old, No Replies

Why is Alexa crawling grub.txt

Alexa crawler is crawling grub.txt on my server

         

sdani

3:17 pm on Aug 29, 2003 (gmt 0)

10+ Year Member



Yesrterday night, Alexa's crawler crawled my grub.txt. The file contains only a numeric ID. Anyone has any idea why would Alexa want my grub id?

jeremy goodrich

4:44 pm on Aug 29, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Alexa will crawl anything & everything they can find online...so, they may have found a link to the file someplace else & then that triggered their spider.

sdani

4:49 pm on Aug 29, 2003 (gmt 0)

10+ Year Member



I know exactly from where they found a link to it .. I installed Alexa tool bar. When I was trying to install grub crawler.. since it was giving me errors (grub crawler) that grub.txt is not found.. I entered in my browser's address bar - "my site * com / grub . txt " .. and I guess Alexa remembered that URL.
Thanks for the hint. I got the answer.

Does google also follow the same stuff? If I install google toolbar and visit my site several times a day, will google also start crawling my web site?

jeremy goodrich

5:07 pm on Aug 29, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



From all the tests, yes, Google spidering will be triggered by a visit from the Google Toolbar.

However, their rep here (GoogleGuy) has denied it.