Msg#: 17810 posted 2:19 am on Oct 23, 2003 (gmt 0)
I have c-panel as my web configurator-whateveratoritscalled, is there something in that or some other stats program that can tell me when and which pages googles spider has crawled? Whats the IP?
Is there a way of knowing who referred the bot? ie:which link it crawled in from?
I had multiple links from a pr7 site that gets updated daily and my serps went up the next day, two days later, same links but I've disappeared. Just wondering what could be the cause, no changes on the pr7 site. Can I tell somehow if google didn't crawl that link?
Also, my site is a pr4 but google indexes the home page and a few others every day. My content changes only minorly once every 7-10 days. Is that normal?
Msg#: 17810 posted 8:13 pm on Oct 24, 2003 (gmt 0)
The most helpful I found, was using a custom 404 page that sends an e-mail. Thus I see any stupid mistakes I make and as I don't have a robots.txt all the crawlers looking for it See if your cpanel allows you to define a custom error page and if yes plug a nice error message in that also sends an e-mail (look on g for custom error pages). M