Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Custom 404 on robots.txt with link to corporate home page

Seeing people from google clicking it - is this a problem?

         

communitynews

6:27 pm on Apr 18, 2006 (gmt 0)

10+ Year Member



For some reason, I've noticed "people" from google (i.e. non-googlebot user agent) from a google owned IP, clicking on the logo on our custom 404 page which links to our corporate website. They are doing this when searching for /robots.txt on our CUSTOMER sites, most of which do not have a robots.txt file. The REFERRER shows the customer /robots.txt.

They've been doing it a couple of times a day. I've looked around the web and many sites without a robots.txt file have all kinds of text returned.

I'm just wondering if perhaps I should have the standard apache 404 message be returned. I know for a fact that the 404 status is being returned.

Any ideas on if this indicates something?

PS: I've checked the number of listings these customers have and they don't seem to be dropping or anything?

PPS: we also upgraded to apache just a few weeks ago from an very old no longer supported http server called Website.