Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Googlebot crawls /m/ by default?

         

lucy24

8:26 pm on Jun 7, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Found something unexpected while running logs* on my test site. It's fully roboted-out, and its domain name is pretty generic, so I had to use a site: operator to force a result. For the test phrase "example" (that is, ahem, the site's name as a phrase) I got three hits, each with the "A description is unavailable" boilerplate:

example.com/m/
www.example.com/
example.com/

Now, the with-and-without www is understandable: Since they have never requested anything from the site, they don't know that one version redirects to the other. But where on earth did they get the /m/ let alone think to put it first? Is this form so ubiquitous that a clever search engine will assume it's present even if they have not one shred of evidence to say so? (It doesn't exist, in fact-- and I did the test search on a desktop.)

Option B is that they're putting up an URL they met years ago, on the off chance that it's still valid. But if so, why not some bona fide pages as well? (A further test search, with site: alone and no search term at all, leads to the same three results.) I've had the name for about 3 years, and cursory research** suggests it was parked since at least 2009 before the owner let it lapse. Was the /m/ format already widespread in, let's say, 2008 or earlier?

If anyone else has got a roboted-out site, see what comes up.


* A human visitor gave google.com as referer. This seemed awfully unlikely to me, since google has never been much for EMDs, so I went exploring.
** Couldn't use Wayback Machine because they look at current robots.txt and refused to show me anything.

aristotle

12:14 am on Jun 8, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Maybe there are (or were) incoming links to those pages. But googlebot can't verify whether the pages currently exist or not. So it has to be given rules to follow as to what to do in such a case. And what you see is the result of those rules, whatever they are.

not2easy

12:50 am on Jun 8, 2015 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



I'd view it as a default request. I recently see it listed as a 404 on at least 2 domains so far. No URL containing "/m/" has never existed on these domains, one of them I've owned since 2001 so I feel confident that it has never existed on that domain. The domain name is a made-up word. They are fishing.

LifeinAsia

12:56 am on Jun 8, 2015 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Definitely fishing- I've been seeing requests for /m/* and /mobile/* for several days on a site that has never had an /m or /mobile in its history.

lucy24

2:05 am on Jun 8, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I'd view it as a default request.

Not a request: a search result. So far I've never seen a request for /m/ or /mobile/ on any site. (Hm, who knows. Maybe they only put in these requests if they have not seen a responsive CSS?)

not2easy

2:42 am on Jun 8, 2015 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



One of the sites where they show me /m/ as a 404 is 100% responsive and I never saw them trying to find /m/ before it was responsive. It is like they think you secretly have a mobile site running parallel to the responsive one or something. There is no sense in it that I can see.