Forum Moderators: open
Crawled 5 files yesterday, 1 file crawled 3 times and 2 other files were crawled twice each. I would be more happy if they crawl all different/unique pages instead. :)
...2 other files were crawled twice each.
Just a guess . . .
One of the requests to www.example.com/file_name.html and the other request to example.com/file_name.html.
[edited by: Woz at 3:39 am (utc) on May 14, 2003]
[edit reason] er, language ... [/edit]
The crawle is indeed indexing both my site, www and non-www version because I saw the index file for the non-www version appearing in www3.
What should I do? Nothing?
Because sometimes it isn't.
...and Google then sorts it out by applying a duplicate content filter. (Just another guess.)
How do you have a www. version, and non www. version of your web site? You mean you have two different web sites? One for www. and the other without it?
And, why? Because of Google?
Right. It is technically possible to have a completely different site at root than what is on the www subdomain.
All backlinks to my domain are either
www.mydomain.com/ or
www.mydomain.com/index.html
Both of those resolve okay.
In the appropriate category in ODP the site is listed without the trailing slash as
www.mydomain.com
As a result, the Google Directory has decided that the site is not listed in ODP.
I have requested that the url be changed to
www.mydomain.com/ in ODP but that request has been ignored.
Can anything be done on the Google side to fix this?
The 64.68 crawlers just won't play ball with my 301 redirect to non-www.
Freshbot doesn't seem to follow 301 redirects. It sees the redirect, then stops. But Deepbot does follow them for me (though I haven't tried non-www).
Although I am not bothering how aggresive the fresh bot is but why would it crawl the same files 2-3 times a day?
I always thought it was checking for updates on those pages...
I'd love to show you my log files from the past few days. I have a small site by the standards here, about 600 pages. The 64.68 crawlers have requested about half of them over the last three days.
Talking server codes the results have been:
200: about 15 times, all for robots.txt except once index.html, once an internal page, twice my custom error page.
302: 2, see above.
301: the remaining few hundreds, I'm not counting them.
The 301s are for requesting pages with 'www.'. Other search engine robots have no trouble with it apart from periodic hiccups but they learn to deal with it. Again it just seems a shame or a waste of time if it comes and seems to want to pull the pages but then won't follow a simple redirect.
Hmm. I'll ask about the 301's with freshbot.
Please... Any news yet? Thank you.
