Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

How does Google's spider crawl DMOZ?

         

thecuezone

2:54 am on Jun 11, 2005 (gmt 0)

10+ Year Member



Is there any specific method how Google crawls DMOZ Directories and Sub-directories?

When viewing Google cached pages of a particular DMOZ directory (Top: Shopping : Sports), I notice that there is a wide discrepancy on the date that particular sub-directories are cached by the spider.
They range anywhere from April 21st to June 10th.

I cannot see any particular logic why some are cached more recently than others. It doesn't seem to be based on alpabetical listing, number of sites in the subdirectory, popularity of the directory (e.g Petanque vs Martial Arts)

Can anyone offer any ideas?

aeclark

7:12 am on Jun 16, 2005 (gmt 0)

10+ Year Member



I may well be wrong, but I would assume the googlebot treats the DMOZ pages like other pages on other sites- the more frequently a page is updated/has its content changed in some way, and the higher the page's PR, the more frequently the page is revisited by googlebot, and re-cached.