Forum Moderators: open

Message Too Old, No Replies

DMOZ outages affecting Google crawl?

category won't load for me...what about Google?

         

arc_light

8:43 pm on Mar 17, 2003 (gmt 0)

10+ Year Member



Hi fellow directory fans - just got a DMOZ listing (yay) but since I can only seem to get the category to load about 25% of the time, I'm wondering if Google's going to have the same problem during its crawl. Which would be bad. Do any of you know if this is the case? I'm not much of an expert on spiders and would appreciate any insight you can offer...thanks much.

rfgdxm1

8:53 pm on Mar 17, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I had the same concern. There is an alternate port that editors use which normally does load, and Google may crawl using that. They may also use the official mirror at [ch.dmoz.org,...] and just change the URL to root for their indexing. Considering that Google considers the ODP to be specially important (they actually on the Google site recommend to webmasters that they submit to the ODP), Google likely would take special efforts to crawl it.

g1smd

9:23 pm on Mar 17, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Google takes a copy of the RDF file once a month anyway. This is primarily used to update directory.google.com but still reflects in www.google.com eventually. I think that the Swiss ODP mirror is the preferred site for people to browse, and spiders to crawl, taking the load off the main server. The Swiss mirror should only be one day behind.