paynt - 6:11 pm on Jun 13, 2001 (gmt 0)

In response to the email I get from my posts on hubs I’m starting this discussion to bring them together and answer some of the terrific questions I receive that I feel will benefit anyone interested. I’m also really hoping that others of you with experience on these issues will jump in and share your views, experiences and ideas with the group. All technical support available will truly be appreciated.

A disclaimer right up front. I do not want to turn this into a promotional list of products people use to make this happen but I do encourage participation from companies with products. There are rules governing this so read them first before posting. Be sure that if you do join in your URL is in your profile. Also, sticky mail and email are great but I’m hoping we can keep much of the information in front of the public with this discussion. If you are game… here goes.

To even begin I suggest you become familiar with previous posts that I’ve participated in that relate to this topic.

Themes part 1
[webmasterworld.com...]
Themes part 2
[webmasterworld.com...]
Discussions on canonicals and links
[webmasterworld.com...]
[webmasterworld.com...]
[webmasterworld.com...]
Hubs/Authority
[webmasterworld.com...]
[webmasterworld.com...]
[webmasterworld.com...]
[webmasterworld.com...]
[webmasterworld.com...]
[webmasterworld.com...]
[webmasterworld.com...]
DMOZ Clones
[webmasterworld.com...]

I’m sure there some here that I’ve missed so if you find one that should be included, please post.

After you read through these you’ll see that I use ODP for my search and creating my directory although I do customize the results. To find out more about using ODP for this must-reads include:
[dmoz.org...]
[dmoz.org...]

Many of the questions I get revolve around using ODP and technical issues. I am not a technical guru but I do have access to one professional and he is worth every penny. If you purchase a product make sure they have great technical support. Here are random answers to questions I’ve received.

It appears to be a googlebot thing as I've seen it now on different hubs I've created, on different servers, using different scripts. It appears that Googlebot leaves a cache behind with every hit. I have never seen anyone anywhere talk about this. Last spider I was getting hit with 25,000 to 38,000 hits a day from googlebot. This can be a huge problem. Through Anaconda and with technical support we were able to set up a purge so I can click a link and purge the cache left by googlebot. Not knowing this can cause severe memory problems on your server. Googlebot hits me nearly every day. Last month there were only four days I wasn't visited by googlebot. I found my sister site was also receiving more googlebot visits than others appeared to be. This appears to be a good thing.

I've created two hubs now using dmoz. My present hub has 43,700 pages indexed and nearly all of those pages are due to the dmoz info. I think it's perfect because you can recreate your own directory to point to related categories but they appear to be your own. I have dynamic URL’s for the search function but we were able to modify the categories to include cgi but no ‘?’.

To prevent unrelated oddball searches coming up in the log files we have been narrowing the search options we offer using the dmoz info. This is important if we want to stay on theme. I would rather have 4,000 pages of the hub on Google that are on theme than the 40,000 plus with many unrelated. This is an area I am still working on and welcome any technical support from people experienced with this.

I do not actually store the dump on my server. That's what Anaconda is about or any of the POD or ODP++ scripts. I'm unfortunately not as techie as I would like to be. Here are a few more resources you can check out for this:

[portalscripts.com...]
Upload_Tools [dmoz.org]

I tried to go through my email and collect answers to most of the common questions I receive. I hope this helps and we can continue to discuss this. I am not an expert on this but I have over a year of experience now with research into development. There are experts among us and I certainly hope they jump in.