Top Searches Script Quote through the roof

Have i misunderstood the complexity of this script

[Whitey]

I asked my developer to estimate the hours required to insert a script that would produce top searches linked back to the product profiles on a given web pages for a given category through a given date range on an exisiting site.

Those results would then be stored and displayed for each previous month on various templates

e.g.

Top 20 A Widgets in Category B for November 2006
Top 20 B Widgets in Category B for November 2006
Top 20 Widgets in all Categories for November 2006

etc. etc

I received an estimate of between 30 and 40 hours work to do this. I thought this would be just a simple matter, or did i assume too much?

[physics]

30-40 hours seems high but also I wouldn't assume something like this is exactly a simple matter. You're going to need an administrative interface and also the programmer has to figure out how to get at the info you want (search strings, urls ... from apache logs or your db?).
Other issues to consider:
Is the developer the same one that programmed your site?
Have you worked with him/her before? Were the rates reasonable then?
Also, the programmers price per hour can be a factor too. If he's charging $100/hour then he should be coding faster than this but if it's $5/hour then things will move slower.

[Whitey]

Thanks Physics - Yes we use them for all of our work and they have been quite good over the years.

Their rates are $100 per hour and they work pretty fast.

The answer i got back when i questioned it was this:

Because of the way we're linking to categories and to detailed pages, this requires us to create a hook directly into the server logs to determine the most popular categories and links, it's not at all simple to accomplish.

Do all these types of solutions need to go to the server logs, or is there another way?

Our site is fairly large - around 125,000 pages

[edited by: Whitey at 5:53 am (utc) on Nov. 10, 2006]

[physics]

I think a better long-term solution would be for them to record the searches in the db as they're entered. I.e. customer enters search term on your site, hits submit, search is performed but also the search term is saved in a 'searches' table in your database. When the customer clicks a url from the search results that url is saved in a 'urls' table and it's id is linked to from the 'searches' table corresponding to the current search. Or some variation of that.
Then to produce the 'top 20' lists you just need to query the database.
Also, this would also allow for other types of reports/stats to be created/run in the future.

[topr8]

we are smaller than you - around 20,000 products, but we developed search along the lines that physics mentioned.

... eg. every single search is inserted into a database, and we pull all kinds of data from this - including top searches, and it is relatively simple, i would say that this was built in during the original development but i feel it should be a method you consider.

[Whitey]

Interesting - could i clarify that most site visitors enter our site's pages directly from the search engines e.g. "Widget green".

They then navigate to another page link like "Widget green with red stripes" or exit the site via an affiliates link.

Does this support my developers idea that the log files must be used or can this type of thing still be achieved?

[edited by: Whitey at 9:22 pm (utc) on Nov. 10, 2006]

[physics]

OK I thought you were talking about an on site search. Still you could adapt what I said to get the search string that was used to find your site by getting the search string from the referring url on the page and load it into the database, etc. Also you could track what affiliate links were clicked by passing them through a redirection script.

[Whitey]

Would the redirection script harm our internal keyword linking, and could it be combined with a "no follow" for the affiliate links [ so that we don't link directly, and pick up duplicate content penalties?

[physics]

You might not want to track the affiliate links at all ... but it might be a good idea to add in that functionality while you're doing the other things (so you can track what phrases drive traffic to what aff links). You could separate the aff links from the internal links by adding some sort of field in the url db, like 'aff'. Nofollow is up to you but might be a good idea.