Welcome to WebmasterWorld Guest from 184.108.40.206
Forum Moderators: open
Thank you for your sticky with your search terms.
You have highlighted a serious problem that FAST must have with their algo.
As you say, you have put in a query for one thing, say something unrelated like "bathroom renovations" and got pages of identical results on a hotel directory.
IMO it would be helpful if the either the moderator allowed you to publish your search term, or someone from Fast contacted you by sticky.
It would be interesting to discuss the implications on the board
For example, some exceptionally relevant pages in Google are now being penalised with lower rankings because of their file type. (e.g. try and find any dynamically generated perl files in the listings - they are there but are so low down compared to eight weeks ago that they are hardly worth listing them!)
Watch out for traffic coming form one of those, Ideavirus. Fast is a technology company, not a searchengine. ATW (AlltheWeb) is a showcase for their technology. Traffic comes from the partners.
I will explain:
this search string pulls pages from our site
This search string is the 2nd set of SERP's see how the URL changes to incorporate the page depth.
Now this is the bit i have been watching for some 6 months in hope that we get a whack more pages in. If you change the number 10 in that query string to read 6000 as per below
3..http://www.alltheweb.com/search?q=+azom&c=web&o=6000&f=+%2Bsiteid.siteid%3A9977258&l=any you will see that it pulls 6000 pages even though the original query only pulls 1000 results. It seems that our extra 5000 pages are filtered.
We know from our logs how deep FAST has been hence the playing around with their query string. We have been able to do this for the last 6 months and eagerly await the inclusion of such a massive volume increase in pages.
Sometimes it will let you click through the entire 6000 pages sometimes not.
The URL's after the 1000 page barrier seem to get messy though inj terms of titles description etc and this to me indicates either a cap or problems with their crawl in some way. I dont know the answer but as this has been going on so long i wondered what you guys thought.