Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: open
One way to work it is to set up Apache's mod_rewrite to redirect Alltheweb spiders to a 404 page and resubmit
the pages you want to see dumped to Alltheweb only. May take a couple of weeks or so but should do the trick nicely.
Are you folks seeing the same with yours?
I'm wondering if Fast is using a low tech pseudo clustering algo now where they just check this result and only take the first from a domain. (admittidly, I've not checked real deep).
are the pages your getting dupes with subpages from the same domain, a-records, or c-names?
It just occured to me what that would sound like to a newbie. The rich suculent spidering of Inktomi 98, with a full bodied indexer from Excite 96, and seasoned to taste with search spices from 97 Infoseek. lol
It is a select crowd, this must be the place.
That wonderful blend was before my time. I hear you guys talk about InfoSeek and when one could optimise, submit (and check ones ranking within a minute) and then re-optimise.
Cannot imagine what that must have been like.
There was a big clean up on all my listings. Had a number of pages which were very similar in content and shape taking up all the results.
Sort of felt a bit like a rudimentary duplicate elimination filter which cleaned it up. Regarding the timing of it, approximately 2 mths ago.
w/in 3 days (ink) is pretty fast, IMO . sure beats the heck out of monthly.
brett, re: gnutella - agreed. i like the way you think. :)