Forum Moderators: open
The english site is already up and running:
[en.deepindex.com...]
The beta version will be online within the next few days. We invite you to submit your sites for free and test out the search abilities over the next six weeks. Our expected launch date for the new Engine is 27th December 2002.
Start for the free crawl on the UK side is to be expected in the second half of december and for germany at the end of january. Both servers will crawl free for 3 months and submissions will be opened in december for both servers.
Interesting use of the Microsoft graphics though, is that legal? (top right corner)
just started this index.
For the graphics : oups, you're right, will be changed in a day or 2. Meanwhile a lot's of sites do use them.
Coming back to main information and meaning of our roll out to english and german version.
The actual english version is on a test server with a small but quick growing index started from DMoz directory. The final version forecasted at end of december should have an index of about 5 million urls and some nice enhancements for better searching. For now we are just looking forward to test the english version and to get tested by natives webmasters and seo's. The french market usage is different from the english or german culture and we have to adapt the engine to these cultures.
All sites submitted will be indexed in www.en.deepindex.com in the test version. After the tests they will ALL be included in the final version.
I hope to get some feedback from webmasters and seo's.
Best regards
Gilbert
sure, the deep crawl hasn't even started yet.
Anyway, the looks like this engine is on the upswing. Lean, fast, where there is a reasonable index size, like for french pages, the algo looks ok, and the folks there are really getting into gear.
First if you start from DMOZ, this means what you will index, ~1,7 page from each link in dmoz.
Second, 5 Million url's in about a month, how many pages per day can you index?
Good question :
As the french market is quite rugh in announcements these days... In fact we can crawl per server 100-300.000 urls a day per server depending on the charge of the bandwith. We are running on five servers located in Paris, Bordeaux and Montreal. We are looking for a fourth data center in the first half of 2003, as we are having projects for other specialized search engines like DeepIndex News. It is depending on how our business plan rolls out, but for the moment it is good looking.
Then you need quite a few hundred servers quite quick if you want to be able to compete with the big boys, Fast can crawl over 150 million pages per day, and i don't even wana guess Google.
And even with simple software available today, if you have an ADSL line, you can crawl up to 100.000 pages per day.
Have you developed everything yourself, or do you continue to build on a software
Gigablast have 8 servers, and they have a index of around 150 million pages, so there must be something you do very differently.
> We invite you to submit your sites for free and test out the search abilities over the next six weeks
Where can I find the free add URL?
...submissions will be opened in december for both servers.
I suppose you mean pay for indexing?!
Yes rising a searchengine costs a lot. As we didn't raised any money, we didn't searched for it either, we got to finance our technical staff (>5 persons) and servers. Of course advertising is one kind of revenu, but in my opinion, to much advertising is not good either, and i prefere to make it pay for crawling. But! Yes there is a but (sorry bad translation from the french). But there is also a free entry as we are doing it on the french side. Just put a link to deepindex on any of you page, mail us (support-enATdeepindex.com) where you put the link and we crawl your site. Of course i prefer paid submission wich works on the following concept : Pay once and be crawled every month or week. I worked for a while whith other paid inclusion programs, but the time limit is not good (in my personal opinion) for webmasters, even if it nice to our business plan.
The normal crawling with a refresh rate of 1 month is indexing links which are not indexed yet but found on pages or on Dmoz directories (other also possible).
High ranking? A good title and a nice body is good for the spider. For flash, you should use NOEMBED tag. For scripts & frames, use NOSCRIPT/NOFRAMES tags.
Best regards
Nick
Best regards
Gilbert
Post scriptum :-) i'll put a submission form this evening on the site.
I agree, you have to fund it somehow.
I'm not sure that paid lisings is the answer as it immediatly limits the data and thereby the scope of the SERPs. That then feeds through to user frustration.
I don't have the answer for your business model, just that observation to help you assess the best ways to fund the search engine.
I really wish you success with your new search engine.
I'm working on SE's for 5 years now, and i always managed to make it the best possible in equity with paid inclusion and free inclusion to have a good balance of the web. But the most difficult is to find the balance in the money you spend and the one you earn with paid customers.
submission is now available :
[en.deepindex.com...]
Just submit homepage and email.
Best regards
Gilbert
yes except french version sites. These have to be submitted at [deepindex.com...] and are paid inclusion program.
other languages will be done also, but we prefere for the moment only english, french and german sites. We could be more quickly, if someone gives us more servers (we are actually running on five servers, and we need more then 10 other servers). If anybody has relation with IBM, HP, Dell, we would be pleased to workout a deal with them... please contact me with sticky mail or directly at gilbert@deepindex.com .
Our plan is to cover europe in 2003, that's makes a lot of languages and countries. So you can of course submit any site, but you should follow our time schedule for perfect indexation.
Best regards.
Gilbert
The databases are separated. In the french version i saw some chinese or japanese urls who screwed up the listing but who where linked by a french site....
[en.deepindex.com...] : it does cost you less than minute ;-)
Deepindex is now deepcrawling uk and us sites on www.en.deepindex.com . You can still submit sites which will be spidered on next business day ( [en.deepindex.com...] ).
I want also to adress many thanks to every body here who did give us some feedback and comments. Your suggestions are welcome.
Many thanks
Gilbert
this might be of some interest.
In the last weeks i got a lot of submission coming from italy and germany. none of them where actually included as deep crawling is currently on the way (some of you might have seen DeepIndex (www.en.deepindex.com)) bot. If there is any mis behaviour of the bot just send me an email.
In any way, i'll finish crawling dmoz and some other directories and will index submitted sites just after this crawl. I will in the same time run the other deepindex sites www.es.deepindex.com www.de.deepindex.com www.it.deepindex.com etc...
All european countries/regions are now welcome. Replys of the support service will be done in english french and german. do apologize please some mistakes in the differents languages, cause when i reply i do not use automate translation but only my knowledge.
Best regards
you might consider also deepindex for Spain and Portugal.
It's still free submission.
I'm setting up all european countries like :
www.es.deepindex.com and www.pt.deepindex.com
Of course deepindex is mostly running on french side, but the european side might come up quite quickly...
Best regards
[edited by: heini at 9:24 pm (utc) on Jan. 2, 2003]
[edit reason] split to here [/edit]
Will you include the Netherlands as well?
Thanks!
Well to answer your question i'd like to answer to the whole european area : YES
I pushed DeepIndex in the whole european internet area in fact. Even if not the whole europe is covered, i did pushed localized subdomains on deepindex for ex.
www.nl.deepindex.com
...
you should see them on the flags at www.en.deepindex.com
As my dutch is quite old (i learned it in school for 1 year) i could need some helps in translations (spanish,italian,portugese,danish, ...)...
Anyway, DeepIndex is actually covering whole europe (and some more) even if english, german and french speaking sites are a little more indexed then the others.
I presume this should nice balanced within a few weeks.
Sorry for mistakes in my writing.
Best regards and again a Happy, Happy new year to all.
[edited by: heini at 1:09 am (utc) on Jan. 4, 2003]
As my dutch is quite old (i learned it in school for 1 year) i could need some helps in translations
Do you mean translation of the website text? Feel free to sticky me if needed.
Some other questions:
- how do you plan to market your search engine/directory in the local markets?
- will there be a relation between the directory and the index?
Thanks and good luck.
Astrid
>- how do you plan to market your search engine/directory in the local markets?
In the first i shall work with local seo's and directories.
If these seo's companies do need a search engine for the animations or conference, send me a little sign, i shall come ... ;-)
If everything works out well local agencies should be build up, but this is not for now.
>- will there be a relation between the directory and the index?
I think you mean the directory which i build up on deepindex france? No. Even the french version should be stopped (except if any body want to use it and maintain it) DeepIndex take's to much time to me. Maintaining a directory is much more consuming than a search engine in human resources.
If you meant the directory like links on the different home page, i must tell you that these are not going in a directory but are keyword related searches in the search engine index.
I hope i answered at least partially your questions. For the translations of webpages i ll pass you a little sticky mail.
Best regards and tot ziens :-)
Gilbert
BTW: do some better filtering on pron sites ... deepindex returned some really uggly xxy and other spammy sites under the top 10 results of very, very (VERY) unrelated searches.
on what kind of terms? send me by mail (support-en@deepindex.com) so i can push them out.
thanks
the bot agent is "DeepIndex" (you can find him / and about him in google, and of course in the logs)
thanks for the comments i always appreciate
best regards
