Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: mademetop
Is your audience global or regional?
The one that comes to mind is Seznam (Czech, I think). Crawls all the time, and behaves itself.
Yeti/Naver is the big Korean name.
There's a Chinese Baidu and a Japanese one.
B, G, and Y are what... 99% of all search?
Unless these smaller SE have some kind of advertising that can be tapped into, there might not be much reason (from a money point of view).
Not investing much here: I am just working on positioning myself for the time when G is just one of several players, and all I am going to do is open robots.txt to those smaller ones that are reported to be well-behaved and of decent quality of their results.
B, G, and Y will be in the top percent for a decade or more.
That does not mean one should not court the smaller SE! Not by a long shot. Just depends on the desired audience... and the SE's in use there.
If you take the total number of daily searches on Google and compare them to the total number of other search engines combined, Google is still miles ahead.
For every hour you spend pleasing Google spend a few mins for the rest.
18.104.22.168 - - [30/Jan/2014:09:08:10 -0800] "GET /robots.txt HTTP/1.1" 200 623 "-" "SeznamBot/3.0 (+http://fulltext.sblog.cz/)"
Nový User-Agent string, který se bude používat od února 2014:
User-Agent: Mozilla/5.0 (compatible; SeznamBot/3.2; +http://fulltext.sblog.cz/)
How on earth have you prevented Seznam from crawling your pages?
All this is assuming you've got an ARIN domain.