So this is a completely new project (Burf got sold and died as we know it), I am now a mobile dev and I really missed the fun of Search engines, how websites work and connect etc.
I had a thought about categories pages so people could search for just technical stuff, or news etc. I know Google does an epic job, it's just interesting to learn about. This is not really a project about making money.
In order for a lot of sites to allow access, they need to create exceptions to existing block rules if you plan to use any of the large server farms (Amazon, Google, OVH, GoDaddy, LeaseWeb, RackSpace, etc)
From a webmaster point of view, we like to know: • who is accessing our digital properties • why you want it • what you plan to do with it • why we should allow your bot to do this. Basically, will it benefit our interests as site owners.