Burf.co

Burf.com returning from the graveyard

[burf2000]

Hi everyone, not been on the site for a long time! For anyone who remembers I built Burf.com, a small search engine about 10 years back.

Last month I decided to build a new engine Burf.co. It's still in progress as I am trying to work out the right niche at the moment.

I have written most of it in Swift which has been great fun. MongoDB for storage, which has given me some headache.

[keyplyr]

Hi burf200 and welcome back to Webmaster World.

What changes/improvements do you plan to make?

Will your index be all sites or genre specific? All file types?

And what will be the full User Agent string and IP range so we can recognize your bot when it comes around in our server logs?

[burf2000]

So this is a completely new project (Burf got sold and died as we know it), I am now a mobile dev and I really missed the fun of Search engines, how websites work and connect etc.

I think the whole privacy stuff has got me thinking but I think DuckDuckGo has that covered? As the moment it indexes everything to check the indexing code works. Indexing pages now seems a bit harder with all sorts of characters in URLs and Javascript rendered pages.

I had a thought about categories pages so people could search for just technical stuff, or news etc. I know Google does an epic job, it's just interesting to learn about. This is not really a project about making money.

Burf Search Engine is the agent.

[Travis]

@burf2000, I wish you all the best in your new project !

[keyplyr]

Burf Search Engine is the agent.

That's the complete UA string?

In order for a lot of sites to allow access, they need to create exceptions to existing block rules if you plan to use any of the large server farms (Amazon, Google, OVH, GoDaddy, LeaseWeb, RackSpace, etc)

[burf2000]

OK what would you suggest?

I rewrote the indexer this morning as its getting too slow at fairly low number (5 mil). Reading up a lot. Yes I am reinventing the wheel but its soo much fun doing it

[keyplyr]

From a webmaster point of view, we like to know:
• who is accessing our digital properties
• why you want it
• what you plan to do with it
• why we should allow your bot to do this.
Basically, will it benefit our interests as site owners.

So an appropriate UA might be:

Burf Search Engine; +https://example.com/info.html

or

BurfBot/1.0; +https://example.com/info.html

Where the info.html is an accessible page on your server than explains the above mentioned points.

Requesting & supporting robots.txt is always a plus :)

[burf2000]

That's really useful information mate, I will put that on the list todo as I think that will also get more people visiting the site to find out more.

[Travis]

Also, make your crawler to respect the robots.txt file.

[burf2000]

Yeah that's also on my list to do, I think that may take me a little while to work out.

[lucy24]

Is it just me or ...

Does the conjunction of “burf” and “returning from” lead to unfortunate mental images?

I guess it’s just me.

[keyplyr]

If you have to ask, it probably is ;)

...and any derived humor is overwritten by the disgusting imagery.

[burf2000]

I think that's the urban dictionary definition for it mate.

On an update of [Burf.co ], I actually started using the common crawl index

[robzilla]

Ahh, the nostalgia of "Submit site".