i recently removed all the +1 buttons from my site after a month of giving it my best go.
i had facebook and twitter on there already, and added +1. i get stacks and stacks of facebook shares and tweets all the time, but next to no +1s. it cant be anything to do with the button placement, because the facebook and twitter are getting loads -- and they are in exactly the same place.
the reason i removed it wasn't to do with page speed or anything like that, but im worried that google will start to see my site as being not very popular, if i dont rack up the clicks.
if it wasn't for the fact that they said it was tied into the algo, and possibly affecting your serps position, then i probably would have left it.
you dont have those kind of considerations to make with the facebook and twitter buttons