Yes, generally speaking, you should include VAT on every price including delivery - especially if you're selling to the general public. I always like to make it clear by stating "(inc VAT)" next to each price.
The only exception is if you're specifically targeting business customers, then it might be an idea not to include it.
Some sites allow you to toggle whether prices include or exclude VAT. But anyway, what's important is that it is made clear. I agree, sometimes I've felt a bit cheated when I've assumed that VAT is included only to get to the checkout to find that the price didn't include VAT.
The only complication is if you're selling outside of the EU - you should not charge VAT on these orders! Which is a bit of a headache to code and also to show in your book-keeping...