If you aren't using a credit card, should you cancel it?

I have a few credit cards that I've used only a couple of times. I will most likely never use them again. Is it better to leave them open and let credit build or close them? I don't want to close them because I heard somewhere it can lower your FICO score.