A button that is visually larger than its clickable area is one of my biggest UI pet peeves.
Here’s a screen shot of an open-source project’s download page. Only the text is actually clickable. However, the green area changes from a dark green to a bright green when the cursor hovers over any part of green area. This creates the illusion that you can click on any green section and perform the desired action.
It’s not a huge deal, but at least once a week I’m caught up by something like this. I wonder if it’s worse for people who are less technically inclined. This is something that’s easily avoided and will definitely confuse some of your users. Don’t do it!