Understand that I'm just guessing here -
If the speaker is front-loaded (mounted to the outside of the baffle), try taking it out and looking around for a couple of screws.
Does it seem to be stuck on just one side? Front edge, back edge, somewhere specific?
Hope this helps!