trouble

I ran into some trouble today when pulling the coil pack off the spark plug underneath the gas tank. The plastic part that goes down into the engine came off fine but the rubber housing remained stuck to the spark plug, I assume. I tried pulling it out off the spark plug with needle nose pliers...

I had to use a flat head screw driver to push the locking pin back in order to remove the panniers. Am I doing something wrong? I followed the manual but they would not come off simply by turning the key and pulling on the handle. Any advice?