You could (it would probably be the best option IMO), but you'll need to source shorter bolts. Or just coat the inner nuts in ACF-50.
I intended taking mine off too but the Givi rack support is designed for them to be in place, so without fabbing up spacers I couldn't mount the topbox with the rack removed..
Luckily it doesn't stand out too much with the panniers removed.