Had the same issue with my Roccat Leadr both with and without Swarm installed. Game would slowly drop from steady 145 fps to 80 until I unplugged the mouse, then it would instantly jump back up to 145 fps. Tried every single usb port on my pc, tried different usb drivers, tried different versions of Swarm. Same result with whatever I try.
Only happens in APB as well, no problems in any other game.