Even Teensy LC would be able to do this at about the same speed. If this is all you need to do, there's no need to spend extra for a 3.2 or 3.6.
For Teensy's part in this system, you can expect an average of about 1 ms and a worst case of approx 2 ms. These times are pretty much entirely due to the USB protocol and how the USB host controller chip in your PC schedules the USB bandwidth.
Something to keep in mind is Teensy isn't the only part of this system. On the PC side, you've got 1 program sending bytes to the COM port and some other software responding to mouse clicks. Unless you've gone to very special steps to run these programs, they will probably run with the normal user level scheduling of whatever operating system you're using. If you're using Microsoft Windows, the OS scheduling latency can vary from almost zero to 16 ms. Obviously there's nothing Teensy can do about that... but I can at least mention it, so you can look into that part and hopefully achieve your low-latency goal.