IIRC there was some timing noise/jitter introduced with USB on the original VGA - with T_3.6 - that went into the Teensy64. But it still allowed ALL of C64 to run there. Initially Frank did a private USBHost it seems to disable/alter something - but scan at that old github looked like he went to PJRC release?
Timing and USB different of course with 1062 and faster USB - for better or worse.
I enjoyed working along with FrankB - good to have somebody else on board. And as cool as the T_3.6 was back then ... T_4.1 with PSRAM and more other is a step up.