Based almost entirely on this post, a typical Teensy 4.1 might be expected to run at about -20ppm. Given that Tascam are pretty reputable, I'd be surprised if their kit is dramatically worse. A one second difference would be 555ppm, well outside...
If "a few seconds" is "three", then the rate difference is 0.1667%, or 1667 parts per million. That is a more than I would expect from a pair of crystal-controlled clocks but not unheard-of. The project I work on is sensitive to timing precision...
You may want to consider what should happen if either or both of the other computers are transmitting during intervals when the control switch is not HIGH. I have not looked into Teensy serial buffering in a while so I cannot offer clear...