I have tried multiple ways of uploading to a Teensy 4.1 headlessly from a Raspberry Pi using the Arduino IDE over VNC, arduino-cli & teensy_loader_cli.

This is the command I use with teensy_loader_cli to upload (.hex file is generated using arduino-cli):
teensy_loader_cli --mcu=TEENSY41 -w blinky.ino.hex -v -s
I am using the -s flag for the Teensy to be rebooted automatically, thus preventing me from manually pressing the button on it.
However, the command sometimes (more than half the time) fails to upload and says I have to press the button even though I have used the -s flag, and other times it says that it could not find any Teensy attached to the Pi.

When these errors occur, the red LED is usually ON on the Teensy and there is no ttyACM0 in the /dev/ directory. Even rebooting the Pi does not do anything sometimes, only uploading to the Teensy using Arduino IDE seems to reset the state of the Teensy such that /dev/ttyACM0 appears once again (this too does not work all the time).

I figured that it must be a matter of getting back /dev/ttyACM0 remotely that would allow uploading to take place once again with the same command.
Is there any way to use teensy_loader_cli or anything else to achieve this?