The chip has a low accuracy 32 kHz oscillator which is supposed to be automatically used if no RTC crystal is present. So at least in theory, it should work.
As a practical matter, all software for Teensy has been developed and tested with a real RTC crystal, so there's a chance some software may depend on it in unexpected ways. Might be wise to desolder the crystal from a Teensy 4.1 and test your program before committing to a custom PCB.