In general, it depends.
If you only use the basic devices (first I2C bus, first Serial port, first SPI bus, 10 analog input pins, pins 0..23 as digitial pins) then it should work fine.
If you want to do stuff with audio, it will need reworking:
- The audio I2S pins are completely different;
- Because of the I2S difference, you want to use the Audio shield revision C on the Teensy 3.x boards, and the Audio shield revision D on the Teensy 4.x boards;
- The Teensy 4.x does not have a DAC pin (digital to audio converter) that you can produce low grade sound on.
Using serial port 2 and 3 on the Teensy 3.x uses different pins compared to the Teensy 4.x.
As mentioned, the Teensy 4.x is not tolerant of 5 volt digital input.
The Teensy 4.x does not have an analog reference that would allow you do to analog reads with a lower voltage than 3.3 volts.
Some displays have special optimizations if you use certain pins on the Teensy 3.x microprocessors. The Teensy 4.x microprocessors have different optimizations, and different special pins. The displays should still work, but they might be slightly slower.
I've written two documents that touch on the differences. The first is an entry that I did in the unofficial Teensy wiki some time ago about the differences. The second is a google document where the first sheet attempts to compare all of the Teensy LC/3.x/4.x for uses of each pin: