I am completely re-breadboarding the whole project, and drawing the circuit diagram as I go to post here.
I decided maybe the better option was to utilise the SPI1 and SPI2 ports on the Teensy 4, but now not so sure that will be a good idea.
First off, I didn't realise the pads for the SPI2 (34-37) were so ridiculously small. For a processor that will plug into a PCB (no room for a 4.1), I can't see any easy way of accessing those pins. I can only think of thin wires with a 4 pin plug on the end.
Secondly, I assume it will take some library adjustment to make the SD card utilise SPI2 and the flash to use SPI1 (The screen in on the default SPI pins).
I don't see how you tell the SD card routine to use a different SPI bus.
I will continue to tinker and then report back with the new circuit (if indeed, I find one that is better)