Here is the board you are dreaming of...
https://oshpark.com/shared_projects/nYL18hj4
This connects a Teensy 3.x or 4.x to any one of the BuyDisplay displays - both RA8875 or RA8876. It connects the I2C bus for a capacative touchscreen. (If you have a resistive touchscreen, then it's probably accessible through the RAiO chip's interface.) If you use one of the "big" Teensys, they will overhang the board.
I did not put a lot of connections on the board. The goal was to make it small. I put some flat-cable (FFC) connectors on to add some satellite boards which hold buttons and encoder knobs. Those boards are not published publicly yet. Because I wanted to experiment, I've put a FFC connector on to hook up the T4.0 SD connector. You can put an SD card on this board or you can FFC over to the BuyDisplay board's SD card FFC. (Note you can't directly plug a T4.0 FFC to the display board as the pins are different.) For the T4, this gives you access to a second SPI port, if you didn't want to use a memory card but you did want another SPI input.
Then, because there was space, I put footprints on for two QWIIC (SparkFun) connectors. Add sensors or knobs, buttons or even a keypad that way. Or use long breadboard headers on the Teensy and hang jumper wires on the back to connect to whatever you need.
Attached is a PDF schematic so you know where all the pins go to.
I'm going to look into using the T4 parallel bus. I would love to get 8 or 16 bits flowing at 30-50MHz. That's enough for real-time video. Unfortunately the Teensy 4.1 doesn't expose the correct pins for the NXP chip's native LCD interface.