The PSRAM page says that each chip can be 8 megabytes, but I was wondering if I could switch it out with a 16 or ever 32 Mb chip instead.
If someone made such a chip you probably could. But as far as I know, today 8MB is the largest chip. ApMemory has said they have no plans to make a larger version, and all other SPI / QSPI RAM chips made by other companies (at least all the ones I know) are much smaller.
But if such a hypothetical chip were found, we'd almost certainly need to edit the software to make it work. It certainly would not be automatically detected and "just work". The existing code looks specifically for the ApMemory PSRAM chip.
Also the Teensy 4.1 page doesn't give a limit on how big the flash memory chip can be, the image given added 128mb NOR flash. Does this flash chip have to be NOR or can I add a NAND flash chip instead?
LittleFS supports both NOR and NAND flash. Bigger is not necessarily always better. Larger NAND flash chips have larger minimum erase size and larger minimum page write size, so they tend to be less efficient for storing small files. The list of supported chips can be found in the readme file.
https://github.com/PaulStoffregen/LittleFS/blob/main/README.md
I believe Winbond W25Q01JV*Q is also supported, but wasn't (yet) added to the readme file.
If you stray from the list of supported chips, you will have to edit the LittleFS library code!
I was wondering if this is specified by the Teensy bootloader on chip or is it a limit of the MIMXRT106 chip.
The bootloader has nothing to do extra external chips. It only programs the main flash memory.
Regarding the size limit, the FlexSPI hardware does impose an upper limit. I believe it is 248 MByte, but it's been a while since I've played with the FlexSPI config. If you use the 2 Gbit NAND chips, you might get slightly less than the full chip's 256 MByte capacity. As I recall, the hardware has an internal limit of addressing 256 MByte and allocating 8 Mbyte for the PSRAM (even if you use only the flash chip) takes away from how much of the flash can be accessed.
If connected by slow SPI rather than fast QSPI (soldered to the bottom of Teensy 4.1), I believe SPI could theoretically access up to 4 Gbyte. Write and erase speed are about the same, but reading is much faster with QSPI / FlexSPI.