I actually think I found a fix. I updated to teensyduino 1.6 beta 5 manually. This includes an update to the ST7735 library which specifically adds the ST7796. I was able to successfully run the "graphicstest" sketch.
Did you ever have any luck with this? I've tried the "graphics test" sketch that comes with the teensy ST7735 library, just changed my pixels to 320x480 (also tried 480x320). It does the same thing you describe, shows some of the intended...