Alright, after following your pin advice it still didn't seem to work. I also grounded the other 'sd' ports on the adafruit display, also without luck.
Then, however, I changed the CPU speed in the Arduino IDE from 96 MHz (optimized) to a regular 72 MHz and the display worked!
Any thoughts on this?