I found another project where someone tried to use a key matrix as joystick buttons, but he wasn't able to clear the keys after the press. I thought the case statement was a little heavy handed as well, so I went with a for loop, and haven't successfully gotten the system to recognize a keypress. I could wire these up to pins, but I've done two, which meant soldering about 150 wires with dupont connectors on the other end, and thought using a 5x8 matrix would be much, much easier this time around...

Current, non-working code. I'm new to this (despite having bought five 3.6s in the last two weeks to get this project done). Two of the boxen have been built the hard way, three to go.
Code:
  #include <Keypad.h>

  const byte ROWS = 4; //four rows
  const byte COLS = 4; //four columns

  byte rowPins[ROWS] = {0, 1, 2, 3}; //connect to the row pinouts of the keypad
  byte colPins[COLS] = {29,30,31,32}; //connect to the column pinouts of the keypad

  char keyMap[ROWS][COLS] = {
    {1,2,3,4},
    {5,6,7,8},
    {9,10,11,12},
    {13,14,15,16}
  };

  Keypad kpd = Keypad( makeKeymap(keyMap), rowPins, colPins, ROWS, COLS); 

void setup() {
    Joystick.useManualSend(true);
}

void loop() {

  for (int i=0; i<LIST_MAX; i++)
  {
    if ( kpd.key[i].stateChanged )
    {
      if ( kpd.key[i].kstate == PRESSED )
      {
          Joystick.button(i+1, 1);
      } else {
          Joystick.button(i+1, 0);
      }
    }
  }
  Joystick.send_now();
  delay(5);
}