There are some CAN FD examples in the library.
You can also try this:
#include <FlexCAN_T4.h>
FlexCAN_T4FD<CAN3, RX_SIZE_256, TX_SIZE_16> FD;
IntervalTimer timer;
int SILENT = 9;
Best to get Classic CAN going first to prove the cabling is correct, then move to CAN FD.
The SILENT pin is to control the CAN transceiver. That is to disable the silent mode.
Your code is not right. You have not setup the baud rate and the SILENT pin.
Try this code:
#include <FlexCAN_T4.h>
FlexCAN_T4<CAN3, RX_SIZE_256, TX_SIZE_16> can3;
IntervalTimer timer;
int SILENT = 9;
...
If you are using two boards like that shown in the photo then you need the 120R terminator. Solder a 2way header pin to JP1 and insert a jumper to enable the terminator.
Later if you are going to connect to your car...
You can not connect CAN RX1 to CAN TX1 like you did. You need to connect a CAN transceiver such as the MCP2562 to pin 23 and 22 then the CAN_H and CAN_L to your CAN network.
You also need another node on the CAN...