There are some CAN FD examples in the library.
You can also try this:
#include <FlexCAN_T4.h>
FlexCAN_T4FD<CAN3, RX_SIZE_256, TX_SIZE_16> FD;
IntervalTimer timer;
int SILENT = 9;
Best to get Classic CAN going first to prove the cabling is correct, then move to CAN FD.
The SILENT pin is to control the CAN transceiver. That is to disable the silent mode.
Your code is not right. You have not setup the baud rate and the SILENT pin.
Try this code:
#include <FlexCAN_T4.h>
FlexCAN_T4<CAN3, RX_SIZE_256, TX_SIZE_16> can3;
IntervalTimer timer;
int SILENT = 9;
...
If you are using two boards like that shown in the photo then you need the 120R terminator. Solder a 2way header pin to JP1 and insert a jumper to enable the terminator.
Later if you are going to connect to your car...