Yes, you are missing SYS_FS and MCLK_FREQ, also in CHIP_CLK_CTRL which sets the clock, this in combination with the RATE_MODE determines the sample rate, see datasheet table 18. I think you'll need to reduce the MCLK frequency sent to the chip to get the lowest sample rates though.
But why not...