Coaxial / SPDIF only has enough bandwidth for DVD-era Dolby Digital 5.1 and DTS 5.1, not enough for Dolby TrueHD / Atmos or DTS HDMA.
DTS HDMA will be downgraded to DTS Core for sending to SPDIF. You can still get 5.1 sound.
There is no easy way to send TrueHD in MKV via SPDIF and get 5.1 sound unless you use software like ffdshow to transcode the TrueHD to Dolby Digital 5.1. TrueHD in original BDMV is an exception, some player can extract the Dolby Digital 5.1 inside the BDMV and send it to SPDIF.