Blu-ray has some really weird restrictions on the encoded bitstream; just putting the video into an MP4 file can cause incompatibility even if you do everything else correctly. If you want to be sure your disc will play correctly on the greatest number of Blu-ray players, you have to use x264 by itself to produce a raw H.264 bitstream separately from the audio and then combine them with your muxer.

There used to be a website that included complete x264 command lines for Blu-ray encoding; you can still read an archived copy.

FFmpeg can output PCM, you just need to select the bit layout when you specify the codec. For example, "pcm_s16le" will give you 16-bit PCM suitable for WAV files (which you can then feed to your muxer). When working with PCM, you'll probably want to use WAV files instead of raw PCM.

The difference between "ac3" and "ac3_fixed" is that "ac3" uses your CPU's floating-point math capabilities and "ac3_fixed" does not. Either one will work equally well for you (though I'm not sure how good FFmpeg's AC3 encoder is).

Any idea which of these is most likely to transcode with x264?

Any idea which audio encoder/frontend to use?

Muxer?

I suppose some guide as to which bitrates actually work is also going to be helpful. I was assuming there's no limit aside from the top end in the spec, but apparently it's not continuously variable.

Maybe there's some limit on the video bit rate choices too?