I tried Internet Friendly Media Encoder, x265vfw, Avidemux and Handbrake, they either don't support Main10 or with very unpredictable 2-pass bitrate/filesize. Can someone recommend a good frontend? Thanks.
I don't do much video encoding but maybe this frontend would help ? http://x265.ru/en/baka-encoder/ (Note: I don't use it myself, so I am just guessing) ... anyway, you'd rather ask questions related to video on http://forum.doom9.org/ IMHO
StaxRip - the most up to date, user-friendly and multifunctional (you can encode to many other formats) frontend for HEVC in x265, also supports hardware HEVC encoding using NVENC.
For x265 it has all the switches, profiles and presets. Can do 10 and 12-bit encodes using quality or multipass setting.
Try this one too http://www.selur.de/ it's my favorite and the one I use for x264 (IMHO x265 is too slow at settings where is beats x264 qualitywise)
I'd personally never recommend Hybrid to anyone. :) Its interface is an absolute garbage and it tends to be encoding slower that it normally has to for some reason.
Another good frontend, which I'd use if there was no StaxRip is Xmedia Recode:
Thanks. Will try them later.