Why are there (big) differences in quality between various AAC encoders? What can these different encoders do to increase sound quality? Doesn't AAC format itself specify all the details of the encoding process?
Not all audio codecs are designed in the same way.
In the concrete case of MP3 and AAC, what is standarized is the format and the tools that the decoder has to know in order to decode the stream.
A valid stream does not need to use all the available tools, and some tools are designed for some subsets and not for others. ( the "LC" part of AAC-LC means the "Low Complexity" profile, which has ended being the only globally used profile).
So the first part about the quality of the encoder is the amount of tools that it supports for the profile.
Next, the decoder is in charge or deciding when it is appropiate to use a tool and when it is not. This is not necessarily a true/false scenario so the encoder has to be "trained".
At last, there is the psychoacoustic aspect of lossy codecs. Modern codecs are intended to generate an output that is transparent (meaning it is not perceived acoustically as being different). This means that the encoder does not try to reproduce the bits of the signal as they originally were, but instead to fool the ear. For this, a knowledge of how the ear works is needed, and this is translated into an algorithm so that the codec decides to give more priority to some parts of the audio than others.
All these things are what cause big differences between an encoder that simply tries to produce a valid stream and another that has been designed and tweaked to be of high quality.
https://hydrogenaud.io/index.php/topic,108954.msg909963.html#msg909963
Doesn't AAC format itself specify all the details of the encoding process?
No, just the decoding process usually.
Specifying what the encoder had to do would be hard since no one knows what the best implementation would look like.