Just checking- have you been resampling your files before passing them to your HE-AAC encoder? General-purpose codecs like HE-AAC or Vorbis, since they've been tuned for music, will avoid resampling so as to preserve the quality of music's high frequencies, but for speech recordings those higher frequencies are just noise, and removing them allows the encoder to spend more bits on things that matter. Depending on your hearing and preferences, the sample rate sweet spot for straight speech could be anywhere from 12kHz to 24kHz.
Is there any other substantially more efficient speech codec ?
What software player (Windows) can you recommend ?
What considerations should I not overlook if I want to keep the files for life (I'm still young :-) ?