Here is a linear chirp (sine with increasing frequency) from 1 Hz to 24 kHz with amplitude 0.8 and its variations with brown, pink and white noise added (noise amplitude is 0.02). NeroAAC, Musepack and Opus 1.0.2 + 1.1 alpha failed on it even at their best quality. Vorbis and MP3 not failed. A lot of files are at these links (encoded samples + spectrograms):
Lossless (http://www.mediafire.com/download.php?3l3oqwoslclicai)
MP3 (http://www.mediafire.com/download.php?v51964jgfhd3x69)
Musepack (http://www.mediafire.com/download.php?dotp9m16b2lp7x5)
NeroAAC (http://www.mediafire.com/download.php?5owspnx0vobk0wr)
Opus 1.0.2 (http://www.mediafire.com/download.php?uej652lcsbbliv4)
Opus 1.1 alpha (http://www.mediafire.com/download.php?964bpv0q3rqxawq)
Vorbis (http://www.mediafire.com/download.php?1u3mr5t9gse68d9)
Most codecs are tuned for actual music, not pure tones, so they probably handle samples like this really poorly. I suspect MP3 does so well just because lame (which I assume you used) is such a mature encoder.
Most codecs are tuned for actual music, not pure tones, so they probably handle samples like this really poorly.
Well, electronic music may contain whatever signals possible, there are thousands of various plugins/generators etc...
Could you try it with QAAC, see if it handles it better than NeroAAC or worse.
For what it's worth I've performed some checks with this sweep and some AAC encoders:
Uncompressed Chirp0to24kHz.flac to WAV using flac -d
FYI and for some reason, the resulting WAV correctly plays in Audacity and Ocenaudio but not in Wavosaur and WaveShop, but this is something else.
Encoded the WAV to AAC using:
fdkaac 0.2.0 with libfdk-aac 3.3.3
FhGAACenc (20120624) using Winamp 5.6.3 encoder (20120620)
Nero AAC codec 1.5.4.0
All encoders were used directly from the command line with their default settings, only the quality setting was used.
I started with fdkaac -m 3
(http://i.imgur.com/TWHQkJT.png)
file size 87 960 bytes, overall bitrate 29.3 kbps
Then I aimed to get about the same overall bitrate with the other encoders but I had to use quite high quality settings to achieve this:
FhG --vbr 5
(http://i.imgur.com/XU4OS8g.png)
file size 87 611 bytes, overall bitrate 29.2 kbps
FhG --vbr 6
(http://i.imgur.com/1eZvPXA.png)
file size 87 719 bytes, overall bitrate 29.2 kbps
Nero -q 0.83
(http://i.imgur.com/pcGFjFC.png)
file size 86 470 bytes, overall bitrate 28.7 kbps (-q 0.84 would give too much bitrate for comparison, over 30.2)
Of course a sweep is not music but I find interesting to see the strategies used for this particular case.
The use of artificial samples to rate codecs is ok as long as ABX hearing tests are still involved. Rating deviations in a Time-frequency diagram as audible/inaudible without ABX is risky. The example softrunner showed for Opus was "obvious enough"(the line was cut = silent parts in the uncompressed WAV), but the deviations you're showing may not be audible at all. This is completely independent from the choice to use musical/unmusical samples.
With spectograms you are still allowed to draw conclusions about the technical nature of a encoder (algorithms used, size of transforms, etc.). For example I would expect transform codecs like AAC Vorbis MP3 to show different anomalies than nontransform codecs like ADPCM, or pure speech codecs. Conclusions about the technical nature of a codec are only interesting if they are not already published, which is unlikely.