Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O (Read 125965 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

I'd like to announce the opening of the public listening test of Apple AAC, Opus, Ogg Vorbis and MP3.
Three years have passed since the last multiformat listening test. The development of Opus was very active in 2013, and Apple have refined their AAC encoder in the last three years.
Today it's time to see their advance in this test.

The following codecs will be tested:
  • AAC iTunes 11.2.2
  • Opus 1.1
  • Ogg Vorbis aoTuV Beta6.03
  • MP3 LAME 3.99.5
  • AAC FAAC v1.28 (Mid-Low Anchor)
  • AAC FAAC v1.28 (Low Anchor)

The homepage of the test:
http://listening-test.coresv.net/


There are 40 samples. From this test, the webpage automatically allocates listeners to odd group and even group, and each listener tests odd samples or even samples.
The Test ends on August 30th, so please don't hurry. It's better to do fewer samples per day and avoid a fatigue.

Every single result is helpful. Your participation will be very valuable.

Best regards;
Kamedo2

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #1
It's becoming a bit obvious to me that these codecs are passing my ability to ABX at 96kbps. I've tried 3 samples so far and 2 of them I could only find the low-anchor. Even the mid anchor eluded me. On the 3rd I was able to find the low and I assume mid anchor but the mid-anchor is basically no worse than the one other encode I found with a barely perceptible artifact. The rest are all 5's to me. I'm not sure if it's worth me submitting results showing virtually all the tested encoders transparent to me or not. I suppose it's a data point voting for how good these encoders are getting. I'll work through some more after a break..

The low anchors are obvious, but the rest of the ones I've tested I'd never know they weren't the originals in normal listening where I'm not focusing on picking out flaws.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #2
A big Thank You to Kamedo2!
Without him this test would be impossible. While the test is prepared in the best traditions of Hydrogenaudio he has gave a new shape to it. 
This time the test will be conducted completely by Kamedo2 and Steve Forte Rio. I don't have an access to mail to recollect results anymore. 


I'm not sure if it's worth me submitting results showing virtually all the tested encoders transparent to me or not. I suppose it's a data point voting for how good these encoders are getting.

If You try your best and still rank 5.0, your result is still valuable. I would suggest to go (very!) slow. Not more than 3 packages per day and with a big breaks between each other. An environment should be relaxing just when someone listen to music. You can do as slow as one package per day.
Feel free to start to submit even one single result right from now.  It will be better for You to be informed that everything is going good and for an organizators who receive and check your results.


When I was testing packages I've noticed that while generally codecs do damn good job there are still some difficult samples where they can have some quite audible artifacts of compression.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #3
It's just been really hard to even find those subtle little parts that reveal the encode, and even when I find them they are miniscule. It could just be the 3 samples I tried today. But, based on these 3, props to the codec developers! 10 years ago I wouldn't have dreamed it would ever be this hard to ABX codecs at 96kbps.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #4
Some at ~96 are outstanding yes, HE and Opus even at ~64, I gave up long time ago trying. Someone told me "if you need to try hard for more than five minutes it's not worth it", he was right. MP3 is easily recognizable at ~96 though.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #5
MP3 in this test is encoded VBR around 130kbps so even that one ca be hard to pick out. In the one sample I did today where I was able to ABX one actual tested codec beyond the low and mid anchor, I'm guessing it was probably the mp3, but even it was really subtle.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #6
Indeed, this is becoming the "hard to ABX" range 
Done four samples and in all of them I've only been able to discern the anchors (at the first listen), but the rest elude me. Tried first with in-ears (Samsung's phone), and switched to headphones (Sony MDR) but still no luck.
Will try some more tomorrow, and decide if i really submit these "results".


Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #7
I've decided I'm only submitting results if I can at least discern both anchors. The low anchor on all the samples so far is so obvious it hurts, but even the mid anchor on many of them is getting rough to discern.

The effect of this might be a data point in itself. If a sample receives fewer results than others it may be taken as evidence that it's a sample where even the mid anchor was encoded so well as to be transparent or otherwise indistinguishable from the tested codecs..

My worry is that since the mid anchor appears to be so close to transparent we may end up with whole suites of results being invalidated and not used due to mistaking the mid anchor with one of the tested codecs. I'll be honest and say the one sample yesterday I was able to find the low, mid and one other ABXable, it was a total guess which was the mid anchor and which was the tested codec. The tell-tale artifacting I managed to find on both was that subtle. I ended up deciding one was slightly more annoying than the other and picked that as the mid anchor.

But it begs the question of how draconian the invalidated results are going to end up being:

* If you rank the mid-low anchor at 5.0, your result of the sample will be invalid.
* If you submit 25% or more invalid results, all of your results will be invalid.

On 2 out of 3 samples I've tried, I failed that 1st test. Since I've done 3 samples and could only find the mid-anchor on one of them, if I had submitted all 3 results (and never submitted any more) I would have had all my results invalidated.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #8
youlord,

Can I ask You whether You perform test using headphones or speakers?

This kind of test would be very difficult using just speakers.

Thank You.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #9
Sennheiser HD-280 pro headphones. I listened to the sample I was able to pick out afterward on a good stereo with 3-way speakers and was unable to hear the artifacts I picked up via headphones. I wouldn't even try doing these tests on speakers.

It could just be the 3 samples I tried yesterday.. The 2 where I completely failed to pick out anything but the low anchor were speech. I'll try a few more tonight.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #10
Yay it's on  Thanks for making it happen! This will be my first listening test i'm participating in so i might have to come back and ask things about ABC/HR.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #11
Sennheiser HD-280 pro headphones. I listened to the sample I was able to pick out afterward on a good stereo with 3-way speakers and was unable to hear the artifacts I picked up via headphones. I wouldn't even try doing these tests on speakers.

It could just be the 3 samples I tried yesterday.. The 2 where I completely failed to pick out anything but the low anchor were speech. I'll try a few more tonight.

Your headphones are fine. Maybe You need some warming-up on some other test samples of this test. Both low and middle anchors have quiet noticeable low pass (7,5 kHz and 10,4 kHz respectively) besides of artifacts of lossy compression. It's to expect that people would hear that.


Yay it's on  Thanks for making it happen! This will be my first listening test i'm participating in so i might have to come back and ask things about ABC/HR.

Sure. You can also visit IRC channel #Hydrogenaudio on the server irc.freenode.net


Indeed, this is becoming the "hard to ABX" range 
Done four samples and in all of them I've only been able to discern the anchors (at the first listen), but the rest elude me. Tried first with in-ears (Samsung's phone), and switched to headphones (Sony MDR) but still no luck.
Will try some more tomorrow, and decide if i really submit these "results".

Yes, it's quite difficult test. The last public AAC test was also difficult for people but not impossible. At least some samples are easy to test.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #12
That may explain it. My hearing rolls off at about 12k or so (last I checked) due to 30+ years of very loud heavy metal and 17 years in a metal band. After you said the mid anchor is low passed at 10.5k I went back and relistened to some of the mid anchor samples I already identified and I can't hear the roll off of the highs on them. The low anchors are so bad it's easy to hear the low pass and the egregious artifacts. I've been identifying mid anchors solely by artifacts, and they have been pretty subtle. Subtle enough I've been forcing myself to ABX them well enough the test will disable the reference on the sample for me. I submitted 2 more results tonight where I found the low and mid anchor, and everything else is all 5's. I may be old and broken enough to not be useful in this test 

Of course my wife can't even pick out the mid anchor artifacts and she is 8 years younger than me and hasn't brutalized her ears like I have.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #13
I think it's important to submit 5.0 results.  If only successfully ABXed results were sent this would give wrong impression.
I also wouldn't care too much about the mid anchor.  Though there may be a chance that the test result be invalidated it's still a preferable situation over not sending results at all.
lame3995o -Q1.7 --lowpass 17

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #14
On the other hand, I have the gear (HD800s) but not the ears! Because my brain is a patzer! 

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #15
I think it's important to submit 5.0 results.  If only successfully ABXed results were sent this would give wrong impression.

Agreed.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #16
Isn't 96kbps good these days?  Not always 100% perfect, but blimey it's close.


Now, what is the story with dragging the test discussion away from HA, declaring all discussion on HA closed, saying the new discussion forum is "just a beginning", but then coming back to HA to solicit test participants?

While there's room on the internet for many audio forums, taking carefully design subjective testing away from HA and saying that's just the beginning worries me.

Cheers,
David.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #17
Could the next test include libvo-aacenc? I would be interested if this codec is as good as the rumors, or just another libfaac.

If you need an encoder, ffmpeg on windows has it (-v:c libvo_aacenc) its also now the default AAC encoder.

If it is any good I might start putting up some bug reports to software like OBS to replace their libfaac.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #18
Could the next test include libvo-aacenc? I would be interested if this codec is as good as the rumors, or just another libfaac.

If you need an encoder, ffmpeg on windows has it (-v:c libvo_aacenc) its also now the default AAC encoder.

If it is any good I might start putting up some bug reports to software like OBS to replace their libfaac.

libvo-aacenc is as bad as the rumors. Using high bitrate doesn't work for this encoder.
http://www.hydrogenaud.io/forums/index.php?showtopic=105959
http://www.hydrogenaud.io/forums/index.php?showtopic=102875

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #19
Oh thanks, looks like people have started ABXing it, last time I checked nothing had come of it, I probably should have searched first, my bad.

edit: though a multi-person test might be valuable.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #20
If somebody know where it can be useful to make an announcement  You're welcome to do it.
Here is one on doom9


Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #22
Could the next test include libvo-aacenc?

Well, as Kamedo2 and lvqcl have indicated that it's not exactly a high quality encoder. But, yes,  who knows?  In some future AAC tests...

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #23
If somebody know where it can be useful to make an announcement  You're welcome to do it.
Here is one on doom9

I am currently writing a Japanese version of the announcement and front page.

Public Multiformat Listening Test @ ~96 kbps [July 2014], AAC, Opus, O

Reply #24
  • AAC iTunes 11.2.2
  • AAC FAAC v1.28 (Mid-Low Anchor)
  • AAC FAAC v1.28 (Low Anchor)


This sounds interesting, thanks in advance for setting it up! However I cannot help but wonder about your selection of aac codecs:

Everybody and their cat (android, handbrake, ffmpeg (private compiles)) seems to be switching to the high quality fdk aac because itunes aac isn't that flexible and as closed source not meant to be integrated into 3rd party apps. While it of course is interesting to know how a real free aac codec like faac does, including the newest codec on the block would have been more meaningful for real world usage?

http://www.iis.fraunhofer.de/en/bf/amm/imp...dkaaccodec.html

Could the next test include libvo-aacenc?


This codec is very unflexible seems to do rather terrible even against ffmpeg's own experimental  aac, see here: https://trac.ffmpeg.org/wiki/Encode/AAC