Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: New Public Listening Test (July-August 2011) (Read 79907 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

New Public Listening Test (July-August 2011)

It’s time to discuss the conditions of new public test.
At least for now there won't be poll vote about possible codecs to test because it will be better to speak personally.
The polls can't tell us if the person had participated or want to participate in such tests.

Previous test at 64 kbps wasn’t difficult but neither very easy for everyone. So 96 kbps is a reasonably good bitrate for new test. 128 kbps have been already tested in previous tests.

In my opinion now we have 2 possible tests:
LC-AAC test  ~ 96 kbps
Multiformat    ~ 96 kbps

One will be performed during July –August of this year and another in next year.
I will strongly suggest to perform first LC-AAC test and only then go for Multiformat.
Because later it can turn into a lot of discussion of not right choice of AAC encoder for Multiformat test.
Second reason is that a lot of work has been already made on future AAC test

Samples and encoders have been alearedy defined in previous discussion.

LC-AAC encoders to test:
1. Nero
2. Apple true VBR
3. Apple iTunes constrained VBR
4. Winner between Winamp's Coding Tehcnologies and Divx (Pre-test)

See here http://www.hydrogenaudio.org/forums/index....c=77272&hl=
Only 4 codecs should be inlcuded (4 + low anchor). Generally too much codecs will lead to loss of listener's concentration.



Some members of HA (Garf, Chris, Alex B, /mnt ) are welcomed to be co-organizers or simply observers of this test to control e-mail  communication with listeners and other conditions (samples, bitrate verifications, packages etc.)

The public test will be started during the end of July, 2011. We have more than one month to discuss all conditions.

Please, share your thoughts here.

P.S. Also it will be nice if someone who knows HTML will help to edit some web pages.
P.S.2. Opinions of the listeners from the last public test will be especially valuable for me.

New Public Listening Test (July-August 2011)

Reply #1
Kinda Bump!

if it will be AAC public test then probably samples are pretty defined.  http://www.hydrogenaudio.org/forums/index....st&p=695576
The quality of the codecs doesn't change from that point.

Unless somebody has different point of view.


New Public Listening Test (July-August 2011)

Reply #2
I have always wanted to see a test of constrained vs true VBR on non "killer" samples.

I would love to participate. Now I have a pair of Sennheiser HD 600 and a Little Dot Mk IV which ought to help with critical listening.

New Public Listening Test (July-August 2011)

Reply #3
I'd love to see how faac stands up to current encoders. The development seems to be idle right now, but I'd like to know the current status in comparison to the alternatives, since I see it being recommended once in a while in #ffmpeg.
It's only audiophile if it's inconvenient.

New Public Listening Test (July-August 2011)

Reply #4
I would love to participate. Now I have a pair of Sennheiser HD 600 and a Little Dot Mk IV which ought to help with critical listening.

Nice.

I'd love to see how faac stands up to current encoders. The development seems to be idle right now, but I'd like to know the current status in comparison to the alternatives, since I see it being recommended once in a while in #ffmpeg.

FAAC 1.23.5 has been included in one of the public tests some time ago. http://listeningtests.t35.com/html/AAC_at_...est_results.htm
The current version is 1.28
I'm not aware if there are any substantial improvements since then.



The good news that probably we will see new Fraunhofer (FhG) AAC encoder in this test.
In this case we should redefine the samples because new codec will be included.

People, prepare to suggest your samples if it will happen. It won't be me who will choose samples.

New Public Listening Test (July-August 2011)

Reply #5
The good news that probably we will see new Fraunhofer (FhG) AAC encoder in this test.


Sounds interesting, i've never seen any tests from a Fhg AAC encoder.
"I never thought I'd see this much candy in one mission!"



New Public Listening Test (July-August 2011)

Reply #8
Though it must have been a really old Fraunhofer encoder. That test is from June 2003*. In 2005, Fraunhofer started rewriting its AAC encoder more or less from scratch, so a lot has changed.

The good news that probably we will see new Fraunhofer (FhG) AAC encoder in this test.
In this case we should redefine the samples because new codec will be included.

Why? Back in early 2010 I spent weeks digging through old HA test material as well as new material suggested by /mnt and others to
  • find samples which highlight different types of artifacts and aspects of encoding quality (see the link you mentioned above),
  • amplified or attenuated them so they won't clip upon decoding (a thing which had not been done in the 2011 64-kb test!),
  • limit them to 15 seconds to minimize listener fatigue (also not done in the 2011 test).

Bashing that entire test set would be a waste of resources. That being said, we could think about adding some samples from the 2011 test. Say, 5 samples to make it a total of 20. Pure spoken or sung vocals, for example, have not been considered in the 2010 test set. And we could think about replacing non-music/non-vocal samples such as applause by more musical/vocal items.

Chris

* at least that's what the file dates in the RARed comments say
If I don't reply to your reply, it means I agree with you.

New Public Listening Test (July-August 2011)

Reply #9
Chris,
You (Fraunhofer's developer) suggest to include your set of samples and include your Fraunhofer AAC encoder into test.

Is it what you want?

Let's put time limits to decide what codecs to test.
Starting from 1st of July no new competitor codec will be accepted to be included. Without exception.
Then we will discuss the choice of samples and settings.

Edit: Grammar

New Public Listening Test (July-August 2011)

Reply #10
Temporally removed.

New Public Listening Test (July-August 2011)

Reply #11
Couldn't iTunes "constrained" versus "true" VBR be done as a sub-test?  Obviously I'm not the most unbiased commentator for a listening test, but it seems excessive.  Unless it's something that the HA community truly doesn't know which is better.

New Public Listening Test (July-August 2011)

Reply #12
Couldn't iTunes "constrained" versus "true" VBR be done as a sub-test?  Obviously I'm not the most unbiased commentator for a listening test, but it seems excessive.  Unless it's something that the HA community truly doesn't know which is better.

Good point.

Although many people want to see comparison between these two VBR modes (as it's indicated in previous discussion past year) it will be difficult to compare them because true vbr doesn't produce the same bitrate as constrained VBR.

true VBR (--tvbr 45) - ~92 kbps
constrained VBR ~100 kbps

The list of candidate codecs for ~100 kbps test:
1. Nero
2. At least one Apple encoder (CVBR or TVBR) or two of them (?)
3. Fraunhofer
4. Winamp's Coding Technologies and Divx (Pre-Test)


Fraunhofer encoder will be publicly available on 30th of June.
All codecs except true VBR can produce 100 kbps  for this test.

I expect suggestions from community.

New Public Listening Test (July-August 2011)

Reply #13
First, it's unacceptable to have the set of the samples provided by the developer of one of included competitors.
Nobody won't tolerate it.  If these samples would be included later everybody will accuse me to be not impartial.

Second, first 50% of samples for previous test were taken from Sebastian Mares's 64  kbps and very few from Gabriel 48 kbps test and second 50% were new. First and second halfs have shown the same results.

You can verify for yourself.


I'm standing on right place.

P.S.The choice of samples, codecs, conditions and bitrate verification  are public.

New Public Listening Test (July-August 2011)

Reply #14
I would very much like to see a multiformat 96kbps test that includes both Nero AND Apple (Quicktime) LC-AAC encoders.

New Public Listening Test (July-August 2011)

Reply #15
Ok, your vote counts.

Let's see if there will be more people who are interested in multiformat.






New Public Listening Test (July-August 2011)

Reply #16
As I understand it, transparency for the latest batch of codecs is believed to be somewhere between 128 kbit and 64 kbit, thus the proposed testing at 96 kbit.

As we try to resolve this further, we should also try to deal with the fact that outside scientific and developer circles, transparency at 128 kbit is not well appreciated. Is there some testing we can do that will bring a greater appreciation?

Many people base their disdain for lossy coding on the performance of early encoders. Can we, to show how far the art has progressed, compare some crusty but widely-used encoder at high bit rate to newer ones at lower bit rates?

New Public Listening Test (July-August 2011)

Reply #17
Many people base their disdain for lossy coding on the performance of early encoders. Can we, to show how far the art has progressed, compare some crusty but widely-used encoder at high bit rate to newer ones at lower bit rates?

Good suggestion. I believe that it is mp3 that is still the main (and maybe the only) choice outside of scientific and developer circles.

New Public Listening Test (July-August 2011)

Reply #18
And at the same bit rate, there's quite a range of MP3 encode quality depending on encoder vintage and settings. Most users assume they get the same quality if they use the same file format with the same bitrate.

New Public Listening Test (July-August 2011)

Reply #19
First, it's unacceptable to have the set of the samples provided by the developer of one of included competitors.
Nobody won't tolerate it.  If these samples would be included later everybody will accuse me to be not impartial.


"My" 2010 testset. . . "Igor's" 2011 testset (and/or who suggested the item for "my" testset)

AngelsFallFirst. . . . not present, but very similar to Sample01
Robots . . . . . . . . Sample05 (the 1978 un-remastered version, 2009 remaster suggested by /mnt)
Kalifornia . . . . . . Sample10 (longer than Kalifornia, a.k.a. Fatboy, suggested by IgorC)
Linchpin . . . . . . . Sample17 (longer version, suggested by /mnt and IgorC). FallOfLife not present
Waiting. . . . . . . . Sample18 (suggested by IgorC)
Applause, Harlem . . . not present, applauses from at least 2004, but I suggested to remove it from my set. (Applause was supported by rpp3po)
Can't Wait . . . . . . not present, item from 2004
FallOfLife . . . . . . not present, item from 2004
Rumba. . . . . . . . . not present, item from 2004
SQAM Selection . . . . not present, items from 1988, publicly released 2008
Memories . . . . . . . not present (suggested by naturfreak)
Hancock. . . . . . . . not present (suggested by IgorC)
Creuza de Mä . . . . . not present (suggested by IgorC)
Trumpet. . . . . . . . not present (suggested by halb27)
BerlinDrug . . . . . . not present (suggested by Bryanhoop)
Girl . . . . . . . . . not present (suggested by IgorC)
Ecstasy. . . . . . . . not present (suggested by IgorC)

P.S. The choice of samples, codecs, conditions and bitrate verification are public.

Sorry if I missed it, but where was the public selection and discussion of the test items for your 2011 test at 64 kbps?

This will be my last post in this thread or any other related to this public test.

Chris
If I don't reply to your reply, it means I agree with you.

New Public Listening Test (July-August 2011)

Reply #20
The order of events:

1. The selection of the encoders.
2. Only then we should choose samples (if they are new competitors).

It doesn't matter who submit the samples.

New Public Listening Test (July-August 2011)

Reply #21
The order of events:

1. The selection of the encoders.
2. Only then we should choose samples (if they are new competitors).

It doesn't matter who submit the samples.


So, is the Fhg encoder in or out?


 

New Public Listening Test (July-August 2011)

Reply #23
IgorC, and anyone else, are free to point out that a certain cause of action might disproportionally benefit a certain encoder. But you can expect every developer to make recommendations that help rather than hurts their cause. This is natural. If the process is open, the community here will identify such situations and call to correct them. This also ensures that in the case you, the organizer of the test does something like that, it is identified and you can correct it so the test stays neutral.

I wasn't entirely happy with how the samples in the past test were selected, but told you I thought it was reasonable enough that people would accept it. You're free to observe how the post-test discussion regarding that subject went. There is some irony you're now upset at another persons recommendation of samples.

If you attack a codec developer, the result is entirely predictable: they will walk away and point to the attacks as evidence the test was biased against them. (And note this will happen regardless if the end result ends up being biased or not) This takes away from the people who help with the effort to organize the test, and of the people who spend time on taking it.

If I'm a person interested in taking the test, and I come to this thread and see the organizer attacking one of the participants, what do you think I'm going to do?

I'm going to leave this thread open because I'm interested in what people want to see tested. But I'm also going to moderate it if the discussion is not conducted in a civilized manner.

New Public Listening Test (July-August 2011)

Reply #24
If the process is open,

Indeed, yes, it's open

they will walk away and point to the attacks as evidence the test was biased against them.

The choice of conditions (samples, bitrate, settings) will be made by all members (except some obvious restrictions as it has happened) . So there is no chance to get it biased way.

Yes, I perfectly admit that I was harsh. I won't repeat such thing again.  But this test stands for human values not commercial interests.



Guys, this test will be for You and made by You. You will choose all samples, You will choose all codecs, You will  choose all settings, You will control all conditions and You will test it.