Skip to main content

Topic: Public, Multiformat Listening Test @ 64 kbps - OPEN (Read 87567 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • vinnie97
  • [*][*][*][*]
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #125
Sebastian, can you tell me if I rated any references?  Results should've arrived from vince dot hart at gmail.  I fear the answer.

Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #126
The key will be available after the test is over so you can check yourself.

People, about two hours left!

Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #127
Hmm... I could use a bit more results for the last 10 samples.  I am extending the test one last time until 15th since I expected a bit more results to arrive shortly before the test ends. Maybe Guru can test, too (BTW, received two mails from you - thanks!).

  • naylor83
  • [*][*][*]
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #128
Come on people! If you feel geeky doing the tests, remember it's all in the name of science!
davidnaylor.org

  • Alex B
  • [*][*][*][*][*]
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #129
Also, even if you are not personally interested about this bitrate range this a good opportunity to practice with relatively easy samples. A future higher bitrate test will be much more difficult for the testers. We have recently tested 48 kbps and now 64 kbps. Possibly the next multiformat test will be 96 kbps. I also look forward to see the discussed 128 kbps MP3 test happening sometime soon.

  • Whelkman
  • [*][*]
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #130
Also, even if you are not personally interested about this bitrate range this a good opportunity to practice with relatively easy samples. A future higher bitrate test will be much more difficult for the testers. We have recently tested 48 kbps and now 64 kbps. Possibly the next multiformat test will be 96 kbps. I also look forward to see the discussed 128 kbps MP3 test happening sometime soon.

This is why I joined. 48 kbps is unrealistically low and thus uninteresting to me right now, but I'm having problems as early as 96 kbps. A new 128 kbps test may be more challenging than many are expecting. 64 kbps is just the right spot where you might be fooled some of the time, making it interesting for testing purposes.

Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #131
Well, in case anyone is interested, the next tests will be:

MP3 at 128 kbps
AAC at 80 kbps (featuring both LC and HE-AAC)
Multiformat at 80 kbps with winner of the AAC test vs. other codecs
And finally maybe a 32 kbps and a 96 kbps test, but these have very low priority
  • Last Edit: 12 August, 2007, 02:01:58 PM by Sebastian Mares

  • Alex B
  • [*][*][*][*][*]
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #132
Well, in case anyone is interested, the next tests will be:

MP3 at 128 kbps
AAC at 80 kbps (featuring both LC and HE-AAC)
Multiformat at 80 kbps with winner of the AAC test vs. other codecs
And finally maybe a 32 kbps and a 96 kbps test, but these have very low priority

It is good to see that you have plans for several tests since no one else has been interested about conducting listening tests.

The 128 kbps MP3 test will be interesting because LAME 3.98 should be soon ready and FhG has a new version too.

Personally, I think 80 kbps is quite near of 64 kbps and 96 kbps would be a better next step for a multiformat test, but I suppose you have carefully thought about the options and followed the ongoing HA discussions more keenly than I.

It would be good to keep the tests flowing in a steady stream and not allow as long breaks as we had now. I think more testers would be willing to participate if the test discussions are constantly kept alive. For example, if a test is not possible in the scheduled time frame because of codec development reasons you could start the next test from the list instead.

Also, I wonder if the test presentations, instructions/help/faq and the results presentations could be developed further. It would be nice to have an attractive and carefully organized web site where you could browse info about the ongoing test, the test schedule and the results easily. Perhaps naylor83 would be interested in helping with this. He did excellent work with the rarewares site.

  • kennedyb4
  • [*][*][*][*][*]
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #133
I too would be more interested in a multi-format at 96 with the AAC candidates being Nero and Itunes VBR.

This bitrate would be of more interest for those using portable devices I think.

  • IgorC
  • [*][*][*][*][*]
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #134
Come on people! If you feel geeky doing the tests, remember it's all in the name of science!

And science should be in the name of better life

  • vinnie97
  • [*][*][*][*]
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #135
Personally, I think 80 kbps is quite near of 64 kbps and 96 kbps would be a better next step for a multiformat test, but I suppose you have carefully thought about the options and followed the ongoing HA discussions more keenly than I.

I disagree.  48 --> 64 --> 80 = 16 kbps between each test iteration.  The difference between Vorbis at 64 and 80 kbps has me very interested in this test.  80 kbps is the portable sweet spot for me when it comes to Vorby...I'd like to see if it's still #1 or if there's a similar transparency being reached across all formats.

Sebastian, you're killing me....I need to know how many references I ranked. 
  • Last Edit: 13 August, 2007, 12:50:20 AM by vinnie97

  • rockcake
  • [*]
  • Members (Donating)
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #136
I'm also with vinnie97 wholeheartedly as regards 80 kbps tests - I'd also be interested in the results & suspect quite a few participants would be pleasantly surprised with the fidelity of the sounds; I for one was a little surprised in this test at the fidelity in some of the non-problem/general-music samples e.g. "Big Yellow" & "Les Voleurs" come to mind as being mostly quite listenable to my ears/head/brain IIRC.
Can't wait for the overall results!

Edit P.S.: I also dread my contributions being thrown out for whatever reason, so I tend to be a bit conservative and listen/rank like beto wrote above.
P.P.S.: Blimey, the next 128 mp3 test will be ruddy hard, I reckon!
  • Last Edit: 13 August, 2007, 01:35:32 AM by rockcake
Vorbis -q3 works for me.

  • muaddib
  • [*][*][*][*]
  • Developer
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #137
Hmm... I could use a bit more results for the last 10 samples.  I am extending the test one last time until 15th since I expected a bit more results to arrive shortly before the test ends. Maybe Guru can test, too (BTW, received two mails from you - thanks!).

It would be really great if people with golden ears would help us with this test. I guess for them it would be very easy and they could do it in 2-3 hours.

  • kdo
  • [*][*][*][*]
  • Members (Donating)
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #138
people with golden ears would help us with this test

But the golden ears would have to rate all these samples between annoying and very annoying.

Whereas people with average hearing can use the full scale for the ratings. Like I did.


Hmm... I could use a bit more results for the last 10 samples.


I have a question:  When somebody ranks the reference, do you disqualify the whole test for this sample or only the result for this particular encoder within the test?

Would it make any sense for those of us who ranked the references to repeat the discarded tests more carefully and re-submit?

  • muaddib
  • [*][*][*][*]
  • Developer
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #139
people with golden ears would help us with this test
But the golden ears would have to rate all these samples between annoying and very annoying.
Whereas people with average hearing can use the full scale for the ratings. Like I did.

Giving low grades wouldn't make any problems.
And because someone can hear very well and find those small differences it doesn't mean that he/she will grade those differences as annoying.

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #140
It would be really great if people with golden ears would help us with this test. I guess for them it would be very easy and they could do it in 2-3 hours.

I'll do it today or maybe tomorrow for all 18 samples - Sebastian will therefore have one more result for each one.

  • Alex B
  • [*][*][*][*][*]
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #141
I think the amount of ranked references in the future tests could be lessened with better instructions. ff123's practise page is very useful, but an updated page (with the JAVA version screenshots) inside the test site would be better.


The following things should be said very clearly:

- If the difference is not obvious, you should do an ABX test.
- If you can't succesfully ABX you should not try to guess. Don't pull either of the rating sliders down.
- If you didn't ABX test an obvious sample it would be good to listen to it once again and verify that you moved the correct slider.


Perhaps the instructions could have some step by step examples with screenshots. For example:

[Screenshot 1, the initial view] & a speech bubble:
"Hmmm... the sample on the left has less overall clarity and the symbals are a bit diminished, could it be the lossy file? Though, I am not sure, better go to ABX."

[Screenshot 2, ABX view, configuring the test]

[Screenshot 3, ABX view after a succesful ABX test] & a speech bubble:
"Oh dear, it was the other way around. This encoder has increased the volume of the higher frequencies slightly and at first sight it sounded better because the reference is not very bright."

and so on...

  • naylor83
  • [*][*][*]
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #142
"Oh dear, it was the other way around. This encoder has increased the volume of the higher frequencies slightly and at first sight it sounded better because the reference is not very bright."


"... Ah, that must mean this is the WMA sample! I'll bring it down an extra notch just for spite."



Edit: Seriously though... Good suggestions!
  • Last Edit: 13 August, 2007, 08:09:46 AM by naylor83
davidnaylor.org

  • muaddib
  • [*][*][*][*]
  • Developer
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #143
All participants in this test that have saved their sessions, could use Alex B instructions to verify that no references were ranked.

I am willing to help in building instructions for future listening tests (for this one it is to late I am afraid). Maybe someone could start this in Hydrogenaudio Wiki (if that is a good place).

  • kennedyb4
  • [*][*][*][*][*]
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #144
All participants in this test that have saved their sessions, could use Alex B instructions to verify that no references were ranked.

I am willing to help in building instructions for future listening tests (for this one it is to late I am afraid). Maybe someone could start this in Hydrogenaudio Wiki (if that is a good place).


That's a fantastic idea. I downloaded the program and a few samples but got bogged down in the mechanics of things. Wasn't really sure what I was doing.

A single clear " how to " manual might have helped.

  • kdo
  • [*][*][*][*]
  • Members (Donating)
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #145
All participants in this test that have saved their sessions, could use Alex B instructions to verify that no references were ranked.

In my case it would mean practically re-doing the whole test from scratcg, because most of the time I didn't do abx.
I did go through abx a few times when I was in doubt, and I did manage to abx successfully every time (or almost every, cannot remember). But there is still a chance I could have ranked a reference somtimes.

It would be nice if Sebastian could notify us by e-mail which of our tests are discarded and which we should re-do and re-submit. (Of course, if such thing is allowed by the test protocol)

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #146
The four first samples are tested on my side.
In order to avoid some mistakes I begin the procedure by ABXing all files and I start ranking them after this ABX procedure. Advantage: it's impossible to accidentaly mark the reference (only one slider can be lowered after a successful ABX test).

N.B. It seems that one competitor tends to make higher frequencies brighter (more noise/energy) which wouldn't be unpleasant if there wasn't additional artefacts. After lossy encoders, maybe the first "glossy" codec... 
  • Last Edit: 13 August, 2007, 10:40:20 AM by guruboolez

  • muaddib
  • [*][*][*][*]
  • Developer
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #147
The four first samples are tested on my side.

I hope you will not get bored until you get to samples 7.-18.

I don't understand why so little results for sample 8. It is easy after short 'getting used to it'. However I will not send my results because we at Nero don't want to have influence on the results.

  • Whelkman
  • [*][*]
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #148
Add to the list explanation of ABX testing behavior. It was far from obvious to me, and I had to redo several samples after experimentation (at the time I thought using the testing mode to eliminate the reference was required in all cases, but apparently this isn't true--this should be covered, too).

Not obvious is that the listener gets only one shot at testing mode unless he wishes to redo all samples within the test or load from a good session. The program also has some strange ideas of what qualifies as a commit; if I begin testing, reset my trials, then decide to come back later, I'll find the test was marked as a failed commit and will not be able to try again without starting over or loading from a successful session save. At least twice I successfully ABXed only to have my results discarded and replaced with a failed commit.

The listener should also know that he needs to run at least eight trials in testing mode. Early on I tried finalizing when ABC/HR first displayed a number (6/6, I believe), but results came up in red and the reference was not discarded, which is not helpful, especially given the retesting issue.

I experienced numerous bugs while using ABC/HR Java including crashes, disappearing menu options, nonfunctional sliders, failed saves, and phantom ABX commits. The program becomes especially unstable after a session save. Are these issues known?

Had I not kept notes on each sample, I probably would have quit due to the above frustrations.

  • [JAZ]
  • [*][*][*][*][*]
Public, Multiformat Listening Test @ 64 kbps - OPEN
Reply #149
I experienced numerous bugs while using ABC/HR Java including crashes, disappearing menu options, nonfunctional sliders, failed saves, and phantom ABX commits. The program becomes especially unstable after a session save. Are these issues known?

Had I not kept notes on each sample, I probably would have quit due to the above frustrations.


You play hard, it seems...

I've done three listening tests so far, and the only bugs i found were the non working audio (that gets fixed manually selecting the output audiocard), and the clicks when pressing play (which doesn't happen to me if i use the soundcard directly instead of java sound engine)

But if you say it happens *After* saving session... i've never saved session. (don't use to get interrupted in the middle of a test)
  • Last Edit: 13 August, 2007, 11:55:06 AM by [JAZ]