Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Multiformat@128 public listening test - CANCELLED (Read 43338 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Multiformat@128 public listening test - CANCELLED

Reply #50 –
Quote
OR another possibility would be to change the packages now, exclude ATRAC and include correct vorbis. Also provide correct vorbis encodes seperately and make new setup files. Maybe that's a lot of work though and people should cooperate too. 

No, the problems are grave enough to justify starting again, and not try to fix while it's running.

I already brought the bittorrent tracker down and deleted the sample packages. Next, I'll put an apology at the place of the 128kbps page, and announce everywhere the test has been cancelled for the time being.

My sincere apologies to everyone that already dedicated their efforts to testing.

If people aren't already too fed up with this test's issues, I'll start it again next wednesday. I'll use sox to correct the volume of $#%! Atrac3 samples and I will use the correct aoTuV version.

Last but not least, I would like to invite you again to subscribe to the listening test newsletter. It's much easier for me to announce test-related news through it.
http://www.rjamorim.com/test/newsletter.html

Again, I'm very sorry.

Best regards;

Roberto.

Edir: I would appreciate if some mod removes this thread from validated news. Move it to listening tests please. And also please change the subject from OPEN to CANCELLED or something like that.
 

Multiformat@128 public listening test - CANCELLED

Reply #51 –
Maybe the default abchr should be the native windows version, with abchr-java included for those who must use it.  That would mean throwing out the encryption, though.  If that's the case, I need to make a quick bug fix to clear the general comments when switching tests.

Also, I offer to do a quick, once-through double-check of the packages just prior to their release.

ff123

Multiformat@128 public listening test - CANCELLED

Reply #52 –
Quote
Maybe the default abchr should be the native windows version, with abchr-java included for those who must use it.  That would mean throwing out the encryption, though.  If that's the case, I need to make a quick bug fix to clear the general comments when switching tests.

That would also mean throwing out gain correction, but I understand the Java ABC/HR has some problems inherent of the Java platform. That's a pity.

Quote
Also, I offer to do a quick, once-through double-check of the packages just prior to their release.


That would be very welcome. Thank-you very much.

Multiformat@128 public listening test - CANCELLED

Reply #53 –
I'm completely in favour of Roberto's decision.

Edit: not good expression, might cause some misunderstanding.

Multiformat@128 public listening test - CANCELLED

Reply #54 –
We've already postponed the listening test a few times and survived so there shouldn't be any problem delaying it for another week, just to sort out the issues.

Multiformat@128 public listening test - CANCELLED

Reply #57 –
A test initiative this complex (wide variety of samples, diverse formats, many encoder versions, etc.) is bound to run into a problem every now and then.  A big THANKS to Roberto for the time he's put into this test, and for having the integrity to call a pause to fix problems rather than pushing forward with too many open issues that would taint the significance of the results.

I hope the people that have put time into testing will be able to continue again when the test is restarted.  I, myself, had just downloaded the package and a couple of samples last night, but had not done any testing yet.

These listening tests are very important to the world of audio compression, and I'm sure everyone here is grateful for the effort put into them by everyone involved.

Multiformat@128 public listening test - CANCELLED

Reply #58 –
Quote
Quote
The spectral view is clearly consistent with beta 2 and not with 040402.  In the beta 2 version, the lowpass is varying above 17 kHz, and in 040402 the lowpass is a constant 17 kHz.

However, if I do a binary compare of the wavs, I find all three are different from each other.

all three are different? 
strange

Oops.  It seems I made a mistake when encoding with aotuv beta 2.  In fact the spectral views for both the beta 2 and 040402 look very similar, with varying lowpass in both (and at -q 4 and -q 4.35).

And I did verify this time that the a file encoded with 040402/decoded to wav, and the wav file in the downloaded package were bit identical.

ff123

Multiformat@128 public listening test - CANCELLED

Reply #59 –
Quote
Maybe the default abchr should be the native windows version, with abchr-java included for those who must use it.  That would mean throwing out the encryption, though.  If that's the case, I need to make a quick bug fix to clear the general comments when switching tests.

Also, I offer to do a quick, once-through double-check of the packages just prior to their release.

ff123

This sounds like a good idea too - and I'm saying this not having installed windows on my computer.

The native version was pretty much reliable in the past but there were lots of people (including me) having problems with the Java version. It seems like the problems are due to limitations in the Java API and thus won't be fixed easily.

The vast majority seem to be windows users anyway and there are probably very few potential participants that don't have access to any windows computer. For the few left the Java version is probably okay for now.

Having some people testing the setup before it's officially announced also sounds very reasonable.

Multiformat@128 public listening test - CANCELLED

Reply #60 –
Did someone noticed that QuickTime 6.5.1. was used for encoding? This mean that the output files are identical to files produced with iTunes 4.5. With the High Frequencies specific behaviour.
I've compared (bit-to-bit) one encoding (Hongroise) packaged to one I've encoded with iTunes 4.5, and they are perfectly identical.

Roberto used iTunes 4.2 GUI, but iTunes 4.5 codec.

I'm not sure this was intended.

Multiformat@128 public listening test - CANCELLED

Reply #61 –
Quote
Did someone noticed that QuickTime 6.5.1. was used for encoding? This mean that the output files are identical to files produced with iTunes 4.5. With the High Frequencies specific behaviour.
I've compared (bit-to-bit) one encoding (Hongroise) packaged to one I've encoded with iTunes 4.5, and they are perfectly identical.

Roberto used iTunes 4.2 GUI, but iTunes 4.5 codec.

I'm not sure this was intended.

Yeah, when uninstalling iTunes 4.5, you have to uninstall QuickTime as well.  Sounds like a pain.

Multiformat@128 public listening test - CANCELLED

Reply #62 –
Quote
Did someone noticed that QuickTime 6.5.1. was used for encoding? This mean that the output files are identical to files produced with iTunes 4.5. With the High Frequencies specific behaviour.
I've compared (bit-to-bit) one encoding (Hongroise) packaged to one I've encoded with iTunes 4.5, and they are perfectly identical.

Roberto used iTunes 4.2 GUI, but iTunes 4.5 codec.

I'm not sure this was intended.

Yes. The Sample_4 is bit identical(screenshot) to iTunes4.5 AAC 128kbps. I noticed characteristic artifact of iTunes 4.5 when I was testing on my familiar rosemary. Dropout or something happens in right channel around 23sec.

Multiformat@128 public listening test - CANCELLED

Reply #63 –
Native windows abchr updated:

http://ff123.net/abchr/abchr.html

1. Fixed bug in which general comments were not being cleared in between tests.
2. Added option to attach nicks to results files, same as in abchr-java
3. Fixed bug in which program was being opened up beneath other windows.  This has a side effect:  all of the dialog boxes flash on/off as abchr is launched.  I judged this to be less annoying than to have abchr not have focus upon launding.

ff123

Multiformat@128 public listening test - CANCELLED

Reply #64 –
Roberto, you are right we can't continue the test with these issues unresolved.

  Great thing that you took the decision to call it off in time!

@shnoffler: I don't know if this discussion is relevant for the Java playback volume issue:
Javasound list archives #008300
You can PM me to discuss Java problems.

Multiformat@128 public listening test - CANCELLED

Reply #65 –
Quote
Quote
Maybe the default abchr should be the native windows version, with abchr-java included for those who must use it.  That would mean throwing out the encryption, though.  If that's the case, I need to make a quick bug fix to clear the general comments when switching tests.

That would also mean throwing out gain correction, but I understand the Java ABC/HR has some problems inherent of the Java platform. That's a pity.

I my opinion it's absolutely essential that you remove these clicks. It's not a blind test when the original version is signalled by a click at the start, and 4/6 coded versions aren't!!!

So please do return to the windows app as the default, and treat any results from the java app with suspicion - maybe cross check to see if any bias creeps in?


As for the volume issues, see my suggestions here:
http://www.hydrogenaudio.org/forums/index....showtopic=20694


Basically, time align all files perfectly, ensuring they are all exactly the same length. Band pass filter the original and ATRAC files 100Hz to 10kHz. Calculate the whole file RMS value for each. (Cool Edit will do this - the values are in dB). Calculate the difference (a pocket calculator will do this!). Dump the band pass filtered files, and load in the ATRAC file again. Amplify it by the difference you have just calculated. Save it. Job done.

You could try calculating the RMS values on the other files too, to see how closely they match. They should be very close.

I'm going to try it myself and report back...

Cheers,
David.

Multiformat@128 public listening test - CANCELLED

Reply #66 –
wise move to cancel and restart, rjamorim

note that we all really appreciate what you are doing!
it cant be esteemed enough what big value your tests have for bringing light into the codec(-marketing) jungle

Quote
Yeah, when uninstalling iTunes 4.5, you have to uninstall QuickTime as well.

good to know 
I know, that I know nothing (Socrates)

Multiformat@128 public listening test - CANCELLED

Reply #67 –
This post is not just about the current test, but about level-matching and ABXing in general...

OK, it doesn't really matter how you measure it (ReplayGain, average or total of 50ms RMS values with or without a bandpass filter), the volume differences between the files are...

1. almost nothing
2. almost nothing
3. almost nothing
4. about 0.1dB
5. almost nothing
6. 3dB

"almost nothing" means about 0.01 to (at the very most) 0.04 dB, which can be ignored.

File 6 needs fixing, and strictly speaking file 4 needs fixing, since true ABX tests try to match levels to within 0.1dB - but with a 16-bit output, you might do more harm than good by making the adjustment.

The file numbers are from the .wav files I have from the Kraftwork sample - I haven't attempted to identify them.


Very important note: files 3, 4 and (especially) 6 aren't time-aligned or the same length. If you don't correct this, then the ReplayGain correction suggestions become:
3. 0.18dB
4. 0.33dB
6. 2.92dB

Using ReplayGain when ABXing and not time-aligning and length-matching the files could introduce an audible level difference where none existed before - don't do it!

Cheers,
David.
P.S. full results...

Code: [Select]
Corrections calculated from ReplayGain
file  correction
0     
1     +0.02
2     -0.01
3     +0.01
4     +0.17
5     +0.00
6     +3.00


Corrections calculated from Cool Edit RMS values...
Without band pass filter               
file   La        Ra        Lt        Rt
0               
1     +0.01     +0.04     +0.01     +0.04
2     +0.01     +0.01     +0.02     +0.01
3     +0.00     +0.01     +0.01     +0.00
4     +0.05     +0.12     +0.06     +0.12
5     -0.03     +0.00     -0.02     -0.01
6     +3.00     +3.01     +3.01     +3.01

With band pass filter               
file   La        Ra        Lt        Rt
0               
1     +0.01     +0.02     +0.00     +0.01
2     +0.01     +0.00     +0.01     -0.01
3     -0.01     -0.01     +0.00     -0.02
4     +0.07     +0.16     +0.09     +0.14
5     -0.05     -0.02     -0.04     -0.04
6     +3.00     +3.00     +3.00     +3.01

Multiformat@128 public listening test - CANCELLED

Reply #68 –
Quote
wise move to cancel and restart, rjamorim

note that we all really appreciate what you are doing!
it cant be esteemed enough what big value your tests have for bringing light into the codec(-marketing) jungle

Too right - on all three counts!

I hope you find my comments on here helpful, rather than overly critical - they're certainly meant to be helpful!

All the best,
David.

Multiformat@128 public listening test - CANCELLED

Reply #69 –
Quote
Roberto used iTunes 4.2 GUI, but iTunes 4.5 codec.

So there's another wrong codec used!  What's the matter with this test? Has it been cursed by the gods or something? 

Anyway, you made the right decision Roberto. I think everyone here is grateful for you work and we are all looking forward to the new (hopefully right this time!) reincarnation of the test. 

regards
echo

Multiformat@128 public listening test - CANCELLED

Reply #70 –
I know that the use of iTunes 4.2 and QuickTime 6.5.0 was decided upon for use in this listening test due to newly found artifacts in iTunes 4.5 and QuickTime 6.5.1.  However, using QuickTime 6.5.0 and thus iTunes 4.2 for this test is a bad idea, and I'll explain why.

QuickTime 6.5.1 fixes a nasty vulnerability in version 6.5.0 that can be exploited remotely.  You can read more about that here.  By using and promoting iTunes 4.2 and QuickTime 6.5.0 in this test, you are promoting sound over computer security.  Never an appropriate trade-off, IMO.  No one should be lead to believe that they should use an exploitable version of a program over a newer, patched, more secure one, and that's exactly what you would be doing by using QuickTime 6.5.0 in this test.

The best thing to do, IMO, would be to just stick with iTunes 4.5 and QuickTime 6.5.1 and its newly introduced artifacts and just let the chips fall where they may.  It may suck for the AAC enthusiasts to see AAC potentially score lower against the other codecs, but that's the price of computer security.

Just something for y'all to think about.

Multiformat@128 public listening test - CANCELLED

Reply #71 –
Quote
I know that the use of iTunes 4.2 and QuickTime 6.5.0 was decided upon for use in this listening test due to newly found artifacts in iTunes 4.5 and QuickTime 6.5.1.  However, using QuickTime 6.5.0 and thus iTunes 4.2 for this test is a bad idea, and I'll explain why.

QuickTime 6.5.1 fixes a nasty vulnerability in version 6.5.0 that can be exploited remotely.  You can read more about that here.  By using and promoting iTunes 4.2 and QuickTime 6.5.0 in this test, you are promoting sound over computer security.  Never an appropriate trade-off, IMO.  No one should be lead to believe that they should use an exploitable version of a program over a newer, patched, more secure one, and that's exactly what you would be doing by using QuickTime 6.5.0 in this test.

The best thing to do, IMO, would be to just stick with iTunes 4.5 and QuickTime 6.5.1 and its newly introduced artifacts and just let the chips fall where they may.  It may suck for the AAC enthusiasts to see AAC potentially score lower against the other codecs, but that's the price of computer security.

Just something for y'all to think about.

yeh, but the assumption is that whatever problems exist in 4.5 version will be fixed for the next one, so we should test what things SHOULD sound like. I don't think the intention is to actually recomment an old version of quicktime, just to determine how the best quality AAC sounds.

Multiformat@128 public listening test - CANCELLED

Reply #72 –
I don't see what the point was in first testing for the best AAC encoder and then chosing a worse one.

Multiformat@128 public listening test - CANCELLED

Reply #73 –
Quote
Yeah, when uninstalling iTunes 4.5, you have to uninstall QuickTime as well.  Sounds like a pain.

What is weird is that I did uninstall QuickTime 

I'll make sure I delete System32/QuickTime in it's entirety when I prepare the samples for next test

Multiformat@128 public listening test - CANCELLED

Reply #74 –
Quote
I hope you find my comments on here helpful, rather than overly critical - they're certainly meant to be helpful!

Oh, they are certainly very helpful.

I'll adhere to your guide once I start preparing samples.

Quote
QuickTime 6.5.1 fixes a nasty vulnerability in version 6.5.0 that can be exploited remotely. You can read more about that here. By using and promoting iTunes 4.2 and QuickTime 6.5.0 in this test, you are promoting sound over computer security. Never an appropriate trade-off, IMO. No one should be lead to believe that they should use an exploitable version of a program over a newer, patched, more secure one, and that's exactly what you would be doing by using QuickTime 6.5.0 in this test.


This test it to compare codec quality, period. As I said several times before, all other variables don't matter: price, ease to use, multiplatform availability, license... in this case, security. At most, I can add a warning to the results page mentioning the security issue.