Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Multiformat@128kbps listening test - FINISHED (Read 181548 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Multiformat@128kbps listening test - FINISHED

Reply #125 –
Quote
I'd like to see sometime a double test. Meaning, another test after the first one with another set of samples, and see how close the final results are to each others.

You are invited to conduce it

Multiformat@128kbps listening test - FINISHED

Reply #126 –
Quote
Quote
Which implementation of ATRAC3 did this test use?

SonicStage2

Quote
I only see flac decompressing the wav, where does it originate from?


Decoding the Atrac3 and encoding to FLAC. There's no other way to distribute the Atrac3 samples.

Real time recording via internal loopback?

Multiformat@128kbps listening test - FINISHED

Reply #127 –
Quote
Real time recording via internal loopback?

Total Recorder.

Multiformat@128kbps listening test - FINISHED

Reply #128 –
It seems people outside HA does not understand the language I speak, they interpret 'yes' as 'no' and viceversa, 'better' as 'worst', 'scientific, objective and repicable by yourself' as 'my mother did it and she owns the truth'

Multiformat@128kbps listening test - FINISHED

Reply #129 –
Quote
I'd like to see sometime a double test. Meaning, another test after the first one with another set of samples, and see how close the final results are to each others.

Couldn't you just split the current test into two 9-sample tests and pretend one was taken after the other.  Comparing the results of these two 'sub-tests' would in effect be the same, and you've got the benefit of 49'000 different ways to create two sub-tests.  I would imagine from the previous comments that you wouldn't find a great amount of discrepancy, I got the impression that gone are the days when WMA is best for classical and mp3 best for metal [span style='font-size:8pt;line-height:100%'](or whatever codec/genre associations there were)[/span]
< w o g o n e . c o m / l o l >

Multiformat@128kbps listening test - FINISHED

Reply #130 –
Quote
It seems people outside HA does not understand the language I speak, they interpret 'yes' as 'no' and viceversa, 'better' as 'worst',

Calm down... 
People who WANT to understand, will understand.
People who do not care - will not...
There is old russian saying, i will try to translate:
When you argue with a fool - take care, other people could see no difference 
[between]
 

Multiformat@128kbps listening test - FINISHED

Reply #131 –
Quote
Erm... it's in the results page. Read the second sentence of "How to interpret the plots:

Now, officially they are tied. But considering Vorbis' score is above MPC's confidence margin, I would say, with some confidence, that Vorbis aoTuV is better than MPC, at this bitrate.

Haha, yeah, figured I could find it out in a few moments, but I didn't really have them when I posted.

You make sense with the confidence margin thing, true, but you're likely going to start confusing the less statistically minded unless you stick pretty hardcore to the 95% confidence interval information. Either that, or you qualify the hell out of any statement that doesn't comply to the 95% interval.

Multiformat@128kbps listening test - FINISHED

Reply #132 –
Quote
You make sense with the confidence margin thing, true, but you're likely going to start confusing the less statistically minded unless you stick pretty hardcore to the 95% confidence interval information.

It's no use. This test is not controlled enough to warrant sticking to the 95% confidence as if it was gospel. Differently from ITU tests, I have no control of participants' listening environment, equipment, training, fatigue, etc. (and that's why ITU tests are damn expensive)

These results are there just to give an idea of how codecs rank. They are not trying to be definitive in what they report. And people should still test for themselves to decide what codec beter suits them, and consider other features like availability, hardware support, etc, etc.

Multiformat@128kbps listening test - FINISHED

Reply #133 –
Quote
I'm surprised this "claim" hasn't been debunked yet. Vorbis did not win.

even if they are both tied (Vorbis and MPC) it means that vorbis (and (posibli) MPC) won.

Multiformat@128kbps listening test - FINISHED

Reply #134 –
Quote
Quote
Was there any pattern in those 54 discarded results as to which codecs' were mis-identified? If, for example, half of those 54 were thrown out because they ranked the reference vs. MPC that would be somewhat interesting.


Hrm... you would have to check the output of Chunky with the command line I posted earlier to see what results are being discarded, and then analyze these results one by one.

Oh, OK, I thought someone may have actually looked at those discarded results already. 

Maybe someone can answer the other part of my question:

Does that mean that all of the users' rankings for that sample were thrown out or just for the codec(s) where they ranked the reference?

Multiformat@128kbps listening test - FINISHED

Reply #135 –
ok, so it looks like the vbr contenders did very well and itunes's cbr held its on.  how safe would it be to assume that using vbr with AAC (for instance the most recent FAAC with FB2K) would be a contender?

Multiformat@128kbps listening test - FINISHED

Reply #136 –
Quote
There are replies:
http://forums.minidisc.org/viewtopic.php?p=22345#22345
http://minidisct.com/forum/showthread.php?threadid=22995

This guy's signature is great!
Quote
Best portable setup = 128kbps MP3 (super high quality, > CD!) -> transcoded to the best codec in the world, uber high quality ATRAC3/LP4 (5000% better than SACD) -> NetMD (faster than ur sh*tty firewire) -> N710 (EU version with 1.2mW x2, OH YEAHHHH BABY!) + MDR-E808 (bestest hedfonez in teh world!)
This will shizz on all ur lame iPods! Its sooooo clear dat I can almos feel teh mud flwing dwn da waterfal!

Worst portable setup = CD -> WAV -> (WaveGain @ 87dB) -> iTunes 4.5/QT 6.5.1 encoded 224kbps AAC or ALAC -> 3G iPod + Etymotic ER-4P

I am actually impressed with most responses there, but apparently some believe that the test is not fair because ATRAC was not tested on a preferred hardware DAC. 
"Facts do not cease to exist just because they are ignored."
—Aldous Huxley

Multiformat@128kbps listening test - FINISHED

Reply #137 –
The arguments at the minidisc forums about hardware encoded Atrac3 sounding better than software encoded make no sense. The opposite actually makes more sense. On hardware, you must be worried about real time encoding, voltage consumption and battery consumption. On software, you can go nuts.

So, if Sony cut corners somewhere, it must have been on hardware due to inherent limitations.

Multiformat@128kbps listening test - FINISHED

Reply #138 –
Quote
The arguments at the minidisc forums about hardware encoded Atrac3 sounding better than software encoded make no sense. The opposite actually makes more sense. On hardware, you must be worried about real time encoding, voltage consumption and battery consumption. On software, you can go nuts.

So, if Sony cut corners somewhere, it must have been on hardware due to inherent limitations.


Even worse is the fact that some people claim ATRAC3 sounds "better" decoded through Type-S or their 1-bit digital amps, so the test is therefore invalid 

I don't think that some people understand the point of comparing lossy codecs: it's not to see which one sounds "warm" or "fat" or "has better bass," it's to compare artifacts, with the best codec having the least number of and/or least annoying artifacts. I want to smack people when they claim that while ATRAC3 sounds worse than MP3 on the computer, it will sound better going through their 1-bit digital amp. NOOOO!!!    An artifact is an artifact...a phasey cymbal or dropout will still be there no matter how good your amp or boost boost is. I'm personally surprised that although many people claim to be able to discern the "higher quality" of a 1-bit digital amp on certain players, they apparently aren't able to pick out what are sometimes blatant artifacts. I wonder how much of that can be attributed to marketing?

Multiformat@128kbps listening test - FINISHED

Reply #140 –
Quote
Quote
I'm surprised this "claim" hasn't been debunked yet. Vorbis did not win.

even if they are both tied (Vorbis and MPC) it means that vorbis (and (posibli) MPC) won.

I agree.  To have a winner, you must have a loser(s).  And there are some notable losers in this test (ie. ATRAC3).  Since Vorbis and MPC are statistically tied, they both won over the rest.

Multiformat@128kbps listening test - FINISHED

Reply #141 –
I've uploaded chunky-0.8.4 which fixes the filename extension problem.  Also, I've changed the default behavior so that it discards files with ranked references (i.e. -p 0.0 is assumed unless specified otherwise).

You can get it, as usual, at http://www.phong.org/chunky/

Quote
Does that mean that all of the users' rankings for that sample were thrown out or just for the codec(s) where they ranked the reference?

Yes, the whole result for that sample is thrown out.  To do otherwise would taint the results.  Even if you just guessed without listening, you would get about half of them right - if you just discarded the wrong ones, you'd still have half left with completely invalid ratings.  The only safe route is to toss the whole result file.

On the other hand, it is possible that the reference was ranked inadvertantly even if they did hear a difference (if it was very subtle).  In those cases (i.e. when the differences are subtle), it's best to make an ABX test - if you are successful, the ranked reference won't cause it to be discarded.  If you fail the ABX test, then you know you probably didn't hear a difference and you shouldn't rank the sample at all (leave it at 5.0).
I am *expanding!*  It is so much *squishy* to *smell* you!  *Campers* are the best!  I have *anticipation* and then what?  Better parties in *the middle* for sure.
http://www.phong.org/

Multiformat@128kbps listening test - FINISHED

Reply #142 –
Quote
I agree.  To have a winner, you must have a loser(s).  And there are some notable losers in this test (ie. ATRAC3).  Since Vorbis and MPC are statistically tied, they both won over the rest.

I meant that Vorbis didn't win when compared to Musepack. That's all. I didn't mean globally. Sorry for the confusion.

Multiformat@128kbps listening test - FINISHED

Reply #143 –
Quote
The arguments at the minidisc forums about hardware encoded Atrac3 sounding better than software encoded make no sense.

How can you jump straight into the conclusion like that?

Hardware ATRAC3 encoder in MD player may use different codebase from software counterpart.

Sonic Stage to ATRAC3 maybe something like Blade is to MP3. We have to test it.

Multiformat@128kbps listening test - FINISHED

Reply #144 –
Quote
How can you jump straight into the conclusion like that?

Hardware ATRAC3 encoder in MD player may use different codebase from software counterpart.

Sonic Stage to ATRAC3 maybe something like Blade is to MP3. We have to test it.

<sigh>

Have you ever even bothered reading the rest of my post?

Here, let me give you some knowledge. That way, you will think twice before posting next time:

On hardware, a developer must be concerned about constraints like voltage consumption, battery consumption, real time encoding, less precision (no FPU), a fraction of the CPU clocks, etc.

On software, the developer can go nuts since none of those restrictions apply.

On codec development, the usual path is first creating a software implementation (that will also be later used for compliancy tests), and then start cutting corners and complexity for the hardware version until it reaches the desired performance.

FOR THAT REASON, I claim it's nonsense. I don't claim it's impossible, maybe Sony has some serious voodoo going on there. But it does go against common sense.

Common sense is that they aren't deliberately putting a worse version of Atrac3 "like blade is for MP3" on SonicStage for kicks and giggles.

You're welcome.

Regards;

Roberto.

Multiformat@128kbps listening test - FINISHED

Reply #145 –
Well, I'm not sure about MPC.
Before the test I mentioned that MPC perhaps is only as good because it uses very high bitrates on this problemsamples! But if the average bitrate is 128 for the tested qualitysetting, there should also be a lot samples with bitrates under 128kbits! Logical, isn't it?
The problem on this test is that most samples had high bitrates and the samples with small bitrates were not ranked as good!

For example you could also modify an mp3-encoder to user very high bitrates (160kbits) on difficult samples and very low (80kbits) on normal samples. In this thest it would probably be better thant the current lame-encoder but in practice there would be a lot of songs which would sound very bad!

I hope you understand what I mean. But perhaps my idea is totally wrong!?

Big_Berny

Multiformat@128kbps listening test - FINISHED

Reply #146 –
Congratulations to Roberto for once more pulling through a tough one. Great work, greatly appreciated.

Regarding the bitrate "issue"...

The encoders in the test were using standard settings, they were not specially tweaked for the test, and you can go ahead and use them with your songs.

So if some of the encoders have a flawed code to choose the bitrate in tough passages of music, well, it's their problem.

I think this test is really useful as an indication of which encoder does a better job with a setting that will end up giving an average 128kbps in a whole bunch of music. And I fail to see what's wrong with the idea.

Multiformat@128kbps listening test - FINISHED

Reply #147 –
Just to throw in my 2 cents, considering the ATRAC and WMA forums' responses:

I think we all more or less knew that there will be such reaction, when we post these results. I even think some hoped for such reaction, so they can say that these people make unsupported claims and such.

I myself trust the results, though I won't change my encoding habits: Lame aps for me, as I only have an mp3-capable portable. The people in the ATRAC/WMA forums won't do that either, IMHO, as they payed a lot of money to be able to use the formats they defend now.

And, to be honest, those people who really care about audio quality, end up at HA finally    And those who don't help the sales of lower quality codec capable devices skyrocket, because they only listen to the commercials.

And don't tell me you didn't read the "2 cents" warning

One more on-topic question: is it possible to send these results to portable manufacturers? Would it make some reason if we'd make a thread for collecting contact email addresses, so we could mail most portable manufacturers, to give them a hint what to develop? E.g. Daisy MM (manufacturer of Diva) wrote me in an email, that they would consider implementing further codecs, if their licensing fees are fine. So why not give the companies a hint?
Life is Real...
(But not in audio :) )

Multiformat@128kbps listening test - FINISHED

Reply #148 –
Quote
Have you ever even bothered reading the rest of my post?

I did read your post on minidisc forum (and agree with you on that sense) before posting that.

My point is if they (minidiscers) claim that their MD hardware encodes better, then we should consider their claim. Similar to how we select the best encoder for other codecs in your test.

Multiformat@128kbps listening test - FINISHED

Reply #149 –
Quote
My point is if they (minidiscers) claim that their MD hardware encodes better, then we should consider their claim. Similar to how we select the best encoder for other codecs in your test.

consider and then dismiss, if there is no proof for the claim, other than general subjective opinions.
if there is (semi-) scientific proof, it will will be gladly accepted.

guess what's gonna happen.
Nothing but a Heartache - Since I found my Baby ;)