Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: 64 kbit/s test ended (Read 16082 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

64 kbit/s test ended

Reply #25 –
The main issue with MP3pro is that it uses SBR. People that don't know how SBR works - at all (they only read something about high frequencies reconstruction at CodingTechnologies' site and decided it sucks) - complain that it ruins sound quality.

It's similar to the joint stereo issues some time ago. Because of a bad encoder implementation, people started shooting at joint stereo, saying that it's bad and ruins the music. Now, all --alt-presets use joint stereo. Will these people keep complaining?

Probably.

64 kbit/s test ended

Reply #26 –
Quote
Originally posted by rjamorim
The main issue with MP3pro is that it uses SBR. People that don't know how SBR works - at all (they only read something about high frequencies reconstruction at CodingTechnologies' site and decided it sucks) - complain that it ruins sound quality.
But SBR is constructed very artificially (although you could say that from all lossy coding but in this case even more so), and thus can't probably ever sound totally transparent.
Juha Laaksonheimo

64 kbit/s test ended

Reply #27 –
Quote
Originally posted by JohnV
But SBR is constructed very artificially (although you could say that from all lossy coding but in this case even more so), and thus can't probably ever sound totally transparent.


Maybe in the general case for all existing samples out there, no, but then again, what does?

For the 64kbps test, I already know my answer (I failed one mp3pro clip).

--
GCP

64 kbit/s test ended

Reply #28 –
Quote
Originally posted by JohnV
But SBR is constructed very artificially (although you could say that from all lossy coding but in this case even more so), and thus can't probably ever sound totally transparent.


Of course not. But we aren't seeking transparency here, but the best quality at very low bitrates.

64 kbit/s test ended

Reply #29 –
Quote
Originally posted by rjamorim
Of course not. But we aren't seeking transparency here, but the best quality at very low bitrates.
You started talking about join-stereo and alt-preset standard, so I didn't figure you meant low bitrate only.. sorry.
Juha Laaksonheimo

64 kbit/s test ended

Reply #30 –
Thank you, ff123, for taking the time to prepare, host, and analyze the listening test.

Will you continue to host the original test samples on your Audio Samples page? Right now, the only way to get the samples is to manually type in http://ff123/net/64test/ and then the zipfile name (since the links on the test page are no longer active).

64 kbit/s test ended

Reply #31 –
Quote
Originally posted by JohnV
You started talking about join-stereo and alt-preset standard, so I didn't figure you meant low bitrate only.. sorry.


I mentioned Joint Stereo to give an example of a technology that people used to bash because of a bad implementation. IMO, that's the same issue with SBR.

64 kbit/s test ended

Reply #32 –
Quote
Originally posted by SometimesWarrior
Thank you, ff123, for taking the time to prepare, host, and analyze the listening test.

Will you continue to host the original test samples on your Audio Samples page? Right now, the only way to get the samples is to manually type in http://ff123/net/64test/ and then the zipfile name (since the links on the test page are no longer active).


The zips will be up for a little while until I find some time to add the originals to the samples page.

ff123

64 kbit/s test ended

Reply #33 –
Many thank, ff123, for conducting the test. I couldn't find the bitrates for the ogg -q0 files on the test result page. Could you please put it up there?

Monty already said the classical samples use about 40+ k, but how about others. Are they around 64 k? In those it won, does it use significantly more than 64 k?

I know nothing about AAC. Is it CBR? If it's also VBR, may we have the bitrate of its files as well?

Hope it's not too much trouble for you to do so. Thanks again.
tw101

64 kbit/s test ended

Reply #34 –
Quote
Originally posted by tw101
Many thank, ff123, for conducting the test. I couldn't find the bitrates for the ogg -q0 files on the test result page. Could you please put it up there? 

Monty already said the classical samples use about 40+ k, but how about others. Are they around 64 k? In those it won, does it use significantly more than 64 k?

I know nothing about AAC. Is it CBR? If it's also VBR, may we have the bitrate of its files as well?

Hope it's not too much trouble for you to do so. Thanks again.


All the bitrates are now posted.

ff123

64 kbit/s test ended

Reply #35 –
Very interesting test. I was amazed how generally low the scores were compared to how I scored (for the few samples I tested). For example, in the Blackwater sample, I could not distinguish any of the codecs from the original!

Could you plot in such a way as to show the spread in the data? A plot of the individual data points and/or standard deviations for each individual clip/codec combination would work. Even a single representative music clip would probably be enough to plot.

I also wonder what percentage of the testers could tell which codec they were listening to, based on the type of artefact heard, thus unblinding the test? Would you suspect this to be an issue?

64 kbit/s test ended

Reply #36 –
Quote
Originally posted by shday
Very interesting test. I was amazed how generally low the scores were compared to how I scored (for the few samples I tested). For example, in the Blackwater sample, I could not distinguish any of the codecs from the original!


One person could not distinguish any of the codecs on any of the samples, including the practice test.  So there is quite a range of sensitivity to artifacting.

Quote

Could you plot in such a way as to show the spread in the data? A plot of the individual data points and/or standard deviations for each individual clip/codec combination would work. Even a single representative music clip would probably be enough to plot.


Here is a plot of blackwater using standard deviations instead of the Tukey's HSD 95% confidence limits:



As you can see, it's not as informative as the 95% confidence limits.

Edit: It's probably better to use the standard error of the mean (divide the standard deviation by the square root of N) instead of the standard deviation anyway.  This gets you a plot which almost looks like the 95% confidence plots I drew.

Quote

I also wonder what percentage of the testers could tell which codec they were listening to, based on the type of artefact heard, thus unblinding the test? Would you suspect this to be an issue?


Monty suspects this to be an issue, and has said on the xiph list that a test cannot be truly blind if the listener is experienced in listening for artifacts.  It is possible, for example, for me (and others) to artificially raise mp3pro's rating and lower Wma's rating on some samples because I know what kind of faults they are prone to.  Also it is possible to grow to despise one type of artifact because one has had so much practice with it, but to not notice another type because of lack of familiarity.

These types of issues have generally not been touched upon in any of the papers (or the one book) I have read, so I don't know how this may have biased the test.

ff123

64 kbit/s test ended

Reply #37 –
Cumulative bitrate and standard deviation perhaps please? (yes, I know I can compute it myself)

64 kbit/s test ended

Reply #39 –
In bitrate calculation, Ogg is heavily penalized for short samples.  Each Ogg file begins with a 3-4kB header, which, in a 20 second sample, is a significant amount of the file size.

The 'bitrate' of the audio in ff123's calculations is including this header.

64 kbit/s test ended

Reply #40 –
Quote
There is an articel linking to this test on Slashdot.

Slashdot - the home of the uninformed (note: not the uniformed, although a Slashdot uniform would be an interesting thing to see).

"In my opinion, 192kbps MP3 is the way to go"

EDIT: This one must win some sort of gold star:
Quote
Man did you guys even do the test or are you just looking at the results? I did the test with my klipsch speakers and I got nothing like what these crackheads got on this site. Makes me wonder what they were listening to. I was at my fiances house and used her speakers and the worse compression sounded the best. Then I cam home to my beautiful klipsch (best computer speakers I have heard) and the difference was amazing. In most of the tests not only could I see a huge difference but I could tell you which one was which. The only ones I had trouble with were the two oggs and on some files the mp3pro. The oggs tend to increase the highs a tiny bit making them very different than the other compressions that tend to remove highs. The mp3pro on some occasions wasas good or a little better than the oggs. But both oggs where in the top 3 every time I did the test. I call for a redo because I think either people were guessing, using bad speakers or had too much wax in their ears. I did my own test with the blind player with ogg and did the two styles of ogg versus mp3 at 96k and 128k. The 96k was horrible but at 128k you could hardly tell the difference from the 64k ogg files. The two oggs were almost identical in every test. The 128 was a little better in almost every test but at twice the size it was incredible. I also tried then to compress the ogg files as much as I could to 45k max bitrate. Man it still sounded good but on songs with lots of simbols you could see obvious compression. Bottom line if you use on a regular basis 128k mp3's switch to ogg at 64k without having them side by side you will _NEVER_ know the difference.

64 kbit/s test ended

Reply #41 –
Quote
Originally posted by Jon Ingram

Slashdot - the home of the uninformed


Sadly true.  Most of them apparently don't know how to read either

Some of the comments from the users on this article are so absurd, they are even kind of scary.

64 kbit/s test ended

Reply #42 –
on songs with lots of simbols

Uhhhhh, what's a simbol? I think this goes straight to the the heart of the problem.
flac > schiit modi > schiit magni > hd650

64 kbit/s test ended

Reply #43 –
Quote
Originally posted by xiphmont
In bitrate calculation, Ogg is heavily penalized for short samples.  Each Ogg file begins with a 3-4kB header, which, in a 20 second sample, is a significant amount of the file size.

The 'bitrate' of the audio in ff123's calculations is including this header.


The bitrates if I subtract 3500 bytes from each of the ogg files come out to be 1 to 2 kbit/s less than the published bitrates.  I don't know if wma or the quicktime have a header as well, but I'll make some sort of note in the results page.

64 kbit/s test ended

Reply #44 –
Quote
Originally posted by Gabriel
Cumulative bitrate and standard deviation perhaps please? (yes, I know I can compute it myself)


Cumulative bitrates are:

mp3pro:  64
oggq0:  63 (61)
ogg64:  67 (65)
wma8:  67
aac:  66

The bitrates in parenthesis are after I subtract 3500 bytes from each ogg file.  Looks like wma8 probably has a header too.

ff123

64 kbit/s test ended

Reply #45 –
Quote
Originally posted by indybrett
Uhhhhh, what's a simbol?

...Cymbal maybe?

64 kbit/s test ended

Reply #46 –
Quote
Originally posted by Jon Ingram

Slashdot - the home of the uninformed (note: not the uniformed, although a Slashdot uniform would be an interesting thing to see).


I was surprised by someone who thought the sample size wasn't large enough ("only" 30 or so) and also talked about using trimmed means.  I guess he didn't notice how many of the differences were shown to exist with confidences > 95%.

Also, I'm surprised that a couple of people just blurted out that the test wasn't valid because it didn't include controls (i.e., the original).

ff123

Edit:  ah, I see that the person who complained about sample size was bitching that the test didn't represent the general population.  I think Dibrom's reply is excellent:

http://slashdot.org/comments.pl?sid=37003&cid=3977932

64 kbit/s test ended

Reply #47 –
Quote
Originally posted by maciey

...Cymbal maybe?


Ya think??
flac > schiit modi > schiit magni > hd650

64 kbit/s test ended

Reply #48 –
I an not sure if wma has a header.
But for sure it has a different bitrate meaning.
For MS, 1kbps is not 1000bps but 1024bps. So wma should be off by about 2 percents.

64 kbit/s test ended

Reply #49 –
Quote
Originally posted by ff123

Also, I'm surprised that a couple of people just blurted out that the test wasn't valid because it didn't include controls (i.e., the original).


I'm not suprised. It's very typical for /. to spew out criticism without even, for example, reading the story or checking the facts.

I started replying to some of the wrongful comments but got frustrated before halfway by the sheer stupidity of most of the comments. A week no /. for me, I need to recover.

--
GCP