MP3@V6 ABX (just a personal test)

Topic: MP3@V6 ABX (just a personal test) (Read 3470 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.

MP3@V6 ABX (just a personal test)

2022-08-24 07:49:49

Well, this is my first post here, so...

As I decided to kinda leave streaming services, in favor of a more personalized experience, with lossless and my own library management (foobar, subsonic, all this stuff), I had to find a good codec for this, so I ended up in this forum. This day, I was trying to find a kinda good "flac-but-good-for-archival-purposes" alternative, what in other words may be: not so compressed, something that I could transcode later (to opus, for my portable) and still sounding good. Then I've thought hybrid wavpack could be a good solution... Maybe I could even store some of those "correction files" in a cold cloud like Amazon Glacier.

But suddenly I thought "are my ears really that good?". Then I decided to do a WV@440 vs FLAC ABX with a sample of this track ("Intro" of The Poison BFMV's album) and result were kinda... Not that good, low accuracy and 70% confidence ~~fortunate or unfortunately, I've even deleted it heh~~. So I managed to do an ABX test with that same sample, but against a MP3@V6 (~115kbps) encoded file. Looking at it, I don't think results were that good, is it right? "Normal" for a beginner in this thing or my ears aren't really that good? $:-\$

Code: [Select]

Completed trials: 11
Number correct: 8 (72.72727%)
Confidence that your results are better than chance: 0.88671875 (88.671875%)

Last played: A

Individual test results [choice/actual] from among 2 possible files:
Test #1	[ A / A ]
Test #2	[ B / B ]
Test #3	[ A / A ]
Test #4	[ B / B ]
Test #5	[ A / A ]
Test #6	[ B / A ]
Test #7	[ A / B ]
Test #8	[ B / A ]
Test #9	[ B / B ]
Test #10	[ B / B ]
Test #11	[ A / A ]

Files in use:

A: C:\abx\01 - Introflac.wav	gain: 0.00 dBFS
B: C:\abx\01 - Intro.mp3	gain: 0.00 dBFS

Ps.: sample is track 01 of "The Poison" (Bullet For My Valentine), time 02:00.655 to 02:14.061. I tried to focus on the violin that keeps going after this lead guitar fade.

Re: MP3@V6 ABX (just a personal test)

Reply #1 – 2022-08-24 08:27:33

Well, you have to test some more music to be somewhat sure. With age, our hearing is getting worse, more for some. I'm almost 50 and my limit is 17 kHz. But I keep my music in lossless format because storage is not that expensive any more, and my mobile phones can hold quite a lot of music so I don't get bored in a week of listening. If I want some more music, I can always transcode it to lossy.

The idea of lossy codecs are that utilizing psychoacoustics they can sound indistinguishable from original (transparent) at some determined bitrate; for mp3 that lies in range of 160+ bitrates, aac is 128+, IIRC, and opus can go as low as 80 kbit, I think. YMMV, of course, not everyone has the same hearing and sensibility to encoding artefacts. So, 70% confidence just means that lossy codec of choice does it's job as it should.

But as I said, you should test more. Welcome to ABX testing

Re: MP3@V6 ABX (just a personal test)

Reply #2 – 2022-08-24 22:05:59

The confidence interval for an ABX test is usually 95% or 99% (p < 0.05 or 0.01), depending on how strict you want to be. Since your confidence is 0.88, you did not pass the test, as you have a 12% chance of getting this result by just guessing. You're pretty close to passing, however, so you can probably pass if you practise a little longer.

Keep in mind that factors such as noise in the room, the use of speakers vs headphones, and how tired you are will influence the results. Don't push yourself to do too many tests at once, because if you're too tired to focus, the results don't mean anything other than that you were too tired to focus.

Re: MP3@V6 ABX (just a personal test)

Reply #3 – 2022-08-25 19:51:44

Quote from: itisljar on 2022-08-24 08:27:33

The idea of lossy codecs are that utilizing psychoacoustics they can sound indistinguishable from original (transparent) at some determined bitrate; for mp3 that lies in range of 160+ bitrates, aac is 128+, IIRC, and opus can go as low as 80 kbit, I think.

A user around here a while ago told me Opus @ 80kbps is similar to MP3 @ V5 (130kbps).

but personally my general minimums (which is typically what I use in general with each encoder) for each of the three following encoders...

-MP3 = V5 (130kbps) (the public listening test basically confirms this is going to be a 'good enough' setting for most people)
-AAC (Apple or FhG) = 96kbps
-Opus = 64kbps (I figure beyond about this point one is better off using a more standardized format, like MP3 or AAC. because I think this bit rate is Opus strength over more standardized formats like MP3/AAC and still offers good enough sound quality on music in general when you just sit back and enjoy)

but honestly, just in my personal opinion, it's plausible to say Opus @ 64kbps is similar enough to MP3 @ V5 (130kbps). because I think if someone is happy with MP3 @ V5, Opus @ 64kbps is in that ball park and takes basically about half of the storage space.

Quote from: Aleron Ives on 2022-08-24 22:05:59

The confidence interval for an ABX test is usually 95% or 99% (p < 0.05 or 0.01), depending on how strict you want to be. Since your confidence is 0.88, you did not pass the test, as you have a 12% chance of getting this result by just guessing. You're pretty close to passing, however, so you can probably pass if you practise a little longer.

Keep in mind that factors such as noise in the room, the use of speakers vs headphones, and how tired you are will influence the results. Don't push yourself to do too many tests at once, because if you're too tired to focus, the results don't mean anything other than that you were too tired to focus.

Which basically sums up my general point in that if you really got to focus fairly hard in general that basically means you are already struggling to notice a difference. hence, when just sitting back and enjoying ones music it won't take all that high of a bit rate to be plenty good enough. so by roughly that standard, it does not take all that much bit rate to pass that standard as I think the general bit rates I mentioned above (with MP3/AAC/Opus) would pass that standard.

because for me the whole point of using lossy files is to find that balance of file size/sound quality and in that regard, since I prefer to stick to the universally supported format of MP3, V5 (130kbps) is a pretty good default setting for many, if not most, people. even the listening tests around here say V5 is pretty good overall and should easily pass the 'when just sitting back and enjoying ones music' test.

p.s. just on my Klipsch Pro-Media PC speakers, which are above average PC speakers, when it comes to MP3, once I reach V7 (100kbps average) the quality is no longer 'very easy' to detect (on any random music) as below that point the overall sound takes a legitimate hit as it's more muffled etc as you can quickly notice it with FLAC vs MP3. but I figure given listening tests around here it's probably a good idea when it comes to MP3 to stick with V5 (130kbps average) to be safe for a minimum. even Opus @ 64kbps is probably closer to MP3 @ V5 than it would be MP3 @ V7.

Re: MP3@V6 ABX (just a personal test)

Reply #4 – 2022-08-25 22:17:02

LAME V5 is widely considered a minimum standard, but it's certainly possible to hear artifacts at V5. A single ABX test on a single track is hardly definitive, as different people have different levels of sensitivity to different artifacts. It's wise to select a variety of tracks from different genres in order to get a more representitive sample of how MP3 sounds on different material. Using samples from one of the public listening tests is also a good place to start.

It's also worth noting that you can train yourself to detect some artifacts by doing ABX testing, and thus a setting you used to enjoy, such as V5, may become inadequate once you know what to listen for. Since we're dealing with lossy audio, everything is a matter of perception, and we can shape those perceptions to some degree if we're willing to take some time to practise.

Notice