Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: How should I interpret my ABX tests? (Read 3451 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

How should I interpret my ABX tests?

Hello guys and girls,  I've been lurking this forum for so long and have gathered such good info from all of you so thnks. Well, today I did some abx tests with apple aac because I was trying to find a good spot for my mobile device and I ended up with a question: if I dont feel confidence, but still manage to get high scores in multiple tests, can it still be considered that I lowkey can discern the difference? I personally feel like I do but I would like to know about the experts in here to clarify this for me if possible.
I did 128, 160, 192 (not included in here) and 256 all of them in cvbr.

128cvbr
Spoiler (click to show/hide)


160cvbr
Spoiler (click to show/hide)


Ok this one was a surprise for me, I did one try and got a high score and right after I did another one to understand if I was really hearing something different and it turns out I scored the same lol

256cvbr 1st try (I had to concentrate here bc its hard and I lost confidence one or 2 times like I was clueless but most of the time I heard a difference)

Spoiler (click to show/hide)


256cvbr 2nd try (since I got shocked with the results on the 1st one I redid and got the same score again with concentration and lots of repetition even though I did this one quite fast)

Spoiler (click to show/hide)


I should also say that the 256 and 128 tests were made with no long pauses. With that being said Id like to say for those who were deciding if 128 cvbr is good quality (like I was for months) for mobile devices, yes it is and is the one im picking for my mobile even though I could discern the small diferences in higher bitrates. A nice user in this forum ThaCrip gave such good info about encoding for mp3, opus and aac and I agree with him so thnk u. This was just something Id like to say but, if anyones wondering, the difference I heard was so minimal that imo I wont go any higher than 256 (if I have enough space) and no lower than 128 or 96 if space is critical. Sorry for being messy juts wanted to share my thoughts and to have my answer clarified. Hope Im not breaking any rules tho bye

Re: How should I interpret my ABX tests?

Reply #1
The former two:

You got 8/8 right. Could that have happened by coin-flipping?
Of course it could, and in 0.39 percent of such coin-flipping experiments it would happen. But 0.39 percent isn't much.
That is, the result is much more consistent with you actually telling the difference than you merely guessing the difference - and so much more that we are confident you weren't "merely guessing".
Please interpret "merely guessing" accordingly: your gut feeling might be that your "telling" is quite a bit of guesswork, but the numbers do say that you are much better at it than the coin would be. The coin is "merely guessing", your results indicate that you are doing much more.


The latter two:

You got 7/8 right. The coin has greater chances of getting "7 or 8 right" than of hitting full score, it happens in around 3.52 percent of the cases. We are not so confident that you are not a coin as we are in an 8/8 case, but still.

Re: How should I interpret my ABX tests?

Reply #2
@Porcus
I get it now, thank u for explaining that. Indeed 256 was hard in a way that demanded too much of my concentration if I did notice something it was minimal and I couldn't care less. Things got really challenging enough for me 160+ anyway...

Re: How should I interpret my ABX tests?

Reply #3
Very few would be able to tell a good lossy encoder [and Apple's AAC encoder has a good reputation] @ 256 from lossless on a typical music signal. I don't know if this track is "typical".
(There are well-known "killer samples" that are proof-of-concept that lossy encoders/codecs cannot hide everything for us, but a quick googling does not indicate that this is a well known trouble maker.)

Did you encode the AAC yourself from the FLAC? If you got them readymade (say you bought one track from iTunes and later got the CD and ripped to lossless) they might be different masterings, where the signal is indeed different (sometimes obviously so, sometimes subtly so). But if you did it yourself ...

Re: How should I interpret my ABX tests?

Reply #4
I did my rip from the official Flac file using foobar to ALAC and then from iTunes to these settings above (I know foobar has the same encoder I think but I just picked iTunes bc is the one I always use). This song is also not common for doing abx test lol I just picked randomly a pop song with an acceptable noise during the final chorus just to experience if I could notice something that would bother me highly with these compressions whenever I'm relaxing or even in a house party. Even 96kbps apple aac was pretty much acceptable . Talking about artifacting with apple aac I couldn't hear none to this date (Im quite blessed for that lol) and I don't even know how they actually sound, probably less annoying than the mp3 one's. I know one song that has, I think you would say pre echo(??), and gives a hard time for lame 320 right at the beginning. SOPHIE - Lemonade song.  sorry I can't proof that right now but its nice to try it later. This doesn't happen with aac even in lower bitrates which really affirms that using  aac is a good compression method

Re: How should I interpret my ABX tests?

Reply #5
Quote
Even 96kbps apple aac was pretty much acceptable

If you start getting curious over this:
* Try exhale for xHE-AAC: https://hydrogenaud.io/index.php?topic=118888.msg996848
* Try Opus.

I am impressed at how much music you can get out of 48 kbit/s or even less. Not saying it sounds awesome, I am saying I am impressed - recall that MP3 has a minimum of 32, anything less was considered to be so totally useless that it wasn't even allowed in the format specification.

Re: How should I interpret my ABX tests?

Reply #6
I should also say that the 256 and 128 tests were made with no long pauses. With that being said Id like to say for those who were deciding if 128 cvbr is good quality (like I was for months) for mobile devices, yes it is and is the one im picking for my mobile even though I could discern the small diferences in higher bitrates. A nice user in this forum ThaCrip gave such good info about encoding for mp3, opus and aac and I agree with him so thnk u. This was just something Id like to say but, if anyones wondering, the difference I heard was so minimal that imo I wont go any higher than 256 (if I have enough space) and no lower than 128 or 96 if space is critical. Sorry for being messy juts wanted to share my thoughts and to have my answer clarified. Hope Im not breaking any rules tho bye

IgorC had a fairly recent test from Oct 2020 with Apple AAC @ 192kbps which shows it's not far from perfection lacking four samples as shown here... https://hydrogenaud.io/index.php?topic=120007.0

stuff like that to me says that with Apple AAC (AAC-LC) it seems largely pointless to use beyond 192kbps tops and I suspect one can go down a fair amount lower (like say 128kbps or 160kbps) without quality dropping much, especially when your just sitting back and enjoying the music it seems unlikely someone would notice the difference. or if they can, it's probably not enough to bother them in real world use, unless they are some kind of hardcore perfectionist where it's more them knowing it's not perfectly transparent, more than it actually mattering.

p.s. thanks for the comment ;)

Even 96kbps apple aac was pretty much acceptable.

Yeah, given listening tests from 2011 (which should still be valid today since I don't think there have been any sound quality tweaks with Apple AAC since) which shows Apple AAC @ 96kbps in CVBR mode at about 4.4 which ain't all that much worse than higher bit rate settings. so off the top of my head... 96kbps or 128kbps is almost certainly the sweet spot for Apple AAC (AAC-LC) as I imagine beyond those two options there are diminishing returns to where, at the most, 160kbps or 192kbps is even worth considering unless one does not care about efficiency in the slightest at which point they might as well just use what Apple themselves uses which I think is CVBR @ 256kbps.

but like others have said... nowadays, storage space is no longer at a premium price which means most people simply won't care about this stuff and will likely use whatever bit rate is available to them on their songs since they won't be forced to care like they did in the old days where if they ignored the inefficient bitrates, they would run out of storage space for their songs quickly and then wonder how to fit more onto their device. but when just about everyone can afford to buy a 32-128GB memory card for a dirt cheap price (or close enough to that standard) nowadays, it's unlikely to matter whether they are using 128kbps or 256kbps etc unless they got a really massive collection etc. so in my opinion, it's somewhat like once storage space of about 8-16GB became common (and at a cheap enough price), people tend not to care about bit rates much anymore since with that level of storage space you can fit plenty of music on it with low-or-high bitrates to where one is not going to have to swap music in-and-out of their device much anymore unlike in the old days. so those people are no longer inconvenienced enough to be concerned with tweaking bitrates of their music.

p.s. I would imagine for those using 96kbps, it's probably safer to use CVBR even though TVBR seems to be more of the common option and is more efficient. but I think it's possible that bit rate can dip a bit too low here and there with TVBR where as CVBR helps prevent this. still, I don't think there is any statistically significant difference between TVBR and CVBR at the same bit rates in general (i.e. I can't really claim TVBR or CVBR is flat out superior to the other) even though based strictly on score, CVBR tends to score a bit higher. so on some level it's the slightly safer choice on sound quality and I figure it's not a bad idea to use especially when your already at the lowest bit rate I suggest using for Apple AAC (AAC-LC) which is 96kbps. so I guess the whole TVBR vs CVBR at the same bit rate basically comes down to whether you prefer a slightly more efficient storage space (i.e. TVBR) or 'maybe' a touch safer on sound quality (i.e. CVBR). but with today's storage space, and there is a minimal enough difference between the two in general, it does not really matter which you choose even though some people around here might still slightly favor TVBR while others might go with CVBR. but differences are minimal enough so someone setting up Foobar2000 with the QAAC (Apple AAC) encoding, will probably just use the defaults which is TVBR mode.

I am impressed at how much music you can get out of 48 kbit/s [with Opus/exhale] or even less. Not saying it sounds awesome, I am saying I am impressed - recall that MP3 has a minimum of 32, anything less was considered to be so totally useless that it wasn't even allowed in the format specification.

Yeah, I played around with Opus @ 32kbps and 48kbps a while ago and considering the very low bit rates the overall sound quality is quite respectable.

still, I tend to be of the mindset with Opus that 64kbps is more of a safe minimum since it's only 16kbps more than 48kbps (which at these bit rates storage space savings would be minimal enough not to be too concerned with unless one is very tight on space and has a rather massive collection etc) and cleans up the sound a fair amount as I don't mind using Opus @ 64kbps on random music although if one has the storage space, they probably best off using 96kbps and forget about it since I feel that's probably the point where things peak before diminishing returns set in by going to higher bit rates especially given this public listening test from 2014 which shows Opus is quite strong @ 96kbps... http://listening-test.coresv.net/results.htm ; so I suspect any average-ish person will not complain about Opus @ 96kbps and it's quite efficient in bit rate to. basically Opus @ 96kbps is one of those good 'set-it-and-forget-it' types of things. like a good default/go-to option for most people in my opinion.

I suspect many people will probably find Opus @ 64kbps to be easily good enough when just sitting back and enjoying their music as it does not sound obviously worse than higher bit rates on typical equipment most people use. hell, I would not be surprised if some people find 32kbps or 48kbps to be passable/good enough with Opus since, like I was saying, until something becomes really obvious, many people just continue to enjoy the music and not really notice any declines in sound quality etc.
For music I suggest (using Foobar2000)... MP3 (LAME) @ V5 (130kbps). NOTE: using on AGPTEK-U3 as of Mar 18th 2021. I use 'fatsort' (on Linux) so MP3's are listed in proper order on AGPTEK-U3.

Re: How should I interpret my ABX tests?

Reply #7
Yeah, I played around with Opus @ 32kbps and 48kbps a while ago and considering the very low bit rates the overall sound quality is quite respectable.

still, I tend to be of the mindset with Opus that 64kbps is more of a safe minimum since it's only 16kbps more than 48kbps (which at these bit rates storage space savings would be minimal enough not to be too concerned with unless one is very tight on space and has a rather massive collection etc)

Opus at 32 was to be impressed over it being even possible ... and I cannot imagine being TOS8'ed over saying that yes I enjoyed getting impressed more than I enjoyed the actual sound quality.

Your choice of 64 over 32 or 96 determines how many weeks of music you get on your cell phone :-o

Re: How should I interpret my ABX tests?

Reply #8
Yeah, I played around with Opus @ 32kbps and 48kbps a while ago and considering the very low bit rates the overall sound quality is quite respectable.

still, I tend to be of the mindset with Opus that 64kbps is more of a safe minimum since it's only 16kbps more than 48kbps (which at these bit rates storage space savings would be minimal enough not to be too concerned with unless one is very tight on space and has a rather massive collection etc)

Opus at 32 was to be impressed over it being even possible ... and I cannot imagine being TOS8'ed over saying that yes I enjoyed getting impressed more than I enjoyed the actual sound quality.

I fully agree with this. Sometimes it's not the absolute sound quality that impresses you, but what is now possible at such low rates. Opus at 64 was the first one to blow my mind, I actually worried that I had maybe damaged my hearing because I couldn't readily identify problems with this codec at 64kbps :-D

Re: How should I interpret my ABX tests?

Reply #9
stuff like that to me says that with Apple AAC (AAC-LC) it seems largely pointless to use beyond 192kbps tops and I suspect one can go down a fair amount lower (like say 128kbps or 160kbps) without quality dropping much, especially when your just sitting back and enjoying the music it seems unlikely someone would notice the difference. or if they can, it's probably not enough to bother them in real world use, unless they are some kind of hardcore perfectionist where it's more them knowing it's not perfectly transparent, more than it actually mattering.

I'd go even further and say no one is going to really notice, or 'be bothered' by the difference 'in use'. Yes, I fully understand the need for testing but are we testing only under ideal conditions, rather than real world ones, and then still not taking advantage of compression levels that provide reasonably adequate sound in actual use? I guess my question is are any of these tests being done using phones in a normal listening environment? Do we have any of that kind of data?
Quis custodiet ipsos custodes?  ;~)