Skip to main content

Topic: Multi-Format Listening Test, Opinions on Bitrates? (Read 3628 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • ezlez
  • [*]
Multi-Format Listening Test, Opinions on Bitrates?
I'm going to conduct a multi format listening test with ABC/HR to determine an overall formidable format, and I wanted know some opinions and recommendations on what fixed bitrates I should use.

Some people argue that high/mid bitrates for any format would easily reach transparency/high fidelity and would thus make the results less significant when comparing them with other formats. In all, I would like to know what bitrates you would like to see in the test. I just need 5 different bitrates to test for each format.

Here are the formats that are going to be tested:
•   MP3: LAME 3.98.2 through Audacity 1.3.11 Beta
•   Ogg Vorbis: aoTuV beta 5.7  though OggdropXPd 1.9.0
•   AAC: Nero 1.5.1  through foobar2000 1.0 , Quicktime/Apple  through iTunes 9.0.3.15
•   Windows Media Audio: Standard & Pro Versions through Windows Media Player 11 ,
•   Musepack: Musepack SV8  through foobar2000 1.0


And, here are my original plans for the bitrates (Note: All of the formats are compressed through VBR, with the exception of WMA Pro 10):

MP3
  • V8 (65 – 105 kbps) [Low]
  • V6 (95 – 135 kbps)
  • V4 (145 – 185 kbps)
  • V2 (170 – 210 kbps)
  • V0(220 – 260 kbps) [Highest]


Ogg Vorbis
  • q 2.0 ( ~96 kbps) [Low]
  • q 4.0 (~128 kbps)
  • q 6.0 (~192 kbps)
  • q 8.0 (~256 kbps)
  • q 10.0 (~499.8 kbps) [Highest]


AAC
  • Encoder #1: Nero 1.51 through foobar2000
    • Q .20 ( ~48 kbps) [Low]
    • Q .40 (~125 kbps)
    • Q .60 (~225 kbps)
    • Q .80 (~325 kbps)
    • Q 1.0 (~400 kbps) [Highest]


  • Encoder #2:Quicktime/ iTunes (Apple) AAC Encoder through iTunes
    • ~80 kbps
    • ~128 kbps
    • ~192 kbps
    • ~256 kbps
    • ~320 kbps


    Windows Media Audio (Encoder: Windows Media Player 11)
    • Format #1: WMA Standard
    • 40 – 75 kbps
    • 50 – 95 kbps
    • 85 – 148 kbps
    • 136 – 215 kbps
    • 240 – 355 kbps


    • Format #2: WMA Professional
    • 64 kbps
    • 96 kbps
    • 128 kbps
    • 160 kbps
    • 192 kbps


    Musepack (SV8)
    • Q3 (~90 kbps)
    • Q4 (~128 kbps)
    • Q6 (~205 kbps)
    • Q8 (~275 kbps)
    • Q10 (~350 kbps)





  • diacon
  • [*]
Multi-Format Listening Test, Opinions on Bitrates?
Reply #1
The extent of your test is absolutely crazy! It is so exaggerated that one could even question your competence to take this further at all. How many samples do you want to test, one? You seem to have zero experience. Why don't you thoroughly read about what others have tried so far first, before endowing the public with this half-baked want-it-all?

  • ezlez
  • [*]
Multi-Format Listening Test, Opinions on Bitrates?
Reply #2
The extent of your test is absolutely crazy! It is so exaggerated that one could even question your competence to take this further at all. How many samples do you want to test, one? You seem to have zero experience. Why don't you thoroughly read about what others have tried so far first, before endowing the public with this half-baked want-it-all?


I apologize for not adding more detail to the topic. I originally posted a topic a couple days regarding the details and plans of the tests, but the plans for the tests themselves are not fully complete. The main reason why I posted both topics (including this one) was to receive opinions and criticism regarding to plans, in order to make them better. The main topic I posted can be found here: http://www.hydrogenaudio.org/forums/index....showtopic=79087  . As for the samples question you asked, of course not. As stated in the main topic i posted, I would use a bare minimum of three different samples for each adjustable bit rate, making it at least 15 tests for each codec. The reason why I said that I will use at least 3 different sample for the 5 adjustable bit rates for each codec is because I don't want to fatigue any of the subjects taking the tests in case the event happens, meaning that there is definitely a strong possibility that I will use more samples, at most 13. I'm using Roberto Amorim's/rjamorim's document on ABC/HR Testing as a guide to the tests that I will conduct. The document can be found here: http://www.rarewares.org/rja/ListeningTest.pdf

  • sauvage78
  • [*][*][*][*][*]
Multi-Format Listening Test, Opinions on Bitrates?
Reply #3
When I see the funny methodology, whatever you find, I will give no credit to the result
Good luck with Nero AAC ~400 kbps  I can hardly ABX half of that bitrate...

Edit: Just a clue: testing 192Kbps is ten time longer than testing 96Kbps. The time you spend is exponentially linked to the rise of the bitrate. So your intention to test 320Kbps & above is obviously insane. It's either you'll give up or you'll become crazy before the end of the test.

... too many codecs, too many bitrates, too high bitrates, no samples.
  • Last Edit: 02 March, 2010, 12:18:15 AM by sauvage78
Rip & Check: EAC Secure [Low/C2]+CUETools [AR Confidence 2+]
Desktop: Flac -4 (for Speed) | CDImage+CUE with F2K
DAP (Android): Opus 128Kbps | Tracks with AIMP
Video: Opus 128Kbps (2.1) | VP10 (2160p60) Asap !!! | Matroska

  • ezlez
  • [*]
Multi-Format Listening Test, Opinions on Bitrates?
Reply #4
When I see the funny methodology, whatever you find, I will give no credit to the result
Good luck with Nero AAC ~400 kbps  I can hardly ABX half of that bitrate...


I realize that testing anything above 250 kbps for any codec would be highly transparent and useless, but the reason why I posted this topic in the first place was to gather opinions on what bitrates would be acceptable for each format. The original plans were a draft, and I am not, of course, using them. I value the criticism and all, but I just want to know what the community wants see in the tests.

  • ojdo
  • [*][*][*][*][*]
Multi-Format Listening Test, Opinions on Bitrates?
Reply #5
The problem with so many codec-bitrate-combinations is that you would need hundreds of listeners to yield a result that has statistical value. With ~10 subjects the ratings will hardly have any significance. I would recommend you to either stick to one codec and try to evaluate at which bitrate it becomes sufficiently transparent OR compare multiple codecs at a certain bitrate (which one depends on what you want your test to be about) to see which one performs best. To try BOTH seems too ambitious to me if you don't have means to motivate many test subjects to participate.

Concerning the samples: that's how you want the test to be positioned. You could concentrate on popular music ("charts") or focus on classical music. I would avoid known problem samples of a certain codec as it would erase all remaining comparability (see above) among codecs.

Edit: corrected spelling
  • Last Edit: 02 March, 2010, 03:42:37 AM by ojdo

  • stephanV
  • [*][*][*][*]
Multi-Format Listening Test, Opinions on Bitrates?
Reply #6
Again I ask you why you want to do this test. You haven't really set a clear goal. If you say what kind of answer you are looking for, it is much easier to establish a methodology.

Example: You don't need to test the very low settings if you are looking for a transparency setting.

I'm sorry to be blunt, but I think if you have such questions I'm doubting your capabilities to setup a test of this scale let alone getting any meaningful results from it.
"We cannot win against obsession. They care, we don't. They win."

  • ezlez
  • [*]
Multi-Format Listening Test, Opinions on Bitrates?
Reply #7
Again I ask you why you want to do this test. You haven't really set a clear goal. If you say what kind of answer you are looking for, it is much easier to establish a methodology.

Example: You don't need to test the very low settings if you are looking for a transparency setting.

I'm sorry to be blunt, but I think if you have such questions I'm doubting your capabilities to setup a test of this scale let alone getting any meaningful results from it.


My goal with these tests is to evaluate different codecs within the same parameters (quality settings; bitrates included) and to try to figure what codec performs better than others within different quality settings. I admit that I don't have much experience with listening tests, or a lot of experience with different lossy codecs for that matter. I now realize that basing listening tests on arbitrarily chosen bitrates is pointless, and the focus should go towards quality settings/modes instead.
Now that I know this, I want to shift the focus towards quality modes instead, but at the moment I don't know at which settings for each codec I should set forth to maintain them on an equal basis for the tests. I want to include 5 quality modes for each codec; low through high. As for samples, I'm going with music of different genres (Classical, Pop, Rock, etc.), each 15 seconds each, and 12 samples per quality mode (60 tests per codec). Again, these are plans in the making, and I'm always willing to improve them