Skip to main content

Topic: Public Listening Test [2010] (Read 121128 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • .alexander.
  • [*][*]
Public Listening Test [2010]
Reply #125
That won't happen. I promise you.
Why?


And why do they use straight tracks for 100m sprint?

99% of HA community are on the same page that  codec should be tested without any bitrate restriction while produce the same bitrate on enough big amount of files.


What is your personal opinion how big amount of files is neccessary to make encoders produce same avarage bitrate after 30' second? Test samples are likely to be short and this can particulary be a reason to get biased bitrates.

  • C.R.Helmrich
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #126
What is your personal opinion how big amount of files is neccessary to make encoders produce same avarage bitrate after 30' second? Test samples are likely to be short and this can particulary be a reason to get biased bitrates.

http://www.hydrogenaudio.org/forums/index....showtopic=77932 55 CDs should suffice

Now, IIRC, at ~128 kbs, all encoders should run at 44.1 kHz by default, am I right? So if we take that bitrate, the sampling rate issue should disappear.

Chris

Oh, I forgot: Who says the number of encoding parameters needs to equal the number of data elements? Even something such as the decision whether to use short blocks or a long block already requires a handful of parameters.
  • Last Edit: 03 February, 2010, 02:34:46 PM by C.R.Helmrich
If I don't reply to your reply, it means I agree with you.

  • rpp3po
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #127
http://www.hydrogenaudio.org/forums/index....showtopic=77932 55 CDs should suffice

Now, IIRC, at ~128 kbs, all encoders should run at 44.1 kHz by default, am I right? So if we take that bitrate, the sampling rate issue should disappear.


That was 55 CDs of classical music. You know what that means. With Q 59 and 55 CDs of 'emesesk' music you end up in the 140-145kbit/s range. And beginning from the next smaller setting, Q 58, downsampling is the default for Quicktime. So no, depending on the encoded content you are not right. But since it seems we won't individually alter Q values to get each track as close to ~128kbit/s as possible, anyway, it might not really matter. Depending on the selection of samples, the global Q value might be higher than 59 and then downsampling is really not an issue.

I have 87 lossless cross-genre albums on my notebook right now. I'll check in a couple of minutes how close its average is going to get to 128kbit/s at Q 59.

PS I have finished the 87 albums. I get a collection average of 129 kbit/s (median at 134 kbit/s) for Q 59. Biggest surprise is the MFSL edition of "A Night in Tunesia" by Art Blakey and the Jazz Messengers from 1960, with an album average of 164 kbit/s. On the low end are Miles Davis recordings from the 50s with ~72kbit/s average.
  • Last Edit: 03 February, 2010, 08:08:38 PM by rpp3po

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #128
Guruboolez (classic, 55 CDs) - 127 kbps
IgorC (varoius, 31 CDs )  -  129 kbps
rpp3p0 (various, 87 CDs) - 129 kbps

The invterval Q59-Q68 produces exactly the same bitrate (Q59, Q60..Q65...).
My suggestion is --tvbr 60 --highest should be tested

It's time to find the same bitrate for other encoders.
  • Last Edit: 04 February, 2010, 09:57:21 AM by IgorC

  • rpp3po
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #129
It's time to find the same bitrate for other encoders.


I'm testing different "target" bitrate settings for the CVBR mode right now on the same set. It's not exactly a target bitrate, because the constraint seems to be enforced asynchronously. It rather behaves as "don't fall below that bitrate on average and scale up moderately when required". So I expect the matching target for CVBR to be in the range of 117-120 kbit/s. I'm verifying this right now, but it needs some time, since about 30GB need to be processed in each run.

If someone else could experiment with Nero AAC q values, that would be great. I'm willing to test that result against the 87 CD set. But since it is kind of slow over VMware, I can't do the experimenting myself.

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #130
itunes 128 CVBR produces 131 kbps on my collection of 31 CDs.  http://www.hydrogenaudio.org/forums/index....st&p=683265

  • rpp3po
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #131
OK, tests are finished. To get equal results to 'TVBR --highest Q(59-68)' = 129kbit/s, I had to use a target bitrate of 121 kbit/s for 'CVBR --normal'.

This is the list of the 87 albums' genres:

Rhythm & Blues: 2
Electronic: 11
Folk: 1
Jazz: 39
Classical: 4
Metal: 2
Pop: 9
Rock: 16
Soul: 3

Jazz is a little overrepresented, but it spans music from six decades with great variety, so I think it's ok.

I found it very interesting that the bitrate distribution is quite different for CVBR, which I did not expect to that degree. Art Blakey doesn't lead anymore (now 133kbit/s), but Boulez: RĂ©pons at 136kbit/s. The lowest is now Radiohead with "Motion Picture Soundtrack" from Kid A at 83kbit/s (TVBR 86kbit/s), but there are only 4 tracks below 100kbit/s at all! The median is at 130kbit/s.

Summary:

TVBR Q(59-68) --highest: average 129 kbit/s, median 134 kbit/s, min 72 kbit/s, max 168 kbit/s
CVBR 121 --normal: average 129 kbit/s, median 130 kbit/s, min 83 kbit/s, max 136 kbit/s

If I would plot a graph of the results, the edges would be very steep for CVBR, only very few tracks are found at the extremes, most form a flat top in the middle.

For TVBR, even though the bitrates goes up as high as 168 kbit/s, there are very many tracks both at the upper and the lower end and pretty equal distribution over the whole (though much larger) bitrate range.
  • Last Edit: 04 February, 2010, 05:37:59 PM by rpp3po

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #132
rpp3po
Thank you for the results.
CVBR 121 produces the same bitrate as CVBR 128.

At least it's confirmation for my previous finding that CVBR 128 and TVBR 60 have comparable bitrate ~130 kbps.

  • rpp3po
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #133
CVBR 121 produces the same bitrate as CVBR 128.


That's not correct. Each CVBR target bitrate results in another filesize. 121 is different from 128, also from 122, 123, etc. Only TVBR changes at every 8th increment.

We also have slightly different results.
  • Last Edit: 04 February, 2010, 10:22:20 PM by rpp3po

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #134
Impossible.

  • rpp3po
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #135

  • IgorC
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #136
Already did it.

From nao's site:
Code: [Select]
Cannot encode with odd bitrates, like 125kbps


It should be bug or something wrong ... on your side.

  • rpp3po
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #137
It should be bug or something wrong ... on your side.


Quicktime on the Mac doesn't have any problem with odd bitrates, it's rather a bug of the Windows port. Both nao's XLD and Apple's afconvert front-end don't have this problem.

If your system cannot encode 121 kbit/s, then try even numbers as 120 and 122. You will see, they produce different file sizes.

PS Quicktime's CVBR target bitrate, is a minimum average, not a real target, as described above. If we compare TVBR 60 against CVBR 128, we will give CVBR an unfair advantage of up to 5% without necessity. Much of TVBR's efforts to save bitrate on non-complex passages would be superseded.

PPS What do you think have I done the whole day? Encode several times 30GB, at different CVBR settings, just to get the same result each time?
  • Last Edit: 04 February, 2010, 11:02:47 PM by rpp3po

  • Alex B
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #138
So what means are you guys using for measuring and calculating the average bitrates? Can the bitrates that programs report be blindly trusted?

I tried demuxing MP4/M4A to raw AAC with Yamb/MP4Box, but surprisingly that actually increased the file size a bit. Perhaps MP4 somehow packs the stream more effectively and the despite the additional MP4 container structure the overall file size can be smaller than the size of the raw stream.

Could the AAC developers explain how AAC is stored in the container and how the bitrate values are saved in the "atoms"? Does the MP4 container structure have a defined size overhead that can be detected or calculated?

EDIT

BTW, is test now settled to be a ~128 test?
  • Last Edit: 05 February, 2010, 03:17:51 AM by Alex B

Public Listening Test [2010]
Reply #139
Could someone give a summary to what has been decided so far? Is the discussion about codecs and their settings over? I am asking because there was already a samples thread somewhere and usually you choose samples after choosing codecs (at least it was so in the past).

  • muaddib
  • [*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #140
I tried demuxing MP4/M4A to raw AAC with Yamb/MP4Box, but surprisingly that actually increased the file size a bit. Perhaps MP4 somehow packs the stream more effectively and the despite the additional MP4 container structure the overall file size can be smaller than the size of the raw stream.
Could the AAC developers explain how AAC is stored in the container and how the bitrate values are saved in the "atoms"? Does the MP4 container structure have a defined size overhead that can be detected or calculated?

Raw AAC streams have frame structure just like mp3, so there is an overhead of a header for each frame.

  • Alex B
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #141
Oh, I see. That explains the slightly bigger size of demuxed AAC.

Do you have any opinion about the proper way of measuring the MP4-AAC audio file bitrates? For instance, is it fine to use foobar?

I have tried also Mr QuestionMan and Mediainfo, but the results are not always identical.


[!--sizeo:1--][span style=\"font-size:8pt;line-height:100%\"][!--/sizeo--]EDIT

There is an excess "the" word in my previous reply: ... and the despite the additional MP4 container structure... . The forum software doesn't allow me to edit it anymore. In my opinion the replies should be editable longer - 24 h or so.[/size]
  • Last Edit: 05 February, 2010, 05:11:14 AM by Alex B

  • rpp3po
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #142
Regarding the recently discussed, determining bitrates is not an issue. IgorC just misinterpreted the line on nao's site. To end the speculation, I have attached a set of Quicktime CVBR files at 126, 127, 128 kbit/s targets (effectively in the 141-143 kbit/s range), which should be "impossible" to produce according to Igor:

Update: I had accidently uploaded CBR, files.  Now it is CVBR.
  • Last Edit: 05 February, 2010, 06:02:03 AM by rpp3po

  • Alex B
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #143
I am not speaking about QT CVBR. I am speaking about determining the bitrates in general. Obviously those programs that I mentioned read info that is stored in the MP4 headers. I am not an MP4 expert and I would like to know if that info is always correct and if I can trust some of the available tools.

You have not explained what software you use for displaying the bitrate values.


BTW, seems like you don't agree with me and C.R.Helmrich that iTunes should be used for encoding CVBR. Correct?

  • .alexander.
  • [*][*]
Public Listening Test [2010]
Reply #144
Oh, I forgot: Who says the number of encoding parameters needs to equal the number of data elements? Even something such as the decision whether to use short blocks or a long block already requires a handful of parameters.


Not me. What I said is that number of principal parameters of stream (and even data elements) is less than 100, and these are common for all competitors.

While samples are short, it will be a test of rate control volatility. And this isn't worth the efforts. IgorC rejected rpp3po' idea since it doesn't look real. But in "real life condition" bitres at the begining of fragment wouldn't be reset. And what is more important the bit reservoir state at the end of fragment wouldn't be irrelevant.

For "real life condition" use looped samples. Then pick fragments with lowest bit-length for each encoder. This way you could check whether CBR is that bad.

After all, rate control is not most intriguing part of AAC. Intra-frame stuff is of more interest.
  • Last Edit: 05 February, 2010, 05:56:23 AM by .alexander.

  • rpp3po
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #145
BTW, seems like you don't agree with me and C.R.Helmrich that iTunes should be used for encoding CVBR. Correct?


Yes. The test's goal should be AAC at 128kbit/s respectively an encoder setting that produces 128 kbit/s on average. This is true for Quicktime TVBR at Q60 and CVBR 121. In contrast iTunes' "128 kbit/s" CVBR preset produces considerably larger files on average. What is missing now, is a matching Q value for Nero.

Edit: And don't forget, we can exactly clone iTunes' encoder by using Quicktime with CVBR --normal.

Alexander, I also supported the idea to use hand picked files, to get as close to 128 kbit/s average, at first. But I don't do anymore. Let us just use a setting for each encoder that will result in a collection average of ~128kbit/s. Some contenders like Quicktime TVBR, which is only alterable in increments of 8, don't allow fine tuning or even want to downsample at some settings. This would be a mess after all. Using one preset for each encoder really seems the most practicable approach.
  • Last Edit: 05 February, 2010, 06:39:30 AM by rpp3po

  • muaddib
  • [*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #146
Do you have any opinion about the proper way of measuring the MP4-AAC audio file bitrates? For instance, is it fine to use foobar?
I have tried also Mr QuestionMan and Mediainfo, but the results are not always identical.

Bitrate from Audio stream must be taken into account in MediaInfo and not the overall bitrate from General.
Bitrate in foobar is also correct.

  • Alex B
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #147
rpp3po,

You are still not saying what do you use for measuring the bitrates...


IMHO in a VBR listening test the encoders that can't be adjusted precisely (i.e. the encoders that have only one suitable bitrate related setting) should be measured and the measured values should be averaged. After that the encoders that can be freely adjusted should be set to produce that average value (naturally always using the same test files).

For instance:

Encoders that cannot be freely adjusted:

iTunes CVBR => x kbps
QT TVBR => y kbps
CT CBR (does it have a VBR mode?) or Divx VBR (I downloaded the demo version, but I have not tried it yet and I have no idea of how it works.) = z kbps

x+y+z /3 kbps = a kbps

Encoders that can be adjusted precisely:

Nero VBR  => adjusted as close to "a" kbps as possible
  • Last Edit: 05 February, 2010, 06:36:24 AM by Alex B

  • rpp3po
  • [*][*][*][*][*]
  • Developer
Public Listening Test [2010]
Reply #148
rpp3po,

You are still not saying what do you use for measuring the bitrates...


Normally just he OS X file info. The OS supports this out of the box. For the 87 album test I have loaded the set into Foobar within VMware. For the three above CVBR 126, 127, and 128 files OS X reports 140, 141, and 143 kbit/s. Foobar: 141, 141, 143. Extracted to raw AAC and calculated by hand: 144.6, 145.1, 147.0.

So I think CVBR can be counted as freely adjustable.

As long as always the same tool is used to determine the bitrates in this test, everything should be ok. I would be fine with agreeing on Foobar.
  • Last Edit: 05 February, 2010, 07:00:44 AM by rpp3po

  • Alex B
  • [*][*][*][*][*]
Public Listening Test [2010]
Reply #149
I measured the CVBR samples you provided:

Foobar:

141, 141, and 143 kbps



Mr QuestionMan:

142, 143, and 145 kbps



MediaInfo:

The AAC bitrates: 141, 144, and 144 kbps

(Oddly only the "126" file shows a nominal bitrate. Something is not very reliable.)

Code: [Select]
General
Complete name                    : F:\Test\cvbr\cvbr126.mp4
Format                          : MPEG-4
Format profile                  : Apple AAC audio with iTunes info
Codec ID                        : M4A
File size                        : 579 KiB
Duration                        : 30s 255ms
Overall bit rate                : 157 Kbps
Writing application              : X Lossless Decoder 20100123, QuickTime 7.6.3, Constrained VBR 126 kbps
Encoding Params                  : (Binary)

Audio
ID                              : 1
Format                          : AAC
Format/Info                      : Advanced Audio Codec
Format version                  : Version 4
Format profile                  : LC
Format settings, SBR            : No
Codec ID                        : 40
Duration                        : 30s 255ms
Bit rate mode                    : Variable
Bit rate                        : 141 Kbps
Nominal bit rate                : 144 Kbps
Maximum bit rate                : 167 Kbps
Channel(s)                      : 2 channels
Channel positions                : L R
Sampling rate                    : 44.1 KHz
Stream size                      : 521 KiB (90%)
Language                        : English

General
Complete name                    : F:\Test\cvbr\cvbr127.mp4
Format                          : MPEG-4
Format profile                  : Apple AAC audio with iTunes info
Codec ID                        : M4A
File size                        : 580 KiB
Duration                        : 30s 255ms
Overall bit rate                : 157 Kbps
Writing application              : X Lossless Decoder 20100123, QuickTime 7.6.3, Constrained VBR 127 kbps
Encoding Params                  : (Binary)

Audio
ID                              : 1
Format                          : AAC
Format/Info                      : Advanced Audio Codec
Format version                  : Version 4
Format profile                  : LC
Format settings, SBR            : No
Codec ID                        : 40
Duration                        : 30s 255ms
Bit rate mode                    : Variable
Bit rate                        : 144 Kbps
Maximum bit rate                : 167 Kbps
Channel(s)                      : 2 channels
Channel positions                : L R
Sampling rate                    : 44.1 KHz
Stream size                      : 522 KiB (90%)
Language                        : English

General
Complete name                    : F:\Test\cvbr\cvbr128.mp4
Format                          : MPEG-4
Format profile                  : Apple AAC audio with iTunes info
Codec ID                        : M4A
File size                        : 588 KiB
Duration                        : 30s 255ms
Overall bit rate                : 159 Kbps
Encoded date                    : UTC 1975-01-22 15:50:31
Tagged date                      : UTC 1975-01-22 15:50:31
Writing application              : X Lossless Decoder 20100123, QuickTime 7.6.3, Constrained VBR 128 kbps
Encoding Params                  : (Binary)

Audio
ID                              : 1
Format                          : AAC
Format/Info                      : Advanced Audio Codec
Format version                  : Version 4
Format profile                  : LC
Format settings, SBR            : No
Codec ID                        : 40
Duration                        : 30s 255ms
Bit rate mode                    : Variable
Bit rate                        : 144 Kbps
Maximum bit rate                : 171 Kbps
Channel(s)                      : 2 channels
Channel positions                : L R
Sampling rate                    : 44.1 KHz
Stream size                      : 529 KiB (90%)
Language                        : English
Encoded date                    : UTC 1975-01-22 15:50:31
Tagged date                      : UTC 1975-01-22 15:50:31



Which one would you pick?

Does the OS X file info box agree with one of these?

EDIT

I saw your edit. Apparently OS X does not exactly agree with foobar, but is close.
  • Last Edit: 05 February, 2010, 07:00:46 AM by Alex B