Skip to main content

Topic: Multiformat 128 kbps Listening Test (Read 239700 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
Multiformat 128 kbps Listening Test
Reply #725
Quote
Sebastian,

The samples are looking great!

Average bitrate between codec is excellent. 

I look so much forward to performing this test 

Keep up the good work!
[a href="index.php?act=findpost&pid=346649"][{POST_SNAPBACK}][/a]


Thanks. Well, the problem is that with the samples I have ATM, LAME and WMA Pro exceed the 10% tolerance margin (max is 140.8 kbps). Nero AAC is only 0.1 kbps away from it. Maybe the average bitrate can be lowered with a movie dialog sample for example.
  • Last Edit: 30 November, 2005, 03:59:48 PM by Sebastian Mares

  • Alex B
  • [*][*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #726
Some comments about the bitrate table:

Did you try this Nero encoder with full tracks? What is the used setting? Does it produce a comparable average bitrate with the other encoders when a big amount of varied complete tracks is encoded?

BTW, did you use -q 4.25 for Vorbis? (I suppose the other encoders have only one possible setting.)

Multiformat 128 kbps Listening Test
Reply #727
For Vorbis, I used -q 4.
For Nero AAC, I used the recommended streaming profile. On my music collection, it produced an average of 132 kbps.
  • Last Edit: 30 November, 2005, 04:21:51 PM by Sebastian Mares

  • Alex B
  • [*][*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #728
In my opinion Vorbis -q 4.25 would be closer with LAME V5n and WMA Pro 50 (I  have no large scale experience of WMA Pro 50). The AAC encoders work differently.

guruboolez chose -q 4 for various tracks and 4.25 for classical in his test, but according to the results I have seen now I think 4.25 would have been a correct setting for all genres. Perhaps 4.2 could be used too. I could try if it makes any difference on my test tracks.

[span style='font-size:7pt;line-height:100%']Edit: typo[/span]
  • Last Edit: 30 November, 2005, 04:48:14 PM by Alex B

  • yahknow1
  • [*][*]
Multiformat 128 kbps Listening Test
Reply #729
I'm new to this (still) and excuse this question if it silly or I'm jumping ahead of things, but, when I try saving the clips, they all come up as "index.php"? What am I doing wrong?

Multiformat 128 kbps Listening Test
Reply #730
Quote
I'm new to this (still) and excuse this question if it silly or I'm jumping ahead of things, but, when I try saving the clips, they all come up as "index.php"? What am I doing wrong?
[a href="index.php?act=findpost&pid=346665"][{POST_SNAPBACK}][/a]


Which browser are you using?

  • yahknow1
  • [*][*]
Multiformat 128 kbps Listening Test
Reply #731
Quote
Quote
I'm new to this (still) and excuse this question if it silly or I'm jumping ahead of things, but, when I try saving the clips, they all come up as "index.php"? What am I doing wrong?
[a href="index.php?act=findpost&pid=346665"][{POST_SNAPBACK}][/a]


Which browser are you using?
[a href="index.php?act=findpost&pid=346666"][{POST_SNAPBACK}][/a]

IE 6.0...never ran into troubles like this with any other sample?

  • user
  • [*][*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #732
Some general thoughts about listening tests:

- the test samples of various format contenders themselves don't need to average to a certain bitrate. Any higher or lower bitrate on samples will work, because that is the goal to be tested: "Does the encoder a good job on distributing bitrate to frames ?"
That means, if we have a broad overview, of the average bitrate a format with specified quality setting averages at,
then it is fine. The samples can vary  a lot from that targeted bitrate. That is vbr.

- Of course, it is another question, which samples you select. Only killer samples, or normal music, or some mix.

- I suggest selecting samples, which result to ca. the target bitrate, but also samples, which go way above the target bitrate with certain encoders,
but also samples, which result clearly below target bitrate with certain encoders.
So, you test "easy to encode" samples, "average to encode" samples, and "difficult to encode" samples.
It will then be interesting, if various encoders behave differently regarding bitrate sample distribution, and more important: "do they have success, eg. when they give certain samples only a low bitrate, or lower bitrate than other contenders ?"

- Maybe somebody here with  a widefold music CD collection, covering several genres, from pop, jazz, classic, rock, techno, audio books, metal etc., should mix some albums, and keep them as reference in a Lossless format, for test encodes with current and upcoming formats/encoders, settings.
So, that we have some HA-reference, which setting yields to which average bitrate.
Eg. vorbis q4 , or q4.25, a q-step of 0.25 might have already some bigger influence in the 100-128k bitrate area, and it should be decided & tested with the vbr formats, that for each format a setting is chosen, which does yield on a broad average of music to same bitrate (with only few percent difference).
Guruboolez did something like this with his classic collection, don't know, if he has some other CDs/genres, which could make it complete.
I could help here, have various classic, jazz, pop, rock etc., but I don#t have installed every format yet. And don#t have the internet/program knowledge, to present the results such good, how guru does it. Or does foobar give everything automatically, or probably a connection/script necessary, to combine foobars values with excel ?
Eg., for 3.97 Lame -V5 --vbr-new, I can confirm, that 130 kbit/s is an average, but it depends  a lot on albums/music. Eg., if you would ask somebody, who owns mainly modern clip-compressed pop music, he would say, that lame 3.97 -V5 -vbr-new will yield to 140, not 130k.
www.High-Quality.ch.vu -- High Quality Audio Archiving Tutorials

  • sehested
  • [*][*][*][*]
  • Members (Donating)
Multiformat 128 kbps Listening Test
Reply #733
Quote
Thanks. Well, the problem is that with the samples I have ATM, LAME and WMA Pro exceed the 10% tolerance margin (max is 140.8 kbps). Nero AAC is only 0.1 kbps away from it. Maybe the average bitrate can be lowered with a movie dialog sample for example.
All the codec is on the high side of the target bitrate of 128 kbps and indeed - as you point out - some are outside the 10% tolerance we have defined.

However the difference between lowest average bitrate (137.0 iTunes) and highest average bitrate (143.1 WMA Pro) is only 4%, which is excellent.

I see no need to adjust the average bitrate for any of the codec.

Although some could argue that the average bitrate is closer to that of a 160 kbps test, I would prefer to think of it as a 128+ kbps test, since 128 kbps VBR results in an average bitrate slightly above 128 kbps.

Multiformat 128 kbps Listening Test
Reply #734
I'd also keep the settings as-is and check the final result when I have all samples. If some codecs continue to fall outside 140.8 kbps, I might replace a sample or two.

  • sehested
  • [*][*][*][*]
  • Members (Donating)
Multiformat 128 kbps Listening Test
Reply #735
What's with the sample Cockney Rebel ("Sebastian")?

The file sample I downloaded is WavePack 855 kbps but the sound has severe artifacts.

  • user
  • [*][*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #736
Quote
I'd also keep the settings as-is and check the final result when I have all samples. If some codecs continue to fall outside 140.8 kbps, I might replace a sample or two.


I don't understand this method. The vbr codecs are free in their behaviour.
Only importance of the to be tested quality settings is, that the target bitrate is nearly the same for all contenders, ie. 128k+- over various music genres.
www.High-Quality.ch.vu -- High Quality Audio Archiving Tutorials

Multiformat 128 kbps Listening Test
Reply #737
Quote
What's with the sample Cockney Rebel ("Sebastian")?

The file sample I downloaded is WavePack 855 kbps but the sound has severe artifacts.
[a href="index.php?act=findpost&pid=346678"][{POST_SNAPBACK}][/a]


Hmm... It's from "Rock Legends - All-Time Greatest Rock Ballads Volume 1" (track 9). I copied it with EAC, extracted the 35 seconds and encoded them with WavPack.
What kind of artifacts are you talking about? Some that are typical to lossy encoders or something else?

  • sehested
  • [*][*][*][*]
  • Members (Donating)
Multiformat 128 kbps Listening Test
Reply #738
Quote
What kind of artifacts are you talking about? Some that are typical to lossy encoders or something else?
Distortion on singers voice most noticably from 0:09 to 0:20

Multiformat 128 kbps Listening Test
Reply #739
Quote
Quote
What kind of artifacts are you talking about? Some that are typical to lossy encoders or something else?
Distortion on singers voice most noticably from 0:09 to 0:20
[a href="index.php?act=findpost&pid=346682"][{POST_SNAPBACK}][/a]


The tracks were copied from vinyl - maybe that's the reason.

Multiformat 128 kbps Listening Test
Reply #740
Quote
Quote
I'd also keep the settings as-is and check the final result when I have all samples. If some codecs continue to fall outside 140.8 kbps, I might replace a sample or two.


I don't understand this method. The vbr codecs are free in their behaviour.
Only importance of the to be tested quality settings is, that the target bitrate is nearly the same for all contenders, ie. 128k+- over various music genres.
[a href="index.php?act=findpost&pid=346679"][{POST_SNAPBACK}][/a]


Yes, but if there are a lot of difficult samples and the average bitrate exceeds 128 kbps + 12.8 kbps, it would be better to replace one of the complex samples so the average bitrate remains near what's tested.
Nero for example produces an average of 131-132 kbps with my music collection and that's what Ivan said, too. With these particular samples however, the bitrate is boosted to more than 140 kbps. That's why I think that replacing one sample is appropriate.

  • rjamorim
  • [*][*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #741
Quote
then the test won't be serving its purpose. as stated, an econder would adapt to a whole track diferently than to a 30 secs clip.[a href="index.php?act=findpost&pid=346637"][{POST_SNAPBACK}][/a]


You could conduce a test yourself. I believe Argentinian law is much nicer than german law, in that aspect.
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org

  • Alex B
  • [*][*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #742
I have the Steve Harley & Cockney Rebel - The Cream Of compilation which includes Sebastian. I could quickly access my LAME 3.90.3 --alt-preset extreme encoding. Steve's voice is "distorded" on my version too. Perhaps the track is not a good choice for an encoder test.

I would like see a strong female voice included. A jazz or opera singer who can sing load and high would be good. Someone who can break glasses with her voice... this kind of sample might produce low bitrates and be difficult for the encoders at the same time.

  • ErikS
  • [*][*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #743
As ff123, user and I among others suggested earlier, it would be good to try to achieve a bitrate distribution of the sample set which resembles the distribution from a large random set. So perhaps add a few samples which lower the average a bit...?

  • Alex B
  • [*][*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #744
As said before, the used encoding options should be determined by checking the bitrates of a large amount of varied complete tracks and after that kept.

The samples should represent many musical genres and be difficult enough to produce audible artifacts. This is a listening test and no one should be interested what bitrates the individual encoded samples are. I don't think the bitrates should even be checked. The encoders should be left on their own to do the best they can with the sample material.

Naturally, the results of the preceding large-scale bitrate testing should be published.

  • Alex B
  • [*][*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #745
I updated my bitrate table with Vorbis -q 4.20 and gathered all previous results in the same table:

bitrates_public2.xls

  • ChiGung
  • [*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #746
Quote
Quote
Sadly we cannot do 2-pass because that would required to be executed on full tracks and Sebastian rightly do not want to get embroiled on copyright issues.
Sorry if already asked, but i don't think anybody would get into trouble for doing this, i mean, he won't keep the whole song, right? He would just encode the file, decode, cut, and delete...

I think it was resolved that the differences concerned are similar in a kind and in severity to Neros ABR encoding choice. iirc all devs who expressed an opinion see little problem with 2passing a joined sample corpus for Wma stds encode if its wished to include wma std in the test. ff123 agreed this would be fine if the samples where of average complexity, id suggest if they're mean achieved bitrate of the 'normal' vbrs doesnt match the target bitrate the 2pass could target the mean achieved bitrates of the other encoders to fairly compensate.
All in all, its not some to worry about at this stage, whatever the samples used a suitible Wma std targeting process is possible if required.
no conscience > no custom

  • ErikS
  • [*][*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #747
Quote
and be difficult enough to produce audible artifacts.
[a href="index.php?act=findpost&pid=346720"][{POST_SNAPBACK}][/a]

I'm repeating myself, but I still wonder how you decide that a track is difficult...?

  • Alex B
  • [*][*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #748
Quote
Quote
and be difficult enough to produce audible artifacts.
[{POST_SNAPBACK}][/a]

I'm repeating myself, but I still wonder how you decide that a track is difficult...?
[a href="index.php?act=findpost&pid=346736"][{POST_SNAPBACK}][/a]

I guess I'm the wrong person to answer to this question. Check my "failure" here:

[a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=39233]A little VBR ~88 kbps ABX-test[/url]

  • NoXFeR
  • [*][*]
Multiformat 128 kbps Listening Test
Reply #749
Vorbis should not be tested with -q 4.00, but in around 4.25. See the bitrate estimate thread for reasons why.