Skip to main content

Topic: Multiformat 128 kbps Listening Test (Read 239844 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • naylor83
  • [*][*][*]
Multiformat 128 kbps Listening Test
Reply #650
Quote
Quote
No, I cannot confirm that. The settings used are going to be settings that average 128 kbps on a large batch of full songs. If the codecs reach a bitrate too high with the given samples in this thread, I will replace the one or the other sample with a different one.

That doesnt contradict what I said....
Quote
If any codec&setting's actual average bitrate for the actual test samples used fell outside target bitrate - adjustments would be made to its setting or the test samples to allow it compete with the others complying with the average bitrate requirement of the test.


Either adjusting the setting or the samples is a deviation from real world usage.
And the fairness of tweaking the test material to raise or lower a codecs utilised bitrate for the challenging corpus is obviously unideal -could benefit or be detrimental to that codecs performance.

But if you decide to do it that way, and find settings averaging 128 kbs for a wide range, then just 2pass the linked test corpus attached to the extra 'normal material' to produce your average 128kbs wma encode, and do the normal manual multipass of same material, to find out the settings for the other codecs to get you their encodes.
[a href="index.php?act=findpost&pid=346054"][{POST_SNAPBACK}][/a]


Oh - sorry - I missed that bit. I think this is getting too advanced for my walnut brain.

Just let me know when I can download the test.
  • Last Edit: 28 November, 2005, 03:13:03 PM by naylor83
davidnaylor.org

Multiformat 128 kbps Listening Test
Reply #651
Quote
Quote
No, I cannot confirm that. The settings used are going to be settings that average 128 kbps on a large batch of full songs. If the codecs reach a bitrate too high with the given samples in this thread, I will replace the one or the other sample with a different one.

That doesnt contradict what I said....
[a href="index.php?act=findpost&pid=346054"][{POST_SNAPBACK}][/a]


It was a direct reply to naylor83's question.

Quote
Quote
If any codec&setting's actual average bitrate for the actual test samples used fell outside target bitrate - adjustments would be made to its setting or the test samples to allow it compete with the others complying with the average bitrate requirement of the test.


Either adjusting the setting or the samples is a deviation from real world usage.
And the fairness of tweaking the test material to raise or lower a codecs utilised bitrate for the challenging corpus is obviously unideal -could benefit or be detrimental to that codecs performance.

But if you decide to do it that way, and find settings averaging 128 kbs for a wide range, then just 2pass the linked test corpus attached to the extra 'normal material' to produce your average 128kbs wma encode, and do the normal manual multipass of same material, to find out the settings for the other codecs to get you their encodes.

(if distibuting all of the samples blindly and losslessly is too much, just distribute the wma decodes losslessly ((if they cant be cut well)) )
[a href="index.php?act=findpost&pid=346054"][{POST_SNAPBACK}][/a]


Well, when using some settings, the encoders proved already that they produce ~128 kbps with most audio tracks. If they reach an average bitrate that is higher than 140 kbps in this test, that is a special case and that's why I see no problem replacing one particular sample that boosts the bitrate with something less complex. Also, you have to keep in mind that you won't have files that are 100% complex at home (I mean, over the whole length) like you have with 20 or 30 seconds samples.

I think we discussed the 2-pass problem to death now. The poll is also due soon and it seems that WMA Pro won (it can change, though). I will include whatever was democratically elecected as mention already.
  • Last Edit: 28 November, 2005, 03:24:40 PM by Sebastian Mares

  • ChiGung
  • [*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #652
Quote
Oh - sorry - I missed that bit. I think this is getting too advanced for my walnut brain.

Just let me know when I can download the test.
[a href="index.php?act=findpost&pid=346056"][{POST_SNAPBACK}][/a]

No probs naylor, the subject matter is indeed unweildy and Ive probably contradicted myself here and there. Im just trying to fix some misapprehension of wmas 2pass vbr tuning method, which seems to me to have got stuck in the works and could actualy be a very handy tool here.

When you compare the processes of manualy finding settings to produce vbr encodes averaging a target bitrate for the test corpus+a bulk of normal tracks, and automaticaly producing an optimal vbr encode which (rather reliably) achieves the target bitrate for the same material - the idea that 2passing is innappropriate doesnt stand up for me. The example of how different the samples would sound inside their parent tracks doesnt make sense to me, because the 2 pass discerns and then uses a global vbr setting for its encode. Its just that different routes are being taken to get to the same desired result.
no conscience > no custom

  • ChiGung
  • [*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #653
Quote
I think we discussed the 2-pass problem to death now. The poll is also due soon and it seems that WMA Pro won (it can change, though). I will include whatever was democratically elecected as mention already.[a href="index.php?act=findpost&pid=346059"][{POST_SNAPBACK}][/a]


Ok, I couldnt spell it clearer, maybe it will grow on you before the next test 
no conscience > no custom

  • Alex B
  • [*][*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #654
At this stage selecting good samples is critical. I wouldn't worry about the bitrates until the samples are gathered. The bitrates will probably be just fine. For example, over 140 kbps is nothing to worry about if all encoders show more or less similar behavior with the same sample. My 25 varied full-length tracks produced this graph:
Quote
[{POST_SNAPBACK}][/a]

Only the WMA VBR q75 bitrates look a bit strange (besides the too high overall average bitrate for this test). The others have of course variation from track to track because the encoders use different technologies, but the average bitrate is not the only factor that affects the quality.

As we know the current encoders are quite good and there is indeed a danger that the samples will be too transparent for many of the testers. I posted a small test report, which explains a problem I encountered recently:  [a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=39233&hl=]A little VBR ~88 kbps ABX-test[/url]. It would be interesting to see some HA members' ABX results of the sample I provided.

Edit:

I forgot to mention that these values don't give us any idea what kind of bitrate fluctuation happens between the track start and end. That may also vary greatly between the encoders.
  • Last Edit: 28 November, 2005, 05:34:29 PM by Alex B

  • naylor83
  • [*][*][*]
Multiformat 128 kbps Listening Test
Reply #655
Ok. Now I understand why you would want to change the samples if they turn out to make some encoders produce too high a bitrate.

I guess it's so that the ratings/scores for the encoders represent more or less what they would get for music in general, or if tested with a very large set of songs.
davidnaylor.org

  • Gambit
  • [*][*][*][*][*]
  • Developer
Multiformat 128 kbps Listening Test
Reply #656
Quote
WTF are you talking about? Do you want to say that Roberto influenced me to think you're a dick? Sorry to disappoint you, but I found that myself.
And pointing out flaws looks different that starting to bitch and flame IMO. You simply came into this thread and started writing how clueless we all are and how God-like you are. I had to ask you and Dibrom multiple times to provide some constructive critism and not only "this is shit", "you are the most clueless guy in this thread", etc.
[{POST_SNAPBACK}][/a]

The thread speaks for itself. I went back and reread everything, and I stand behind everything I said. And I don't think I was ever inappropriate, with the exception of the "clueless" comment. And with that I quickly recognized my mistake and removed my post. So I always tried to keep this strictly in the discussion level and never said or took anything personally. Can you and your friends say the same? (No need to answer, as I said, the thread speaks for itself.)

Quote
Quote
It was rude of Gambit to make the comment that he did.  But on the other hand, he was making an observation based on actual behavior.  Given all that has transpired in this thread and now this, I can't say that what he said was wrong (sorry), even if it wasn't very tactful to make it a public observation.
[a href="index.php?act=findpost&pid=345961"][{POST_SNAPBACK}][/a]


Could you give me some precise and objective facts why you think I am the most clueless guy in this thread?
[a href="index.php?act=findpost&pid=345971"][{POST_SNAPBACK}][/a]

Just FYI. I didn't say you are the most clueless guy in this thread. I said that with your comments you prove that you are one of the most clueless people in this thread. I was mostly refering to the LAME discussion where you obviously didn't understand the problem. And objective facts? Just look at the 25 pages of this thread. You came completely unprepared into this, but that's not the problem. The problem is how you and others deal with criticism.

Quote
Quote
It was rude of Gambit to make the comment that he did. But on the other hand, he was making an observation based on actual behavior. Given all that has transpired in this thread and now this, I can't say that what he said was wrong (sorry), even if it wasn't very tactful to make it a public observation.

He made an observation?! He treated Sebastian as "clueless person" (before abusing of the moderating tools by removing the insult and cleaning my quote  ).
[a href="index.php?act=findpost&pid=345968"][{POST_SNAPBACK}][/a]

I didn't abuse anything. Any user can go back and edit their posts. I decided to take it back, as I realized that some people might take that comment personally, and I wanted to avoid that. Sadly that is something that you fail at.

Quote
Note that it's really difficult to follow your thoughts:

• « I definitely would like to see both WMA Standard and Pro included (...) can say whatever you want about WMA, but it at least had gapless support way before some of the other codecs had it (...) And I'm very much interested in the differences between WMA Standard and Pro. »
[a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=38723&view=findpost&p=341454]http://www.hydrogenaudio.org/forums/index....ndpost&p=341454[/url]

• « (...) and I think we agreed that it's useless to test WMA Pro »
http://www.hydrogenaudio.org/forums/index....ndpost&p=345495

• « #@%$& piece of WMA crap...  »
http://www.hydrogenaudio.org/forums/index....ndpost&p=345527

Very interesting first, then useless and finally piece of crap   And then you wonder about the lack of organisation in the debate?
[a href="index.php?act=findpost&pid=345916"][{POST_SNAPBACK}][/a]

Guru, the trick with quoting stuff out of context is getting old. Everybody can go back and see how things really are...

Anyway, this is really getting off-topic, so there is no need to reply. I just wanted to set things straight. I will stay out of this test from now on.

I really wish you a successful test. Good luck.
  • Last Edit: 28 November, 2005, 06:20:18 PM by Gambit
Burrrn - http://www.burrrn.net/
MPEG Audio Collection - http://mac.sourceforge.net/

Multiformat 128 kbps Listening Test
Reply #657
Come on, please, make this stop...

Edit: I swear this is the last time I write about the things that went wrong with this thread... Warn me if I do it again! The real problem began on page 6 where Dibrom wrote that the test is going into the wrong direction although nothing was decided. Also, you Gambit began bashing Gabriel because you didn't want Shine to be included. You started nitpicking because Gabriel wrote that the anchor shouldn't really be seen as competitor and therefore doesn't really require intensive testing. He obviously didn't use the right words but that doesn't give you the right to call bullshit what he says and also indirectly call him an idiot because you never expected him to write such BS. I think some statements like that are enough to make the whole atmosphere tense.

---

What I wanted to know now is whether you want to decide something regarding the sample or should I create the sample set and you swallow it as-is?
Basically, I have 18 new samples which are all new. We could either make if open for discussion which ones of those 18 to replace with old samples and which ones to keep, or I go over the set myself and replace what I think is appropiate (although that's going to be hard).
  • Last Edit: 28 November, 2005, 06:20:20 PM by Sebastian Mares

  • JeanLuc
  • [*][*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #658
This is getting absolutely ridiculous ... maybe it would be better to take WMA completely off the list and add the actual official OggEnc build instead to see how it performs against aotuv ...
The name was Plex The Ripper, not Jack The Ripper

Multiformat 128 kbps Listening Test
Reply #659
Quote
This is getting absolutely ridiculous ... maybe it would be better to take WMA completely off the list and add the actual official OggEnc build instead to see how it performs against aotuv ...
[a href="index.php?act=findpost&pid=346105"][{POST_SNAPBACK}][/a]


Well, that would be rather unballanced - we'd have 2 AAC encoders and 2 Vorbis encoders. As mentioned already, I will include whatever wins in the poll about the 5th competitor.
  • Last Edit: 28 November, 2005, 06:26:17 PM by Sebastian Mares

  • JeanLuc
  • [*][*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #660
Quote
Well, that would be rather unballanced - we'd have 2 AAC encoders and 2 Vorbis encoders. As mentioned already, I will test whatever wins in the poll about the 5th competitor.
[a href="index.php?act=findpost&pid=346106"][{POST_SNAPBACK}][/a]


That's why I already put in my vote for WMA Pro (although I've been for WMA Std in the first place for known reasons) ... it's just the simple fact that I really don't like the way this discussion gets 'personal'.

We claim to be an unbiased community in search for the truth in all matters related to audio but the way some people behave seems to prove the exact opposite.

Sorry ... I just needed to discharge some steam ... 
The name was Plex The Ripper, not Jack The Ripper

  • ErikS
  • [*][*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #661
Quote
What I wanted to know now is whether you want to decide something regarding the sample or should I create the sample set and you swallow it as-is?
Basically, I have 18 new samples which are all new. We could either make if open for discussion which ones of those 18 to replace with old samples and which ones to keep, or I go over the set myself and replace what I think is appropiate (although that's going to be hard).
[a href="index.php?act=findpost&pid=346100"][{POST_SNAPBACK}][/a]


It wouldn't be useful to publish the raw samples for various reasons. However, what would be very useful is if you could publish a table of the bitrates for the samples x encoders. Ideally the average bitrate should be the same for all encoders (with the exception of the anchor of course).
  • Last Edit: 28 November, 2005, 06:29:46 PM by ErikS

  • Alex B
  • [*][*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #662
Perhaps some people are still co-operative and willing to help (without trying to start a new debate about the samples).

I too would like to see a table of the samples. You could include the bitrates produced by the already selected encoders.

You can use my table as a templete in case you don't have one already.

bitrates_public.xls
The graph part is included in the xls file.

If you can make the samples available perhaps some opinions about the suitability from experienced persons would be good to have, but that is up to you.

  • ChiGung
  • [*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #663
Quote
It wouldn't be useful to publish the raw samples for various reasons. However, what would be very useful is if you could publish a bitrates for the samples x encoders. Ideally the average bitrate should be the same for all encoders (with the exception of the anchor of course).[a href="index.php?act=findpost&pid=346108"][{POST_SNAPBACK}][/a]

I dont think it matters much if the setting is shown to reach a target for a larger 'targeting corpus' or 'sample corpus' One or the other is preferable to targeting both because manipulations either way could distort the tests difficulty for affected codecs.
I brought up the subject of bitrate targeting to show how wma's 2pass can be used > to target a bitrate, whether thats for sample corpus or target corpus (specified or casualy unspecified /'generaly establishing known behaviour') the found settings are to achieve some targeted bitrate [while allowing normal variation between samples] 2pass vbr can be used in a way which is fairly analogus to the manual 'unspecific' method to hit the same target (bitrate@material type).
I thought it was decided soberly to include wma standard in the test, i can see compelling reasons to do so. You really want to drop it now because of a small technical inconviences and what will be recognised as a misunderstanding once the penny drops?
  • Last Edit: 28 November, 2005, 06:55:58 PM by ChiGung
no conscience > no custom

  • ErikS
  • [*][*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #664
Quote
I dont think it matters much if the setting is shown to reach a target for a larger 'targeting corpus' or 'sample corpus' One or the other is preferable to targeting both because manipulations either way could distort the tests difficulty for affected codecs.

I don't see any reasons to why the average bitrate shouldn't be same over both the small corpus of actual test samples and the nearly infinite corpus which include all music. It's just the same as to say that the sample corpus should be selected so that it is representative to the larger corpus as far as bitrate for the chosen codecs is concerned. It's not unfair in any way - quite the opposite I think...

Quote
I brought up the subject of bitrate targeting to show how wma's 2pass can be used > to target a bitrate, whether thats for sample corpus or target corpus (specified or casualy unspecified /'generaly establishing known behaviour') the found settings are to achieve some targeted bitrate [while allowing normal variation between samples] 2pass vbr can be used in a way which is fairly analogus to the manual 'unspecific' method to hit the same target (bitrate@material type).

Well, in my opinion 2-pass vbr can be used if and only if the sample correspond to the "infinite corus" when it comes to average bitrate.
Quote
I thought it was decided soberly to include wma standard in the test, i can see compelling reasons to do so. You really want to drop it now because of a small technical inconviences and what will be recognised as a misunderstanding once the penny drops?
[a href="index.php?act=findpost&pid=346115"][{POST_SNAPBACK}][/a]

What are you talking about? Is this directed to me or someone else? I always preferred to test WMA Pro over plain WMA, so I most definitely don't have any problems with this switch. The technical inconveniences you mention have nothing to do with that.

  • Alex B
  • [*][*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #665
Quote
Ideally the average bitrate should be the same for all encoders (with the exception of the anchor of course).
[{POST_SNAPBACK}][/a]

How come? Ideally the VBR encoders are used with the fair predefined quality settings. It doesn't matter what bitrates the individual samples are. This test will test encoders in a quality-based mode (VBR). The bitrate findings are gathered here: [a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=38955]Bitrate estimates for the upcoming 128 kbps test[/url]

OR

If you mean that the bitrates should be the same for all encoders when a big amount of various music files are encoded, then the suitable quality settings are already found, except for Nero if a new version is going to be tested.

Also, WMA Pro VBR q50 and Musepack q4 fit fine in the target range. Here are the Musepack bitrates for my test file set: Musepack 1.15v q4 = average 133 kbps, median 129 kbps.

[span style='font-size:7pt;line-height:100%'](This set, which I originally gathered for a LAME VBR bitrate evaluation has proven to be quite average for the "various" genre. I observed the bitrate behavior of my LAME aps file archive when I selected the test set. I started with 200 selected tracks. Then I compared the bitrates with my lossless archive and made also several test encodings at lower bitrates. Eventually I ended up with this set of 25 tracks.)[/span]

[span style='font-size:7pt;line-height:100%']Edit: added "if a new Nero..."[/span]
  • Last Edit: 28 November, 2005, 08:12:01 PM by Alex B

  • ChiGung
  • [*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #666
Quote
I don't see any reasons to why the average bitrate shouldn't be same over both the small corpus of actual test samples and the nearly infinite corpus which include all music.
because the test corpus is expected to be more challenging, more complex.
Quote
It's just the same as to say that the sample corpus should be selected so that it is representative to the larger corpus as far as bitrate for the chosen codecs is concerned. It's not unfair in any way - quite the opposite I think...
I dont think its unfair either, if manipulations where required to make both sample and 'infinite' fit targets those manipulations could be unfair. Not a big issue for me as i said...

Quote
I brought up the subject of bitrate targeting to show how wma's 2pass can be used > to target a bitrate,

Quote
Well, in my opinion 2-pass vbr can be used if and only if the sample correspond to the "infinite corus" when it comes to average bitrate.
We are in agreeance then. 2pass can target that bitrate just as well as the rest of the codecs 'manual multipass' can target it - by using a normalised subset of the ideal 'infite corpus'
Quote
Quote
I thought it was decided soberly to include wma standard in the test, i can see compelling reasons to do so. You really want to drop it now because of a small technical inconviences and what will be recognised as a misunderstanding once the penny drops?

What are you talking about? Is this directed to me or someone else?
Its directed to readers of the thread.
Quote
I always preferred to test WMA Pro over plain WMA, so I most definitely don't have any problems with this switch. The technical inconveniences you mention have nothing to do with that.
Your preference is noted
  • Last Edit: 28 November, 2005, 07:35:21 PM by ChiGung
no conscience > no custom

  • ChiGung
  • [*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #667
I imagine how through the course of this thread Sebastian Mares is slowly but surely beginning to look more and more like his Avatar. 

It wasnt my intention to contribute to that but I fear I may have, sorry mr Mares.

I think its gonna be a hell of a test in the end'

Stick it out 

'gnight
no conscience > no custom

  • JohnV
  • [*][*][*][*][*]
  • Developer
Multiformat 128 kbps Listening Test
Reply #668
Well, looks like WMA Professional is gonna win the poll for the 5th contender.
Good thing for Microsoft and WMA when the test results are out, since most people outside HA probably don't realize there's actually a significant difference between these codecs.  The other one, higher quality - almost nowhere used, other one lower quality and used very widely.
Interesting to see the results though.
  • Last Edit: 28 November, 2005, 08:43:12 PM by JohnV
Juha Laaksonheimo

  • ErikS
  • [*][*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #669
Quote
because the test corpus is expected to be more challenging, more complex. [a href="index.php?act=findpost&pid=346126"][{POST_SNAPBACK}][/a]


Why?

I think you make a couple of mistakes if you do that.

Firstly, who defines complex? Is it defined by looking at how high bitrate the encoders allocate for a particular part? Then you only pick samples where they succeed to allocate an appropriate amount of bits, and exclude those where there might be a flaw in the psymodel and it assignes a low bitrate to a complex sound.

Secondly, if you have some other criteria for complex, then you only sample a small subset of the big music corpus, and will miss inportant aspects of how the encoders perform on other parts - for example if you define complex as having a high PE, then you exclude almost all classical music. And I know some people will complain then.. (Besides favouring ecoders which use PE for their bitrate allocation, but then I'm back to my first argument.)
  • Last Edit: 28 November, 2005, 09:06:00 PM by ErikS

  • =trott=
  • [*][*][*]
Multiformat 128 kbps Listening Test
Reply #670
Quote
[partially my fault. But I didn't imagine that WMA is so problematic and that's why it took so much time. [a href="index.php?act=findpost&pid=345840"][{POST_SNAPBACK}][/a]


Already I feel we can draw a very valuable conclusion from all this discussion: that it is one thing to say that wma has good quality compared to other codecs, quite another to prove it. I wonder then how microsoft does it?

  • =trott=
  • [*][*][*]
Multiformat 128 kbps Listening Test
Reply #671
Quote
Well, looks like WMA Professional is gonna win the poll for the 5th contender.
Good thing for Microsoft and WMA when the test results are out, since most people outside HA probably don't realize there's actually a significant difference between these codecs.  The other one, higher quality - almost nowhere used, other one lower quality and used very widely.
Interesting to see the results though.
[a href="index.php?act=findpost&pid=346135"][{POST_SNAPBACK}][/a]


True, but if you think about it...we also say that mp3 is still competitive despite promises of aac replacing it...but this is only valid if the encoder is lame. if we were to do a comparison of (just as example) lame and itunes' mp3 encoder, it would both be mp3 but quality would be quite different...Nevertheless people can also say: mp3 won, so I'll just put mp3 as the default in my itunes.

  • =trott=
  • [*][*][*]
Multiformat 128 kbps Listening Test
Reply #672
Quote
Quote
because the test corpus is expected to be more challenging, more complex. [a href="index.php?act=findpost&pid=346126"][{POST_SNAPBACK}][/a]



Firstly, who defines complex?

Secondly, if you have some other criteria for complex, then you only sample a small subset of the big music corpus, and will miss inportant aspects of how the encoders perform on other parts - for example if you define complex as having a high PE, then you exclude almost all classical music. [a href="index.php?act=findpost&pid=346138"][{POST_SNAPBACK}][/a]


Thirdly, what's a corpus ?
  • Last Edit: 29 November, 2005, 03:36:52 AM by =trott=

  • naylor83
  • [*][*][*]
Multiformat 128 kbps Listening Test
Reply #673
Quote
Well, looks like WMA Professional is gonna win the poll for the 5th contender.
Good thing for Microsoft and WMA when the test results are out, since most people outside HA probably don't realize there's actually a significant difference between these codecs.  The other one, higher quality - almost nowhere used, other one lower quality and used very widely.
Interesting to see the results though.
[a href="index.php?act=findpost&pid=346135"][{POST_SNAPBACK}][/a]


On a sidenote: Is there a command line encoding utility for WMA Pro? I'd like to try that beast out.
davidnaylor.org

  • ErikS
  • [*][*][*][*][*]
Multiformat 128 kbps Listening Test
Reply #674
Quote
Thirdly, what's a corpus ?
[{POST_SNAPBACK}][/a]


[a href="http://en.wikipedia.org/wiki/Corpus]http://en.wikipedia.org/wiki/Corpus[/url]
http://en.wikipedia.org/wiki/Text_corpus