Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Listening tests sample selection (Read 25999 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Listening tests sample selection

Reply #25
Quote
But are they? clearly, the listening test I was linking to shows otherwise. And while I agree that the two samples in there were carefully chosen, they clearly highlighted that not all VBR codecs do this fine all the time. Your argument is opposed to the results of the test! ohmy.gif

I agree that these two samples show that VBR codecs don't do a good job all the time. But I believe that they do a good job most of the time. Of course it would be better if we had more samples like that for the listening tests. But finding more such samples is difficult, they are so rare (well maybe not so much with classical or other tonal or low volume parts).

Quote
While I do agree with that, you also need to compare apples with apples. If the test is titled '128kbps listening test', you need to make sure you are comparing stuff at 128kbps. If the test is entitled 'recommended transparency settings listening test' then you don't have to make sure bitrate is the same, and you will compare quality AND bitrate. It all boils down to make sure you don't confuse people and call your listening test by its right label.
But you see you are comparing apples to apples. If I encode my whole music collection with the settings used for the listening tests it would come out at exactly or really close to the target bitrate. Deviations (even huge ones) within samples or even within whole albums don't really matter. My whole collection would meet my bitrate/space criteria.

Quote
I think we just have a terminology problem here, but I strongly disagree with that. In my terminology, more bits will mean better quality (assuming the same encoder). The ABX test will be here to prove that this difference in quality is audible or not.
Quote
So if you throw more bits, you will have a better quality (in theory). Is it audible or not is a different problem.

This is the part that we disagree. There are so many more things that an encoder has to decide apart from bitrate. All those psychoacoustic settings play a big role to quality as well. Throwing more bits while using a flawed psychoacoustic model won't help you much. Try a small example: Pick a sample and encode it with lame --preset 128. Then do another encoding at --preset 160 but set the lowpass cutoff to a high frequency (say 22khz) or even use the -k switch. My bet is that the 128kbps encoding will sound better (meaning closer to the original) than the 160kbps one. And that with just playing with the lowpass using the same encoder. I remember vorbis having severe pre-echo problems up to -q7,-q8 with older (version 1.0?) encoders. These were for the most part fixed by later tunings that had little to do with bitrate inflation. Now these same problems are not that serious at all even at lower -q settings.

Quote
Also, don't forget that even though the difference btw A and B might be non-audible to most under standard listening conditions, the simple fact of having a bass/trebble button, an equaliser or just doing a recoding (of an MP3 in to Ogg for example) will degrade the quality of the file further. Starting with more bits will help, even though these bits looked overkill in an ABX test. More bits into the original will mean fewer distortions/artefacts/frequencies cut out, hence a more versatile file.

Well, that has do only with somebody's listening habits. But being perfect with any eq setting is not what a lossy encoder tries to do. Most people here would advise you not to use any eq with your music. If you want to use extreme eq settings (small ones would not matter) or you are conserned about transcoding to another lossy format then my advice is to use a lossless encoder rather than a lossy one.

Listening tests sample selection

Reply #26
Quote
So in an ideal test, you would have 160kbps, 130kbps and 90kbps samples. THAT would be representative, not merely 128kbps.
[a href="index.php?act=findpost&pid=319058"][{POST_SNAPBACK}][/a]
I don't think you understand what I'm saying. Let's take a real world situation. Person A wants to compress MP3s at 128kbps to get a lot of storage out of his small flash based DAP. He can use CBR at 128kbps or VBR at a target bit rate of 128kbps. You're saying comparing CBR at a given bit rate to VBR at a given target bit rate is unfair.

I'm saying while it may technically be unfair it's representative of what someone in the real word who was deciding between CBR and VBR would do. They aren't going to try to match bit rates to make the comparison fair, they are going to use preset x and be done with it.

Listening tests sample selection

Reply #27
Quote
I don't think you understand what I'm saying. Let's take a real world situation. Person A wants to compress MP3s at 128kbps to get a lot of storage out of his small flash based DAP. He can use CBR at 128kbps or VBR at a target bit rate of 128kbps. You're saying comparing CBR at a given bit rate to VBR at a given target bit rate is unfair.

I'm saying while it may technically be unfair it's representative of what someone in the real word who was deciding between CBR and VBR would do. They aren't going to try to match bit rates to make the comparison fair, they are going to use preset x and be done with it.
[a href="index.php?act=findpost&pid=319321"][{POST_SNAPBACK}][/a]

I completely agree with you. My point focus on the selection of the samples with which you will conduct the test. Let's say you have a 5000 songs collection. With your VBR presets, you will get an average bitrate of 128kbps. Now I can safely bet that you will have at least 1000 songs under 128kbps (we will call them 'Set A') and 1000 songs over 128kbps (we will call them 'Set B').

What I am saying is that if you conduct a listening test and you include only samples from 'Set A' in your listening test, you're biaising the test, bacause this set is NOT representative of your entire collection as far as 'complexity' is concerned. You will test only the behavior of vbr encoders on 'low complexity' songs. Hence CBR encoders have an unfair advantage because they use more bits on your samples than your VBR encoders.

The samples used in a test must be representative of the entire collection on which you tuned the '128kbps presets' of your VBR encoder.

If you compare the most complex song of your collection (which will be encoded at 256kbps for example), of course VBR will win. I don't need a listening test to confirm that: It uses twice as many bits!

Now from that comparison, would you deduce that the VBR encoder is always better? I hope not!

Now if you compare your 'simplest' or 'least-complex' song (excluding silences which are meaningless), your VBR encoder will have thrown only 80kbps at it. Chances are it will sound worse than the CBR encoder which is throwing in 48kbps more!!! Of course, this is theory bacause no test allow us to conclude so...

So this clearly shows that if your sample selection tends to be only in 'Set A' or 'Set B', the test tend to be biaised.

Note to the picky people: When I make such statements as 'more bitrate is better', of course, I mean with the same encoder and with the same parameters! 

Listening tests sample selection

Reply #28
Quote
But you see you are comparing apples to apples. If I encode my whole music collection with the settings used for the listening tests it would come out at exactly or really close to the target bitrate. Deviations (even huge ones) within samples or even within whole albums don't really matter. My whole collection would meet my bitrate/space criteria.

Let's make a quick statement that I think will explain my thinking:
I do a test between two files, one encoded with LAME VBR 64kbps, one with LAME CBR 64kbps. This is the same encoder, same parameters (psy model, low pass, etc...). Chances are that a higher bitrate will lead to higher quality. Hope you agree on that one.

Now, how does VBR works: It will allocate more bits to the 'complex' files and less bits to the 'easy' files. Hence:
1. If you do this test on a 'complex' sample, VBR will sound better.
2. If you do this test on an 'easy' sample, CBR will sound better.

What have test 1 proven? It has proven that the VBR encoder did its job.
What have test 2 proven? It has proven that the VBR encoder did its job.

But none of these tests have proven anything as far as CBR vs VBR is concerned. If you test only complex files, VBR will win hands down every time.

Now of course, while this is true with LAME vs LAME, it might not be true with Ogg vs AAC or any others, but still what was an absolute in my example becomes a bias/trend. You cannot compare fairly VBR vs CBR only on complex samples.

Quote
This is the part that we disagree. There are so many more things that an encoder has to decide apart from bitrate. All those psychoacoustic settings play a big role to quality as well. Throwing more bits while using a flawed psychoacoustic model won't help you much. Try a small example: Pick a sample and encode it with lame --preset 128. Then do another encoding at --preset 160 but set the lowpass cutoff to a high frequency (say 22khz) or even use the -k switch. My bet is that the 128kbps encoding will sound better (meaning closer to the original) than the 160kbps one. And that with just playing with the lowpass using the same encoder. I remember vorbis having severe pre-echo problems up to -q7,-q8 with older (version 1.0?) encoders. These were for the most part fixed by later tunings that had little to do with bitrate inflation. Now these same problems are not that serious at all even at lower -q settings.

You guys are incredible. In all my posts I try to be overly cautious and try not to say 'So if you throw more bits, you will have a better quality' but 'So if you throw more bits, you will have a better quality (same encoder, same parameters)'. Now in the last one I forget it and then you just attack me on that point... Did you read the rest of the thread? My other messages? I've repeated so many times the same thing that all my thought is in there. Yet you have to fight on a grammar mistake on my very last message...

Of course, if I compare a blue toyota with a red Ferrari, it could be erroneous to conclude that red cars are faster than blue ones. I thought we would be on the same paghe there. My point is that A GIVEN CODEC WILL SOUND BETTER WITH MORE BITS. How can that be wrong?

Listening tests sample selection

Reply #29
Quote
Let's make a quick statement that I think will explain my thinking:
I do a test between two files, one encoded with LAME VBR 64kbps, one with LAME CBR 64kbps. This is the same encoder, same parameters (psy model, low pass, etc...).


Are you sure about that? I think there are more settings changed than a CBR/VBR switch. I don't really know, I never have looked at a single encoder's code. Maybe a developer could tell us what happens.

Quote
Now, how does VBR works: It will allocate more bits to the 'complex' files and less bits to the 'easy' files. Hence:
1. If you do this test on a 'complex' sample, VBR will sound better.
2. If you do this test on an 'easy' sample, CBR will sound better.


Are we talking about files (songs) here or are we talking about samples? These are two different things. If you are talking about songs, then your logic is flawed (I know I sound like a Vulcan ) That is because in a song could not be characterized as 'complex' or 'easy'. There could be difficult parts and there could be easy parts. And if your resulting VBR encoding has a lower average bitrate than your CBR file, there could be parts that have a much higher bitrate, even if 90% of the song is what you call 'easy'. Now which parts are going to sound worse? The 90% of the VBR song with the lower bitrate as the encoder decided or the 10% of the CBR song that has less bits than the VBR one? I think you assume that lower average bitrate means that the bitrate never exceeds the CBR bitrate.

Quote
You guys are incredible. In all my posts I try to be overly cautious and try not to say 'So if you throw more bits, you will have a better quality' but 'So if you throw more bits, you will have a better quality (same encoder, same parameters)'. Now in the last one I forget it and then you just attack me on that point... Did you read the rest of the thread? My other messages? I've repeated so many times the same thing that all my thought is in there. Yet you have to fight on a grammar mistake on my very last message...


Hey lighten up. Noone is fighting nobody here. And yes, I did not bother to read anything and started posting just because I felt like it like picmixer did and I don't know who else...    Try being less edgy OK?

Quote
My point is that A GIVEN CODEC WILL SOUND BETTER WITH MORE BITS. How can that be wrong?


Well, that's easy. You already answered it yourself in a previous post I didn't read.
Quote
In my terminology, more bits will mean better quality (assuming the same encoder). The ABX test will be here to prove that this difference in quality is audible or not.

A certain bitrate could be transparent for a given sample. Raising the bitrate won't help any more because it is already transparent! So if better 'quality' is not perceived as better is it really any better?

EDIT: Hey, why don't the quotes show up properly? 

Listening tests sample selection

Reply #30
Quote
EDIT: Hey, why don't the quotes show up properly? 
[a href="index.php?act=findpost&pid=319484"][{POST_SNAPBACK}][/a]

Because you forgot one quote start (
Quote
). I have fixed your post.

Listening tests sample selection

Reply #31
Quote
So if better 'quality' is not perceived as better is it really any better?

Possibly, in terms of post processing.

Listening tests sample selection

Reply #32
Quote
Quote
So if better 'quality' is not perceived as better is it really any better?

Possibly, in terms of post processing.
[a href="index.php?act=findpost&pid=319511"][{POST_SNAPBACK}][/a]

Of course it does! If all you plan to do is listen to your file on a crappy headphones, then I agree that there is no point. If you plan to do anything else with the files, then there is a point.

Let's take my example: I ripped all my CDs to my HDD, encoding with LAME insane (CBR 320kbps). I sure could have ripped it at around half the bitrate and wouldn't have noticed the difference in a casual listening environment. But then now, all my CDs are in my parent's basement: This was one of the objectives. So when I want to encode 64kbps Ogg or AAC files for my Palm/iPod, I get them from the MP3 already ripped, cause I don't feel like driving 1/2 hour and dig in dusty boxes.

I know, I know, I shouldn't transcode, quality is worse. Who cares? Can anyone really pretend to hear a difference between CD->Ogg@64 and CD->MP3@320->Ogg@64? And what about CD->MP3@128->Ogg@64? I'm sure there are higher chances there.

And what If I decide to buy these $3000 speakers next year? I am to re-rip all my CD collection because all of a sudden 128kbps is not transparent anymore for me? And what if I invite this friend to my place that can detect a 320kbps from the original in 1/2 second (I have one) and just cannot listen to a 160kbps LAME encode? Am I to stop playing music when he is around?

Ripping a CD to MP3 is not something absolute. It is a convenience. Higher quality means higher quality, even though you might not be able to hear it in a test does not mean it is a waste. Whan I ripped my collection, it was for archival purposes and I don't feel like having wasted any disk space.

Listening tests sample selection

Reply #33
Quote
Quote
Now, how does VBR works: It will allocate more bits to the 'complex' files and less bits to the 'easy' files. Hence:
1. If you do this test on a 'complex' sample, VBR will sound better.
2. If you do this test on an 'easy' sample, CBR will sound better.


Are we talking about files (songs) here or are we talking about samples? These are two different things. If you are talking about songs, then your logic is flawed (I know I sound like a Vulcan ) That is because in a song could not be characterized as 'complex' or 'easy'. There could be difficult parts and there could be easy parts. And if your resulting VBR encoding has a lower average bitrate than your CBR file, there could be parts that have a much higher bitrate, even if 90% of the song is what you call 'easy'. Now which parts are going to sound worse? The 90% of the VBR song with the lower bitrate as the encoder decided or the 10% of the CBR song that has less bits than the VBR one? I think you assume that lower average bitrate means that the bitrate never exceeds the CBR bitrate.
[a href="index.php?act=findpost&pid=319484"][{POST_SNAPBACK}][/a]

The problem is the same. In a file that will average 128kbps, the parts encoded > 128 will sound better in the VBR version end the parts < 128 will sound worse. Assuming - once more - the same encoder and the same settings.

As far as LAME CBR vs LAME VBR is concerned, AFAIK, in a VBR mode you activate a bit allocation mechanism that will decide for each frame how many bits it should consume. After that you get to the same algorithm that will encode a frame to a specific bitrate.

Of course, it is a bit oversimplifying, but higher bitrate on a portion of a song will usually mean higher quality. Of course a LAME dev could enlighten us more on that subject.

Listening tests sample selection

Reply #34
Quote
In a file that will average 128kbps, the parts encoded > 128 will sound better in the VBR version end the parts < 128 will sound worse. Assuming - once more - the same encoder and the same settings.
[a href="index.php?act=findpost&pid=319542"][{POST_SNAPBACK}][/a]

I always thought, that VBR schemes aimed at a constant quality. So in theory, the parts > 128 should sound as good/bad as the parts < 128, except for extreme killer samples of course.

A snare drum will be encoded with 160 kbs, for example, and a vocal part with equal length gets 96 kbs, resulting in an avarage of 128 kbs. The vocal part *should* sound the same as if it was encoded with 128 kbs, so we "saved" a number of bits on that part. I thought that was the idea behind VBR.

The equivalent CBR file will allocate 128 kbs to both parts. However, in the part of the snare drums that is not enough and in the part of the vocals it is too much. So, this file will sound worse.

As a general rule, CBR will always sound worse than its VBR counter part, providing that the algorithm does its job. Therefore, in listening tests almost all the time VBR is chosen if it is a available for a certain codec.

Listening tests sample selection

Reply #35
Quote
The equivalent CBR file will allocate 128 kbs to both parts. However, in the part of the snare drums that is not enough and in the part of the vocals it is too much. So, this file will sound worse.[a href="index.php?act=findpost&pid=319570"][{POST_SNAPBACK}][/a]

What is 'too much'? How can you say bitrate is too much? At 32kbps (for example), nothing is transparent, so the parts of the vocals will sound better with a CBR encoder than with a VBR one. (again, assuming same encoder, same parameters). So short of listening to it, how can you assert 'this file will sound worse'? All that you can assert is 'parts of this file is likely to sound worse, part of this file is likely to sound better'.

Quote
As a general rule, CBR will always sound worse than its VBR counter part, providing that the algorithm does its job.
[a href="index.php?act=findpost&pid=319570"][{POST_SNAPBACK}][/a]

Emphasis mine. But isn't that the point of a listening test to confirm this assertion?

My point is that if you compare only 'high complexity' samples, you cannot say 'Ogg is better than Atrac3'. However, you can say 'Ogg is better than Atrac3 on high complexity samples'. And, by the way, high complexity samples are anything but representative of anyone's music collection as a whole.

Listening tests sample selection

Reply #36
Quote
Quote
The equivalent CBR file will allocate 128 kbs to both parts. However, in the part of the snare drums that is not enough and in the part of the vocals it is too much. So, this file will sound worse.[a href="index.php?act=findpost&pid=319570"][{POST_SNAPBACK}][/a]

What is 'too much'? How can you say bitrate is too much? At 32kbps (for example), nothing is transparent, so the parts of the vocals will sound better with a CBR encoder than with a VBR one. (again, assuming same encoder, same parameters). So short of listening to it, how can you assert 'this file will sound worse'? All that you can assert is 'parts of this file is likely to sound worse, part of this file is likely to sound better'.

For a pure sine wave for example, a limited (low) number of bits is necessary to encode it lossless, let alone transparant. So, throwing more bits at it is what I mean with too much.

Quote
My point is that if you compare only 'high complexity' samples, you cannot say 'Ogg is better than Atrac3'. However, you can say 'Ogg is better than Atrac3 on high complexity samples'. And, by the way, high complexity samples are anything but representative of anyone's music collection as a whole.
[a href="index.php?act=findpost&pid=319573"][{POST_SNAPBACK}][/a]

I agree with you that the sample set should be chosen carefully, and it should depent on the bitrate. For example, if I do a listening test at ~140 kbs (which I've done recently), I choose samples with different levels of "problems": samples that are likely to result in severe problems at that bitrate, slight problems and samples that should do fine at that bitrate.

I think you have a point that on avarage, the samples should avarage the same bitrate as close as possible (by mixing problem, normal and easy samples).

However, it is also true that the problem samples are the "bottleneck", especially for the higher bitrates (> 160 kbs). The easy samples will sound (close to) transparent anyway.

Listening tests sample selection

Reply #37
Quote
For a pure sine wave for example, a limited (low) number of bits is necessary to encode it lossless, let alone transparant.
[a href="index.php?act=findpost&pid=319580"][{POST_SNAPBACK}][/a]

Yes, but I thought we were talking about music here.    This is a corner case, and transparency is rarely achieved even on 'easy' samples at bitrates < 80kbps, so the point is moot.

A quality based codec will generate (in theory) a file that sounds as good or as bad all along. A CBR codec can be transparent on half the file and worse than its VBR counterpart in the rest. Which one is 'the best sounding file' is a matter of choice...

Listening tests sample selection

Reply #38
Quote
Quote
For a pure sine wave for example, a limited (low) number of bits is necessary to encode it lossless, let alone transparant.
[a href="index.php?act=findpost&pid=319580"][{POST_SNAPBACK}][/a]

Yes, but I thought we were talking about music here.    This is a corner case, and transparency is rarely achieved even on 'easy' samples at bitrates < 80kbps, so the point is moot.

Agreed.

[offtopic]Sine waves are not music, but maybe a new group of musicians stands up in the future, making avant garde sine music or something  [/offtopic]

Quote
A quality based codec will generate (in theory) a file that sounds as good or as bad all along. A CBR codec can be transparent on half the file and worse than its VBR counterpart in the rest. Which one is 'the best sounding file' is a matter of choice...[a href="index.php?act=findpost&pid=319583"][{POST_SNAPBACK}][/a]

I think that's the point, yes. Probably the best way to select a set of samples is the following:

1) Determine at which setting the VBR algorithm results in an avarage bitrate equal to the CBR one, using a large number of "normal" samples (a "representative" set).

2) Next, choose a certain number of samples out of that set randomly. To make sure that all kinds of samples are included (I'm convinced that problem samples should at least be part of the test set), the number of final samples should be high enough.

One problem of this method is, that only experienced testers can participate, because most samples will be (almost) transparent to the general listener.

Therefore, current listening tests are a trade-off, I think.

Listening tests sample selection

Reply #39
Quote
One problem of this method is, that only experienced testers can participate, because most samples will be (almost) transparent to the general listener.

Agreed at 128kbps, not so much at lower bitrates such as 64, 48, 32 etc...

Listening tests sample selection

Reply #40
Quote
Of course it does! If all you plan to do is listen to your file on a crappy headphones, then I agree that there is no point. If you plan to do anything else with the files, then there is a point.

Are lossy encodings only for listening to crappy headphones? And I thought that some settings were transparent for most music on any hi-tech gear... You see the only point of having lossy files is listening to them, at various settings on any kind of equipment. If you want anything else go with lossless. End of story.

Quote
The problem is the same. In a file that will average 128kbps, the parts encoded > 128 will sound better in the VBR version end the parts < 128 will sound worse. Assuming - once more - the same encoder and the same settings.

Well that's the whole point. VBR is about having constant quality. I don't care if some parts sound worse than their CBR counterpart and some better. All I care about is that I decided on a setting, and all the parts in all my music have the same quality. The same quality I decided to use as a space/quality compromise. Because lossy is about making compromises. If you don't want any, then again go with lossless.

Quote
I know, I know, I shouldn't transcode, quality is worse. Who cares? Can anyone really pretend to hear a difference between CD->Ogg@64 and CD->MP3@320->Ogg@64? And what about CD->MP3@128->Ogg@64? I'm sure there are higher chances there.

Of course someone can hear a difference! Very severe artifacts are being created from transcoding. There were several listening test in this board with this subject from guruboolez and some other members. Try doing a search, you'll see that audible differences exist even when transcoding from a lot better lossy sources than mp3@128 or mp3@320.

Quote
And what If I decide to buy these $3000 speakers next year? I am to re-rip all my CD collection because all of a sudden 128kbps is not transparent anymore for me? And what if I invite this friend to my place that can detect a 320kbps from the original in 1/2 second (I have one) and just cannot listen to a 160kbps LAME encode? Am I to stop playing music when he is around?

Well, once again, use lossless...

Quote
What is 'too much'? How can you say bitrate is too much? At 32kbps (for example), nothing is transparent, so the parts of the vocals will sound better with a CBR encoder than with a VBR one.

Heh, that's exactly what I said once and had rjamorim correct me that digital silence is totally transparent at 32kbps. Hell, you could even go a lot lower and still won't notice any difference... The point is that if the encoder decides that it should allocate 32kbps for a part, then that is because it decided that 32kbps is enough to keep the quality at a constant level.

Quote
Emphasis mine. But isn't that the point of a listening test to confirm this assertion?

You assume that the encoder doesn't do a good job at allocating bits. If you think you can help making things any better, the sources are there, take a look...

Quote
My point is that if you compare only 'high complexity' samples, you cannot say 'Ogg is better than Atrac3'. However, you can say 'Ogg is better than Atrac3 on high complexity samples'. And, by the way, high complexity samples are anything but representative of anyone's music collection as a whole.

Well, feel free to organize a listening test with any kind of samples you decide are best. Good luck trying to get any meaningful results with the low complexity samples though (well you could make an ultra-low bitrate listening test and have meaningfull results...). People (including me) already have a hard time ABXing critical samples at medium (~128) bitrates.

Listening tests sample selection

Reply #41
At some point, I decided to reduce my postings about listening test conduction to the minimum. I was becoming too much of "HA's listening test authority", and I think that is not good. Other people need to come up with their own expertise on the subject.

Anyway...



I am amazed you guys can waste so many words on a non-issue. This is sickening.

This thread starts off from a flawed idea: That test conducers (that is obviously me and, to a lesser extent, Darryl), chose their samples based on complexity.

This is false.

In all my tests, what I worried the most was not if sample x or y were too much or too little complex. My worries were that the sample selection reflected a wide range of styles: rock, pop, classical orchestral, classical solo, a capella, jazz, hard rock, choir... (no J-Pop, ever!).

Why? Because my tests were always meant to represent reality. People don't build their music collections thinking "oh, will this track be a complex one on LAME, or will it go easy on it?". They have their collections built on personal taste, and that's it.

Of course, a side advantage of that decision was that complex samples and easy samples were more or less evenly distributed. A capella and classical solo are pretty much unavoidably easy. Likewise, hard rock and rock can be complex if correctly chosen (Waiting anyone?)

Then again, most of my tests were done on 128 kbps. If you focus on easy samples at that bitrate, you'll only end up with a handful of useless results, where all codecs are tied near the top. So, I'm only to be expected to favour complex samples there. To create a balance of sorts, I focused more on easy samples at the 64 and 32kbps tests I conduced. But I always favoured variety of styles over complexity issues.


Last but not least: I didn't read the whole thread. So, I won't comment on why your solutions won't work, as I suspect other knowledgeable people already did so. I'm on vacations now, on dial-up, and can't be arsed to read everything written here.


Peace;

Roberto.

Listening tests sample selection

Reply #42
Quote
Of course someone can hear a difference! Very severe artifacts are being created from transcoding. There were several listening test in this board with this subject from guruboolez and some other members. Try doing a search, you'll see that audible differences exist even when transcoding from a lot better lossy sources than mp3@128 or mp3@320.

My point is that MP3s that sit on my HDD are there for different purposes than just pure listening. It's a convenience.

Quote
Well, once again, use lossless...

We are obviously not talking about the same thing... 

Quote
You assume that the encoder doesn't do a good job at allocating bits. If you think you can help making things any better, the sources are there, take a look...

I am just saying that nothing is perfect, and encoders like everything else makes mistakes at times. I thought the point of a listening test was to point these mistakes out. If your preferred encoder decides to allocate more bits to the complex parts, at the same average bitrate it will allocate less bits to other parts. So there is a give, it is a tradeoff. By ABXing only complex samples, you just ignore it.

Quote
Well, feel free to organize a listening test with any kind of samples you decide are best. Good luck trying to get any meaningful results with the low complexity samples though (well you could make an ultra-low bitrate listening test and have meaningfull results...). People (including me) already have a hard time ABXing critical samples at medium (~128) bitrates.[a href="index.php?act=findpost&pid=319718"][{POST_SNAPBACK}][/a]

It's funny how this argument comes up every time someone says any kind of criticism about anything else. I was just trying to help by pointing out something that seems a little biaised. People seems to think I am attacking them and I am trying to discredit their test. I am just trying to help!

I am aware that 128kbps 'easy' samples are next to impossible to ABX. This is less true at 64kbps and certainly mostly false under.

rjamorin, thanks for your input. Here:
Quote
Regarding samples, I have enough now, thank you. It's a bit hard to decide which ones to use, since almost all of them are more or less killer samples.
[a href="index.php?act=findpost&pid=287547"][{POST_SNAPBACK}][/a]

I see some test being conducted on only killer samples, hence testing only the codecs on those killer samples. When I see results from tests where all VBR codecs allocate more bits on 16 samples out of 18, I tend to think something is not quite right. Especially when on both other samples (where VBR codecs are under the target bitrate) they score lower than the CBR codecs.

Now don't get me wrong, I'm not saying the test is a fraud or something. I am just trying to understand...

Listening tests sample selection

Reply #43
pieroxy: killer samples are not sample encoded with only +10% than the average bitrate. The recent collective tests were never performed with killer samples. Some are "difficult"¹, some are less "difficult" but still difficult, and others are not difficult at all. There are not killer samples.
If you want a killer stressing some VBR encoders, try maybe this one.


¹ if by difficult we mean "encoded with high bitrate".

Listening tests sample selection

Reply #44
Quote
pieroxy: killer samples are not sample encoded with only +10% than the average bitrate. The recent collective tests were never performed with killer samples. Some are "difficult"¹, some are less "difficult" but still difficult, and others are not difficult at all. There are not killer samples.
If you want a killer stressing some VBR encoders, try maybe this one.


¹ if by difficult we mean "encoded with high bitrate".
[a href="index.php?act=findpost&pid=321985"][{POST_SNAPBACK}][/a]

Thanks for shedding some light ont the subject. So you mean "killer" samples are not necessarily "difficult" samples...  Damn terminology 

Listening tests sample selection

Reply #45
Quote
killer samples are not sample encoded with only +10% than the average bitrate.

To add to the confusion, in my opinion, the kraftwerk sample used in the multiformat 128k test is a killer sample for mp3, wma and atrac3.

 

Listening tests sample selection

Reply #46
Quote
To add to the confusion, in my opinion, the kraftwerk sample used in the multiformat 128k test is a killer sample for mp3, wma and atrac3.[a href="index.php?act=findpost&pid=322117"][{POST_SNAPBACK}][/a]


Yes, it turned out to be a killer sample, but I didn't know it beforehand. I chose it because a) I only had DaFunk and wanted another electronic sample, b) Man Machine rocks and c) it was proposed to me by my master himself.

Interestingly, iTunes performed very, very well on this sample, considering it faithfully stuck to 128kbps.