Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Public, Multiformat Listening Test @ 64 kbps - OPEN (Read 119803 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #75 –
Off-topic,

Since the original thread was closed, and there is still a sense of doubt regarding the legality of disclosing the results of which the MS codec WMA 10 pro would be included, I would suggest that after the test is finished, the results be disclosed, but do not include the rating of WMA 10 pro yet.

If according to this test, WMA 10 pro be either superior/inferior to any of the other codecs, then in the next discussions, the moderator and other members may state that:

"IMHO, codec X is better/worse than WMA 10 pro"

without being thrown with TOS#8. Just don't disclose the results online (but hey, you can PM anytime! just pass the message).

It's better this way than being thrown with a subpoena or court orders or warrants of arrest...

Let the MS guys worry about the hidden result, and let us just have our final announcement of the results without WMA like saying (for example):

"HE-AAC came out No. 1. WMA 10 pro is included in the test, but we are not going to disclose its score. HE-AAC is still No. 1. Don't use WMA 10 Pro--it sucks" or if it's favourable to them, just keep a mum.

They won't stand a chance with that one! Har Har Har!
"Listen to me...
Never take unsolicited advice..."

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #76 –
Quote from: Rio on
Off-topic,

Since the original thread was closed, and there is still a sense of doubt regarding the legality of disclosing the results of which the MS codec WMA 10 pro would be included, I would suggest that after the test is finished, the results be disclosed, but do not include the rating of WMA 10 pro yet.


I must have missed that discussion... why would publishing the results of this listening test be illegal? Because Microsoft pwnz the internets?

Edit: I've just looked through the pre-test thread and couldn't find any such discussion... could you fill me in?
davidnaylor.org

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #77 –
There appears to be a recording/ripping "click" artifact in "Sample02 - Bachpsichord" just after 4 seconds. While its effect is largely nullified since it appears in all samples, I am able to "cheat" in ABXing since this short burst of random noise sounds noticeably different when encoded. I might skip this sample because I'm ABXing based on a technicality. Any advice for this situation?

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #78 –
Quote from: Whelkman on
There appears to be a recording/ripping "click" artifact in "Sample02 - Bachpsichord" just after 4 seconds. While its effect is largely nullified since it appears in all samples, I am able to "cheat" in ABXing since this short burst of random noise sounds noticeably different when encoded. I might skip this sample because I'm ABXing based on a technicality. Any advice for this situation?


Well, you could zoom in on some other part of the track and ignore the click.
davidnaylor.org

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #79 –
Quote from: naylor83 on
Well, you could zoom in on some other part of the track and ignore the click.

That's true, but, personality-wise, it's not easy for me to give up advantages when I know they exist. It's like pretending to be shocked at a movie's end when you solved it in the first act. Now that this artifact is known, we cannot be confident others aren't also ABXing based on it, consciously or unconsciously. I'll sleep on it, but I'll probably declare this one a five-way tie (by not submitting anything).

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #80 –
So you say that you can rate a codec based on one click? It's like saying Car A is better than Car B because of one screw that doesn't even have a "vital" function. I understand that some people focus on certain parts of the music and rate a coded depending how that specific part sounds like, but rating a codec based on something that isn't even intended to be part of the music (and wouldn't disturb your listening experience) is weird.

Edit: Bear in mind that you don't have to ABX only, but also rank each codec.

Quote from: Rio on
Since the original thread was closed, and there is still a sense of doubt regarding the legality of disclosing the results of which the MS codec WMA 10 pro would be included, I would suggest that after the test is finished, the results be disclosed, but do not include the rating of WMA 10 pro yet.


Pardon?  I would also like to know where you got that information from regarding this listening test in 2007. There are no such claims that wouldn't allow me to disclose the results in the WMP 11 final package. And even if, in Germany, EULAs have no legal meaning.

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #81 –
Quote from: Sebastian Mares on
So you say that you can rate a codec based on one click? It's like saying Car A is better than Car B because of one screw that doesn't even have a "vital" function.

Were I evaluating eighteen "pristine" cars and one of the cars had a noticeable scratch, I'd remove it from testing because the other seventeen lacked obvious physical defects, just like I am doing with this sample set.

Quote from: Sebastian Mares on
Edit: Bear in mind that you don't have to ABX only, but also rank each codec.

5-way tie. Honestly, it's not far from it in any case. I had a more difficult time with that particular sample than others.

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #82 –
Quote from: Whelkman on
Quote from: Sebastian Mares on
So you say that you can rate a codec based on one click? It's like saying Car A is better than Car B because of one screw that doesn't even have a "vital" function.

Were I evaluating eighteen "pristine" cars and one of the cars had a noticeable scratch, I'd remove it from testing because the other seventeen lacked obvious physical defects, just like I am doing with this sample set.

Quote from: Sebastian Mares on
Edit: Bear in mind that you don't have to ABX only, but also rank each codec.

5-way tie. Honestly, it's not far from it in any case. I had a more difficult time with that particular sample than others.


I'm with you on this one. If there is a CD-style click in one of the samples (haven't got to nr 2 myself yet) then I don't understand why it has been included...

Quote
but rating a codec based on something that isn't even intended to be part of the music (and wouldn't disturb your listening experience) is weird.


Not so weird if you ask me. Simple human nature to pick the easy way out. The question is, why was this sample included, if it contains a ripping error? (Does it?)
davidnaylor.org

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #83 –
I didn't notice there was an issue and the sample was available for use in listening tests. And as I said - you are supposed to rate how the audio files sound, it's not all about ABXing. Yes, you may "cheat" now by successfully ABXing and therefore making ABC/HR disable the reference sliders, but then do you rate the codecs based on a few milliseconds? On the other hand, you can test for pre-echo in this way or how encoders deal with vinyl clicks / ripping errors.  Personally, I don't see this as a huge issue.

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #84 –
Quote from: Sebastian Mares on
I didn't notice there was an issue and the sample was available for use in listening tests. And as I said - you are supposed to rate how the audio files sound, it's not all about ABXing. Yes, you may "cheat" now by successfully ABXing and therefore making ABC/HR disable the reference sliders, but then do you rate the codecs based on a few milliseconds? On the other hand, you can test for pre-echo in this way or how encoders deal with vinyl clicks / ripping errors.


Well, I may be doing things wrong, but that's basically what I do. I find a position that I can ABX and rate the samples at this position.

If I would just listen to the whole samples and rate them overall, they would all be 5.0 except the low anchor.
davidnaylor.org

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #85 –
Focusing on something like the first couple of seconds is OK in my opinion, but focusing on one single click to rate a sample that has several seconds - well, this is not the way it should be done. That click has only several milliseconds - even it the click was 1 second long, it would only make out 4% of the sample. And like I said, what if the sample was ripped off a vinyl?

Edit: And BTW, if you think all 5 tracks sound the same, why not submit the results even if it's a 5-way-tie?

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #86 –
Quote from: Sebastian Mares on
Edit: And BTW, if you think all 5 tracks sound the same, why not submit the results even if it's a 5-way-tie?


Because I want to prove to myself that I can tell the difference. That's half the reason we're all doing this, right?

Edit: Also, I wan't to help find the best codec. And if I don't zoom in on the details, how will I know which is best?
davidnaylor.org

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #87 –
OK, your POV. As I said, I don't see this as a problem. I have no idea if that rip is from a vinyl or not, so maybe it's not clipping or an extraction error at all. You can rate that click if you want and focus on the click alone like you are trying to teach me that it's the best way, or you can think about whether or not that click is really so important for your overall listening experience / pleasure. Personally, I would either ignore it or rate it together with the rest of the music (or at least some seconds if not the whole sample) and deal with possible artifacts like with any other sample (microattacks, preecho...). I would also like to end the debate about sample 2 at this point since I see no point in continuing such a discussion.

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #88 –
Quote from: Sebastian Mares on
OK, your POV. As I said, I don't see this as a problem. I have no idea if that rip is from a vinyl or not, so maybe it's not clipping or an extraction error at all. You can rate that click if you want and focus on the click alone like you are trying to teach me that it's the best way, or you can think about whether or not that click is really so important for your overall listening experience / pleasure. Personally, I would either ignore it or rate it together with the rest of the music (or at least some seconds if not the whole sample) and deal with possible artifacts like with any other sample (microattacks, preecho...). I would also like to end the debate about sample 2 at this point since I see no point in continuing such a discussion.


No need to get all upset or angry. I'm not trying to teach anyone anything, I was merely joining in a discussion I thought was interesting, which is what I thought forum boards were for. I won't rate sample 2 from that single click if it is an obvious fault of some kind.

And yes, I also listen to a couple of seconds when rating the samples. (Although I would say that the deciding sound is usually only some fraction of a second - i.e. a snare, a cymbal, etc.

Edit: Perhaps I should explain further: I listen to the whole sample, but focus on the "weak spots".
davidnaylor.org

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #89 –
Quote from: naylor83 on
Edit: Also, I wan't to help find the best codec. And if I don't zoom in on the details, how will I know which is best?


Finding the best codec is a question of averages, and discarding worst-case scenarios. ABXing is only useful to detect differences, not quality.

I think rating this sample taking how the encoders encode the artifact is useful, just like how we rate preecho on some hard-samples. What cannot be done is rating the whole sample only by this, if there are other artifacts present aswell.  (don't know, haven't done it yet).

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #90 –
Quote from: Whelkman on
There appears to be a recording/ripping "click" artifact in "Sample02 - Bachpsichord" just after 4 seconds. While its effect is largely nullified since it appears in all samples, I am able to "cheat" in ABXing since this short burst of random noise sounds noticeably different when encoded. I might skip this sample because I'm ABXing based on a technicality. Any advice for this situation?


I'm the 'author' of this sample, ripped several years ago.
If you listen closely to this sample you can ear several clicks that occur on the whole sample - and on the whole album if you got it. It's not a ripping issue nor a random noise: it's a mechanical sound. Like most instruments on earth harpsichord sound isn't electronically generated and the process required to create a note from the instrument isn't completely noise-free. This kind of annoyance are very common with solo instrumental music and are often magnified by the recording conditions (microphones close to the performer/instrument): fingers on the flute, "breath" of the organ, fingers on the bow of guitars/luth, etc...

Ah yes, these small details may cause annoying artefacts on some situations like low-bitrate encoding.

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #91 –
Quote from: naylor83 on
Edit: Perhaps I should explain further: I listen to the whole sample, but focus on the "weak spots".


Yes, and everyone, including myself, do that too. However, our views of "weak spots" seem to differ.

Quote from: guruboolez on
[...]


Hey, you're alive! Long time since I last saw you online here.

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #92 –
Quote from: Sebastian Mares on
Quote from: naylor83 on

Edit: Perhaps I should explain further: I listen to the whole sample, but focus on the "weak spots".

Yes, and everyone, including myself, do that too. However, our views of "weak spots" seem to differ.


Umm... ok...? BTW, now that we know the mysterious "click" is part of the recording, I'd say it's quite fair to use it as part of the rating. I wouldn't have otherwise, though.
davidnaylor.org

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #93 –
That's the thing I don't understand - why wouldn't you? Even if it was vinyl clicking or clipping or extraction errors - why not? Maybe it was intentionally there to see how codecs behave when fed with such material? In my first test we even had a transcoded sample.

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #94 –
Quote from: Sebastian Mares on
That's the thing I don't understand - why wouldn't you? Even if it was vinyl clicking or clipping or extraction errors - why not? Maybe it was intentionally there to see how codecs behave when fed with such material? In my first test we even had a transcoded sample.


Oh, right. I misunderstood you. I thought you wouldn't either... hmm.. well, I guess I feel a sound codec shouldn't have to perform well on such things as ripping errors. Anyway, no big deal. I guess, in general, a codec which copes well with a click will also cope well with certain types of sounds in music, so I guess you're right
davidnaylor.org

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #95 –
Quote from: guruboolez on
If you listen closely to this sample you can ear several clicks that occur on the whole sample

Thanks. Upon further investigation I discovered a few more of these clicks. I had suspicion they existed in the source but couldn't pass judgment without original material. I am familiar with orchestral noise and pitfalls of "one take" recordings, but the clicks in this case seem magnified.

Quote from: naylor83 on
I'd say it's quite fair to use it as part of the rating. I wouldn't have otherwise, though.

I agree. Now that the artifacts are known to be part of the original recording and not extraction errors, its testing use is acceptable.

Quote from: Sebastian Mares on
That's the thing I don't understand - why wouldn't you? Even if it was vinyl clicking or clipping or extraction errors - why not?

Correct me if I misunderstand, but the goal of this test is to see how well these codecs perform on typical well mastered, well ripped general consumer material. Certainly the bulk of the samples within reflect that viewpoint; is anything in here recorded from tape, vinyl, or anything that deviates significantly from the Redbook format? Analog and extraction artifacts force the listener to focus on outlier cases not typical of modern mastering processes. If it is indeed your intention to gauge low bitrate performance on mastering errors, then I misunderstood the goal. I brought the issue up because I believed I found a flaw in the test.

In any case, I disagree with the suggestion to ignore the clicks and trudge through as if they don't exist. Doing so forces the listener to be "honest", which blind testing seeks to avoid, while simultaneously handcuffing the tester by forcing him to use the second best detection method. As it stands, Id' say this particular sample is valid though not ideal for generic testing. I believe the average (well mastered) orchestral track to be less noisy but am not an orchestral aficionado so I will defer to the experts on this one.

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #96 –
To clear up some things.

I did not notice those clicks just until you pointed out that there is one after the first 4 seconds. Then I listened carefully and noticed it indeed. I did not include the sample because of those clicks, but simply because it features a solo instrument (harpsichord) and because the sample is also quite old.

Even if the click was an extraction error, I fail to see why that would be a huge problem in the test. The only advantage one might have (if he even notices those clicks) is that he can, more or less, successfully ABX the original and all samples making ABC/HR disable the reference sliders. Then it's up to the user again to rate and I highly doubt that he can give fair ratings by considering only a couple of milliseconds of a 25 seconds track, and those milliseconds not even containing anything especially related to music. What would you have done if I included a sample taken from a vinyl? Aren't we allowed to use lossy codecs for that task? Personally, I copied most of my audio tapes to 80 kbps MP3 because it's suitable for my ears.

Quote
Correct me if I misunderstand, but the goal of this test is to see how well these codecs perform on typical well mastered, well ripped general consumer material.


Really? No where did I mention well mastered and well ripped general consumer material. Red Hot Chili Pepper's "Californication" is everything else than well mastered, but that didn't keep me from encoding it to MP3, even though all tracks clip like hell.

So - all in all it's your choice if you rate the click or not, but not submitting the results for the track only because of some clicks is something I don't understand. I also wouldn't even give much importance to those damn clicks, even if they were extraction errors (which they are not thanks to guru's clarification), simply because it doesn't change my listening experience if a certain click suffers from pre-echo or not. The annoying thing is the click itself that is also present in the original and not some artifact that changes the sound of a simple click if the rest of the file is transparent.

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #97 –
@Sebastian

I'm sorry I got confused regarding the legality of the disclosure of the results of this test using WMA Pro 10.  I just picked up from where the other (closed) thread left off.  At any rate, since there would be any issue regarding it...

then ROCK AND ROLL!!!

Can't wait to see the results.  Much as I would like to participate, I don't have the minimum equipment.

"Listen to me...
Never take unsolicited advice..."

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #98 –
Quote from: Rio on
Much as I would like to participate, I don't have the minimum equipment.

You might be surprised. I'm plowing through with $35 Sennheiser PX-100 headphones and a noisy air conditioner a few feet away.

Public, Multiformat Listening Test @ 64 kbps - OPEN

Reply #99 –
Quote from: Rio on
Can't wait to see the results.  Much as I would like to participate, I don't have the minimum equipment.


IMO there is no need for high quality equipment for this test. Headphones (<$30) and descent integrated soundcard should be enough. Noisy environment is bigger problem IMO.