Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: 64 kbit/s Test has started! (Read 17230 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

64 kbit/s Test has started!

Reply #75 –
Quote
Originally posted by kdo
@ff123:

How long will you be accepting the results?

I made my first test today. And I already see I won't be able to do more than one test a day. most probably even not every day. (as I think I better abx everything, to be sure).

Also, would you like people to send every result one by one, or better all of them together when all are done?

-----------------------

and a suggestion for the abchr:
could be a good idea to make an option to save the current session in a humanly unreadable binary file, so that one could continue the interrupted test later.


I will accept results for a while yet, as they are still coming in strong (thanks Garf, for the slashdotting).  Submit results as best suits you.  About the option to save a session, I've added that to the ever growing list of suggested improvements.

ff123

64 kbit/s Test has started!

Reply #76 –
i downloaded atrain and beautyslept so far, uf, they are both so annoying i cant even care to evaluate, iam downloading one more...
PANIC: CPU 1: Cache Error (unrecoverable - dcache data) Eframe = 0x90000000208cf3b8
NOTICE - cpu 0 didn't dump TLB, may be hung

64 kbit/s Test has started!

Reply #77 –
Quote
I made my first test today. And I already see I won't be able to do more than one test a day. most probably even not every day. (as I think I better abx everything, to be sure).

Don't get too disheartened. Some of the samples are *much* harder to distinguish between than others -- I've tested 3 clips so far, and my scores range from 1.1 (barely listenable) to 5.0 (indistinguishable from the original)... and the very first clip I tested had two samples rated at 5, because I just couldn't hear a problem.

And of course, that's how it should be. At least *1* sample is supposed to be 'CD quality' at this bitrate, after all .

64 kbit/s Test has started!

Reply #78 –
For what it's worth, I've excused mysefl from the test (64kbps is too easy to tell which codec is which) but sent my predictions of outcome to ff123 before it started.

I find it odd, actually, that folks are considering Ogg the codec to beat when 64kbps is relatively very untested.  128-ish kbps has seen about two and a half years of solid tuning now.  The 64kbps mode is essentially the same one as from rc3 (which no one used because it was unlistenable ;-) plus noise normalization and somewhat more aggressive stereo to plow bits back into treble.  How  good it would turn out in the big bad world was a huge unknown; it sounded good on my primary test sample suite, but that's only about thirty samples.  I do think Ogg has at least an edge over the competition, even not considering that 64kbps has alot of tuning room left, but I also am not going to pretend Ogg wins every one of these samples.  Hell, I even picked WMA as winner for one, co-winner for another (classical solo instruments *should* be easy to encode... they're the only thing WMA doesn't run out of bits on and close its metallic lowpass like a rocket-powered garage door).

What I've found out so far: Ogg noise normalization can 'wheedle' when there's a single dominant note with lots of harmonics and no background noise (ala electronic piano/organ, solo string instrument or a single almost solo vocal with reverb).  This is due to noise normalization restoring a harmonic that should probably best be ignored.  Also, the same basic sample with the instrument far to one side of the stereo image with reverb bouncing back and forth is causing midrange noise to crop up in the center where it's not masked.

So, this test has two samples in it that go into the tuning suite because they currently sound, in my opinion, atrocious (well... actually, they should sound much better than they do, even though one of them gets murdered by all the encoders).  Two more are going in as well for brushing up as they sound good, but Ogg should win those tests and in my opinion it currently doesn't.  I'll tell you which ones they are after the test. 

As for the other samples.... well this is a test where everyone should have artifacts with the possible exception of the classical solo instruments; they're exceptionally easy due to the harmonic structure.  Ogg blows the Bach due to the NN wheedle, and I'll have to kill that.  The others... it's only a question of which artifacts people hate more.  Layla, for instance, will be an interesting sample for the comments.  ff123 can publish my pre-test predictions when this is all done :-)

64 kbit/s Test has started!

Reply #79 –
Can I echo two points:

1. If there's any doubt (and it appears, surprisingly, that there is), people must ABX before being allowed to grade less than 5.

(ff123: having ABXd correctly, can we have an indication of which one we're supposed to be grading on the main page - otherwise you have to listen carefully to both again! OK, maybe that's a good further test of our ability to hear a difference, but after ABX it's a waste of time.)


2. I take the point about the overall impression created by an encoder over an hour being a different matter from the artefacts that are audible within a 20 second test extract. But no one is giving you that hour's worth of music in this test; just the 20 seconds. So shut up and ABX it! (Yes, I know, you already did - that's why I dared to be so rude!).

You see, it's the audible differences that create the different impression over longer listening - you've just got to find them when ABXing - so long as some fall within the supplied clip, it's possible. (If they don't ocurr in the clip, then you won't know that the codec isn't transparent - in which case, for this clip, for you, it is tranparent, and that's all we can prove anyway!).


Some people talk about "vague differences, impressions and feelings" and it's pure placeabo. Others talk about the same things, and it's just because they've not learnt to spot and recognise the artefact(s) that are giving them that impression. However, the perceived difference is real. I believe that, to prevent contamination of the results from the first group (placeabo), we must require the second group to learn the skill and patience to ABX. Because, when all you have is a 20 second sample, then if it creates a different impression due to an audible difference, then you will be able to ABX it. Eventually.


Cheers,
David.

64 kbit/s Test has started!

Reply #80 –
What if someone plays two samples at blind-random several times and consistently ranks one as 5 and another as 4.5 on a 1-5 scale of quality (unbenknownst to him while he's doing it). Then he attempts to ABX these two samples, and fails to achieve a result that can confidently be considered non-random.

What would this mean? Do you think it would happen anyway?

64 kbit/s Test has started!

Reply #81 –
Quote
Originally posted by 2Bdecided
Can I echo two points:

1. If there's any doubt (and it appears, surprisingly, that there is), people must ABX before being allowed to grade less than 5.

(ff123: having ABXd correctly, can we have an indication of which one we're supposed to be grading on the main page - otherwise you have to listen carefully to both again! OK, maybe that's a good further test of our ability to hear a difference, but after ABX it's a waste of time.)


I dont completely agree with this:
If you abx a sample, you know how a sounds, how b sounds, and then you have to decide if you hear a or b.
With the ABC/HR, you just know how the original sounds!, and then you have to identify which of the two candidates it is.

For this reason, I would say that it is more difficult to find the correct one in an ABCHR than in an ABX, and letting the program tell you which is which, is cheating.


Quote
Originally posted by 2Bdecided


Some people talk about "vague differences, impressions and feelings" and it's pure placeabo. Others talk about the same things, and it's just because they've not learnt to spot and recognise the artefact(s) that are giving them that impression. However, the perceived difference is real. I believe that, to prevent contamination of the results from the first group (placeabo), we must require the second group to learn the skill and patience to ABX. Because, when all you have is a 20 second sample, then if it creates a different impression due to an audible difference, then you will be able to ABX it. Eventually.


if you ABX one sample, you are supposed to know which is the difference between the original and the lossy sample. If you know it, the ABCHR test should be so easy for you.
If you can't ABX, but you feel you hear a difference,  it shows that
a) your equipment doesn't let you distinguish it correctly after different retries
b) you're tired
c) you don't pay enough attention
d) there's something in the sample that disturbs you and makes you fail.
There is a sample in this test where I confused the original with the lossy one some times, because there was already some sort rough sound that was more accentuated in the lossy one. I was only sure which was X when hearing to the lossy one. With the original, I was not sure and failed to identify it sometimes. This doesn't happen with ABCHR.

e) ok... if you want, I can add placebo too.

64 kbit/s Test has started!

Reply #82 –
There are enough results at this point for each sample that I could close the test right now.  But if there are people who would still like to submit results, send them in, and I'll accept them for another day.  Thanks to everybody who participated.

ff123