Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Public MP3 Listening Test @ 128 kbps - CLOSED (Read 162579 times) previous topic - next topic
0 Members and 2 Guests are viewing this topic.

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #125 –
Thank you for contributing Peter. Yes, blind testing is hard. Who'd have thought 128kbps MP3 could sound this good, hey?

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #126 –
ok, I just spent about 10 minutes on each set, on average, with certain sets getting a LOT more time.  I've found the anchor in every set, and defects in about half.  It was practically impossible for me to discrern any differences in the louder samples.

I only found 5 or 6 really obvious defects throughout all 14 sets.  In some sets, I would find some defects that were so minor, it took me dozens of plays to convince myself it was real, even though I could ABX them >90% of the time.  Problem is, I'm worried there are other defects this minor that I just didn't have the time or discipline to catch.  I would listen to 2 second excerpts over and over again to find them, and I can't do that with every 2-second slice of each sample of each set.  I'm worried that my "stumbling" upon a tiny defect will make one encoder look worse than it really is since I didn't have the time to find similiar defects in the others.

Is the idea that this stuff should even out since everyone else is in the same boat?  Or is every else taking this way t0o seriously like I am?

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #127 –
I don't want to drag this thread too off-topic, but am I missing something with Slashdot comments?  I don't use the site regularly, and I find navigating the comments near impossible.  I posted two comments eight hours ago (before bed) and it took me around fifteen minutes this morning to find one of them (still haven't found the other).  What's with all this unhiding?  It doesn't seem to work very well.  Maybe I just don't get the logic behind it; it's very confusing for a simple HA user like myself.

Edit: Even though I'm not a big RSS user (5' 8") I think I'd appreciate an RSS feed as opposed to the page "layout".
I'm on a horse.

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #128 –
Quote from: Synthetic Soul on
I don't want to drag this thread too off-topic, but am I missing something with Slashdot comments?  I don't use the site regularly, and I find navigating the comments near impossible.  I posted two comments eight hours ago (before bed) and it took me around fifteen minutes this morning to find one of them (still haven't found the other).  What's with all this unhiding?  It doesn't seem to work very well.  Maybe I just don't get the logic behind it; it's very confusing for a simple HA user like myself.


Slashdot uses a user moderation system.  Posts are ranked from 1 to 5.  You choose what "level" of posts to see.  Selecting a higher number means you will only see posts that other users consider useful,select a lower number to see more posts.  By default, new users' posts are given a 1.  As other users see your posts, they may decide your comment is useful and mod you up. Over time, if you get a lot of positive feedback, your default may rise to a 2 or 3. 

The idea is to prevent meaningless chatter and allow intelligent discourse to rise to the top, but honestly, slashdot isn't that cool any more.  Its kind of sad.

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #129 –
Slashdot hasn't been cool since 2002 - merely serviceable. The comment system used to be barely comprehensible, but the AJAX work they've been doing has been horrendous from what I've seen today.

I'm more saddened by the lack of knowledge displayed by a lot of the commentators.

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #130 –
OK, many thanks for the info guys.  I don't want this to go any further off-topic.

I've actually made it slightly more useful (on quick inspection) by registering and setting some options.  At least it seems I don't have to spend ten minutes unhiding things now.

While I'm here: congratulations to Sebastian, and thanks to kwanbis for his help in stirring up a little more interest.  I guess this whole thing is about altering people's misconceptions, and it seems like there's a few on Slashdot who could learn a few things.
I'm on a horse.

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #131 –
I posted on PSW and on Stereophile.

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #132 –
Quote from: Peter Sperl on
Is the idea that this stuff should even out since everyone else is in the same boat?  Or is every else taking this way t0o seriously like I am?


The way I understand it is this: Since the sample order is random in the test you shouldn't be able to prefer certain codecs "by accident", so in the long run all of them will be tested equally well. The less careful the test is conducted by the participants, the more results it will take to arrive at statistically meaningful results though, especially for good codecs which will probably be crammed closer together near 5.0.

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #133 –
Quote
ok, I just spent about 10 minutes on each set, on average, with certain sets getting a LOT more time. I've found the anchor in every set, and defects in about half. It was practically impossible for me to discrern any differences in the louder samples.

I only found 5 or 6 really obvious defects throughout all 14 sets. In some sets, I would find some defects that were so minor, it took me dozens of plays to convince myself it was real, even though I could ABX them >90% of the time. Problem is, I'm worried there are other defects this minor that I just didn't have the time or discipline to catch. I would listen to 2 second excerpts over and over again to find them, and I can't do that with every 2-second slice of each sample of each set. I'm worried that my "stumbling" upon a tiny defect will make one encoder look worse than it really is since I didn't have the time to find similiar defects in the others.

Is the idea that this stuff should even out since everyone else is in the same boat? Or is every else taking this way t0o seriously like I am?


Exactly the same here, my method is to listen to the low anchor, find the part of the sample where it REALLY fails, then listen to that excerpt with the other codecs to find artifacts. They all tend to exhibit the same failure modes, just to differing degrees. Some of the louder samples, I agree, are utterly impossible to abx at all.

Sam

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #134 –
But don't forget to also listen the whole sample for all encoders. It might happen that some encoders have artifacts at places where the low anchor is not so bad.

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #135 –
Quote from: sammydee on
Exactly the same here, my method is to listen to the low anchor, find the part of the sample where it REALLY fails, then listen to that excerpt with the other codecs to find artifacts. They all tend to exhibit the same failure modes, just to differing degrees.

I doubt it. Different codecs have different psymodels, and why should they produce artifacts in the same places?

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #137 –
Quote from: Peter Sperl on
...Many people are saying the test has recevied poor results due to difficulties with setup and the like. I disagree.  ....

I totally agree with you. Though I did have some problems setting up the test I think it at least on a Windows system and for people doing the test when reading this thread the setup shouldn't be a major problem. I also agree that it's the listening test itself which is the real challenge, and I had a hard time finding deviations from the original.
lame3995o -Q1.7 --lowpass 17

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #138 –
Quote from: halb27 on
Quote from: Peter Sperl on

...Many people are saying the test has recevied poor results due to difficulties with setup and the like. I disagree.  ....

I totally agree with you. Though I did have some problems setting up the test I think it at least on a Windows system and for people doing the test when reading this thread the setup shouldn't be a major problem. I also agree that it's the listening test itself which is the real challenge, and I had a hard time finding deviations from the original.
Uh say what?! How the heck can you say you disagree with people who say they find the test difficult to set up? If people find it difficult, then they do! Even Sebastian admitted there were some things which weren't totally clear.
//From the barren lands of the Northsmen

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #139 –
Quote from: DigitalDictator on
Uh say what?! How the heck can you say you disagree with people who say they find the test difficult to set up? If people find it difficult, then they do! Even Sebastian admitted there were some things which weren't totally clear.

We are talking about slashdoters. They are (where?) supposed to be more or less knowledgeable guys.

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #140 –
Quote from: kwanbis on
Quote from: DigitalDictator on
Uh say what?! How the heck can you say you disagree with people who say they find the test difficult to set up? If people find it difficult, then they do! Even Sebastian admitted there were some things which weren't totally clear.

We are talking about slashdoters. They are (where?) supposed to be more or less knowledgeable guys.
Well, I was one of those guys who found the tutorial a bit awkward. And then when someone says he disagrees that I find it to be a hassle to go through, it kind of ticked me off. That is not debatable really.

Once done, it won't be a problem I guess, but for first timers, and for people who have English as the second (or third) language, you REALLY have to pay attention to what you are doing. So that will probably discourage quite a few people from even wanting to participate. I think we have seen that already (i.e. extended test)
//From the barren lands of the Northsmen

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #141 –
Thanks a lot to the people who submitted. The people who did not start testing already but who would like to participate, it would be very nice if you could focus especially on samples: 5, 6, 8, 9, 10, 11, 13, 14.

Sample 1 already has 25 valid results, so please do not test it (of course, if you already did, you can submit your results for it, but if you just started, maybe invest time and effort in testing the aforementioned samples).

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #142 –
Thanks for providing the test. I've submitted only 2 samples. I found no trouble setting this up in Linux, besides needing to set my soundcard in settings.

The test itself was taking more time than I would have guessed. And then I skipped the difficult samples.

With headphones on I noticed some serious errors though that are easily detected by carefull listening. Only two codecs seem to do a real good job. My guess is that that's lame in both cases. But who knows what surprises this test will bring. With the new deadline, I'll see if I can do one or two samples extra.
A secure audio ripper for linux: code.google.com/p/rubyripper

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #143 –
Updated number of results per sample:

Sample01: 26
Sample02: 17
Sample03: 17
Sample04: 18
Sample05: 16
Sample06: 15
Sample07: 19
Sample08: 16
Sample09: 14
Sample10: 16
Sample11: 15
Sample12: 18
Sample13: 16
Sample14: 16

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #144 –
Hi, this is the first time I've done one of these - I'm going through every sample (I'm unemployed right now so have a lot of free time lol  ) and am finding it very interesting yet challenging. I'm sure my mediocre setup isn't helping things (noisy onboard sound + Sennheiser CX300s) - there's been a few samples when I couldn't tell any difference between the non-l3enc samples.

Just a question regarding making comments - will you know what way our samples were numbered when you look at the results? I'm just wondering because I've said made some comments comparing samples.

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #145 –
If I mistakenly identify a reference file as the compressed one, will my results be still valid for the other (correctly identified) samples of the set?

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #146 –
Quote from: Zilog Jones on
Just a question regarding making comments - will you know what way our samples were numbered when you look at the results? I'm just wondering because I've said made some comments comparing samples.


Yes.


Quote from: Cutter on
If I mistakenly identify a reference file as the compressed one, will my results be still valid for the other (correctly identified) samples of the set?


It depends on the number of ranked references. If you submit 5 results and 4 of them contain ranked references, I am afraid I will have to throw away all of them because I cannot be sure if the single one where you didn't rate the anchor wasn't only a coincidence.

The best thing you can do when you want to rate a sample is to perform an ABX test between the reference and the respective sample. If you get at least 7/8, ABC/HR will automatically disable the slider for the reference so you cannot rate it by mistake. If you cannot hear any difference in ABX, then it also doesn't make sense to rate that sample at all because ABC/HR already demonstrated you that the sample is transparent for you.

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #147 –
Thanks.

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #148 –
I've downloaded several of the samples. I can always spot the low anchor, but I'll be damned if I can hear a difference from the reference in any of the other encoders.

Could someone maybe tip me off as to which sample is easist to ABX, and what I should be listening for?

Public MP3 Listening Test @ 128 kbps - CLOSED

Reply #149 –
The differences can be found in more complex sounds or where different sounds overlap, especially when containing high frequencies. Try Sample05, between 2.0 and 16 secs there are several sounds with distortions.

I wonder if it's fair to give hints for a listening test