Skip to main content

Topic: LAME 3.96b regression examples (Read 28691 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • Jebus
  • [*][*][*][*][*]
  • Developer
LAME 3.96b regression examples
This is NOT a general 3.96b ABX results thread. This is just for people to post (any) samples which regress from 3.90.3. This means that I only want cases where 3.96 is WORSE than 3.90.3.

Please try to keep this on-topic. The idea is that if this thread stays empty, we should push for 3.96 to become official. And if it doesn't stay empty, samples will help the LAME devs with future versions.

Any and all settings are fine... just make sure you post how you encoded it, and link to samples.
  • Last Edit: 09 March, 2004, 04:47:14 AM by Jebus

  • Moitah
  • [*][*][*]
LAME 3.96b regression examples
Reply #1
Here's a sample which I can ABX (15/16) when encoded with 3.96 --preset standard: Dream Theater - The Dance of Eternity (8 sec, FLAC).  Listen to the part where the snair hits 4 times, about 1.5 seconds in.  I first noticed this while testing --preset 128, it happened in 3.96 but I didn't notice the same artifact in 3.90.3 --alt-preset 128.

  • PVNC
  • [*]
LAME 3.96b regression examples
Reply #2
I tried a 28-second sample from Tool's track (-) Ions. I chose this to test first, as it gets an average bitrate of 384 kbps using Vorbis GT3b1 at q5.

Original sample is in FLAC format.

I encoded mp3 versions using only the commandline --alt-preset 128 in RazorLame. I used LAME 3.90.3 modified and 3.96b1.

FLAC and mp3 versions have ReplayGain information applied.

I found that the 3.96b1 version has some kind of distortion during the lower volume parts of this clip - between the peak volume levels of the buzzing.  I was able to use this to ABX 3.96 from 3.90.3 8/8.

(-) Ions

  • guruboolez
  • [*][*][*][*][*]
  • Members (Donating)
LAME 3.96b regression examples
Reply #3
Isn't the upload forum a better place for this topic? Maybe should someone create a topic dedicated to this collection, or move this one.

  • Dologan
  • [*][*][*][*]
  • Members (Donating)
LAME 3.96b regression examples
Reply #4
Also, I would encourage people to also post improvements from the old encoder, since otherwise we might be skewing our preception. Usually, improvements also introduce other minor flaws that nonetheless end up making the thing a good trade-off.

  • PVNC
  • [*]
LAME 3.96b regression examples
Reply #5
Quote
Isn't the upload forum a better place for this topic? Maybe should someone create a topic dedicated to this collection, or move this one.


I agree.  I didn't know that my uploaded sample would end up on the portal.  It was rather embarrassing.

  • westgroveg
  • [*][*][*][*][*]
LAME 3.96b regression examples
Reply #6
Code: [Select]
ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\ringing-edit--APS-3.90.3.wav
   9 out of 10, pval = 0.011


Code: [Select]
ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\ringing-edit--APS-3.96b1.wav
  9 out of 10, pval = 0.011


Code: [Select]
ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\ringing-edit--APS-3.96b1.wav
   13 out of 16, pval = 0.011


There is a HF hiss present (0:01-0:02).
  • Last Edit: 08 March, 2004, 03:29:00 AM by westgroveg

  • westgroveg
  • [*][*][*][*][*]
LAME 3.96b regression examples
Reply #7
Code: [Select]
ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\castanets-edit-2-APS-3.96b1.wav
   12 out of 17, pval = 0.072


Guitar sounds watery.

  • westgroveg
  • [*][*][*][*][*]
LAME 3.96b regression examples
Reply #8
Moitah's The Dance of Eternity sample uploaded at HA,

  • Moitah
  • [*][*][*]
LAME 3.96b regression examples
Reply #9
Play - Is It Love (4 sec, FLAC) (try this link first, to save HA bandwidth)

--preset standard: This takes a lot of concentration for me to ABX, I got 22/32, 23/32, and 14/16 which adds up to 59/80 (p<0.001).  The first time the high note is played on the guitar (1.2 secs in) sounds smoother in the original.
  • Last Edit: 08 March, 2004, 01:40:15 PM by Moitah

  • amano
  • [*][*][*][*]
LAME 3.96b regression examples
Reply #10
Quote
Code: [Select]
ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\LAME test samples\castanets-edit-2-APS-3.96b1.wav
   12 out of 17, pval = 0.072


Guitar sounds watery.

As this is a regression thread, it dosn't make sense to just post just non-transparent samples. You have to compare them to the ABX results of 3.90.3.

And please don't post ABX results that aren't statistically valid. 12 out of 17 is certainly not a statistically convincing result. Otherwise this thread will be spammed with useless information.
  • Last Edit: 08 March, 2004, 03:37:48 PM by amano

  • amano
  • [*][*][*][*]
LAME 3.96b regression examples
Reply #11
Quote
Play - Is It Love (4 sec, FLAC) (try this link first, to save HA bandwidth)

--preset standard: This takes a lot of concentration for me to ABX, I got 22/32, 23/32, and 14/16 which adds up to 59/80 (p<0.001).  The first time the high note is played on the guitar (1.2 secs in) sounds smoother in the original.

I don't think that is valid to add up session results (maybe some other more in depth in the ABX science may tell us for sure).

And please compare your results with 3.90.3. To be fair chose the same session procedure. and try to ABX 3.90.3 against 3.96.

EDIT: I think everyone should act according to tigre's proposals in the LAME poll thread:

Quote
# Posting results in the thread requires:

    * Upload or link to sample
    * ABX results Original<->3.90.3, Original<->3.96, 3.90.3<->3.96, with detailed description of the difference(s)
    * Report about software/hardware used: Soundcard (resampling?), Player/ABXtool, DSPs (shouldn't be allowed, besides resampling to 48kHz and volume reduction/replaygain to prevent clipping <- both a 'must'), Amplifier, Speakers/Headphones

# Results must be confirmed by someone else before they are included in 'official' statistic, p-values must be < 0.05 for at least 2 people.
  • Last Edit: 08 March, 2004, 03:35:58 PM by amano

  • indybrett
  • [*][*][*][*][*]
  • Members (Donating)
LAME 3.96b regression examples
Reply #12
Since we have a regression thread, maybe we should also have a progression thread.
flac>fb2k>kernel streaming>audiophile 2496>magni>dt990 pro

  • westgroveg
  • [*][*][*][*][*]
LAME 3.96b regression examples
Reply #13
Quote
As this is a regression thread, it dosn't make sense to just post just non-transparent samples. You have to compare them to the ABX results of 3.90.3.

And please don't post ABX results that aren't statistically valid. 12 out of 17 is certainly not a statistically convincing result. Otherwise this thread will be spammed with useless information.

It was the only test I did with 3.96, I also did 1 ABX test with 3.90.3 got 5/5 thus my post of the sample.

Anyway your not a moderator, you didn't start the thread or even post a sample so I don't really see what your interest is, you just must like telling everyone what to do  .

(HA Wiki)
Quote
A difference is concluded to be heard when 13 correct identifications out of 16 trials is achieved

This is what we are expected to score for a valid problem sample, people, not what amano wants us to do.
  • Last Edit: 09 March, 2004, 02:13:22 AM by westgroveg

  • Jebus
  • [*][*][*][*][*]
  • Developer
LAME 3.96b regression examples
Reply #14
Westgroveg, i appreciate the results you are posting but PLEASE, i specifically requested that ONLY regressions from 3.90.3 be posted here, NOT general ABX results.

EDIT: Corrected Westgroveg's name
  • Last Edit: 09 March, 2004, 03:45:45 AM by Jebus

  • westgroveg
  • [*][*][*][*][*]
LAME 3.96b regression examples
Reply #15
Quote
Westgroveq, i appreciate the results you are posting but PLEASE, i specifically requested that ONLY regressions from 3.90.3 be posted here, NOT general ABX results.

My first sample as you can see I was able to ABX with both 3.90.3 & 3.96

My second admittedly is questionable but I also counter checked with 3.90.3, what's the problem?

& my nck is westgroveg.

  • freakngoat
  • [*][*][*]
LAME 3.96b regression examples
Reply #16
Quote
My first sample as you can see I was able to ABX with both 3.90.3 & 3.96

My second admittedly is questionable but I also counter checked with 3.90.3, what's the problem?

Sorry, don't mean to butt in, but they mean your ABX results aren't worse with 3.96 than they were with 3.90.3; they appear to be equal or even better. Regression testing means running past tests over again on a new version of software to make sure that nothing broke or got worse (regressed) with the newer version.

  • Jebus
  • [*][*][*][*][*]
  • Developer
LAME 3.96b regression examples
Reply #17
Quote
Quote
Westgroveq, i appreciate the results you are posting but PLEASE, i specifically requested that ONLY regressions from 3.90.3 be posted here, NOT general ABX results.

My first sample as you can see I was able to ABX with both 3.90.3 & 3.96

My second admittedly is questionable but I also counter checked with 3.90.3, what's the problem?

& my nck is westgroveg.

As Freakngoat just posted, the idea is that we ONLY post cases where 3.96 is WORSE than 3.90.3. You just posted cases where they both mess up - which isn't the point of this thread.

  • westgroveg
  • [*][*][*][*][*]
LAME 3.96b regression examples
Reply #18
Quote
Quote
My first sample as you can see I was able to ABX with both 3.90.3 & 3.96

My second admittedly is questionable but I also counter checked with 3.90.3, what's the problem?

Sorry, don't mean to butt in, but they mean your ABX results aren't worse with 3.96 than they were with 3.90.3; they appear to be equal or even better. Regression testing means running past tests over again on a new version of software to make sure that nothing broke or got worse (regressed) with the newer version.

Oh, sorry I misunderstood, I thought the thread asked, samples which are problematic for both 3.90.3 & 3.96, sorry Jebus.


  • Jebus
  • [*][*][*][*][*]
  • Developer
LAME 3.96b regression examples
Reply #19
np

I added a clarification sentence to the first post

  • westgroveg
  • [*][*][*][*][*]
LAME 3.96b regression examples
Reply #20
I just had a go at PVNC's (-) Ions sample (I used the contained mp3's for ABX)
Code: [Select]
ABX Results:
Original vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\3.96b1 (-) Ions.wav
   16 out of 16, pval < 0.001
C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\3.96b1 (-) Ions.wav vs C:\Documents and Settings\Administrator\Desktop\TEST SAMPLES\3.90.3 (-) Ions.wav
   10 out of 10, pval < 0.001


Both have a huge distortion (0:02-->0:04), 3.90.3 sounds slightly better.
  • Last Edit: 09 March, 2004, 05:54:19 AM by westgroveg

  • westgroveg
  • [*][*][*][*][*]
LAME 3.96b regression examples
Reply #21
Quote
Here's a sample which I can ABX (15/16) when encoded with 3.96 --preset standard: Dream Theater - The Dance of Eternity (8 sec, FLAC).  Listen to the part where the snair hits 4 times, about 1.5 seconds in.  I first noticed this while testing --preset 128, it happened in 3.96 but I didn't notice the same artifact in 3.90.3 --alt-preset 128.

I find these rock samples hard to ABX, can you tell us what type of artifact to listen for?

  • Moitah
  • [*][*][*]
LAME 3.96b regression examples
Reply #22
Quote
I find these rock samples hard to ABX, can you tell us what type of artifact to listen for?

I'm not sure what it's called, pre-echo maybe.  If you start with 128 it should be easier.

I need to do more testing with my second sample, it might be happening in 3.90.3 as well.

  • 2Bdecided
  • [*][*][*][*][*]
  • Developer
LAME 3.96b regression examples
Reply #23
Quote
I don't think that is valid to add up session results (maybe some other more in depth in the ABX science may tell us for sure).

You should always add all ABX result together for the same sample. It is statistically valid.

What you should never do is do more than one set of ABX tests, but only report some of them - you always need to report all the ABX testing you did with a sample. Selective ABX results are not valid.


Also, watching the result, and stopping when it's "statistically valid" isn't right either, but I haven't seen an explanation of what the correct statistics are in this case. The advice has always been to decide how many trials you will take before you start. But if you watch your result, then the "standard" statistics (which are found in most ABX programs) are wrong.

Cheers,
David.

  • Moitah
  • [*][*][*]
LAME 3.96b regression examples
Reply #24
My second sample isn't a regression, I ABXed 16/20 and 12/12 (total 28/32, pval < 0.001) for 3.90.3 --alt-preset standard (I think I was able to ABX 3.90.3 better because I knew what to listen for, not because it sounded worse than 3.96).