Skip to main content

Topic: Near-lossless / lossy FLAC (Read 124410 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • 2Bdecided
  • [*][*][*][*][*]
  • Developer
Near-lossless / lossy FLAC
Reply #25
SebG,

Ah, I see. Well, it might be interesting to try. It sounds like you're tempted to do it!


Josh PM'd me to point out that the FLAC frame/block size can be set from the command line using the -b command. I've just tried it, and it works as expected: with the MATLAB code changed to use 1024 sample blocks, it seems I get better compression, but I need to try more samples. On the one I tried (41_30sec) this shaved another 20% off, though that sample encodes 3% more efficiently in 1024 sample blocks than 4096 blocks anyway.

Cheers,
David.

  • jcoalson
  • [*][*][*][*][*]
  • Developer
Near-lossless / lossy FLAC
Reply #26
Slightly offtopic: Where'd you get the gspflac.dll??? I wanna!11!!! 
https://sourceforge.net/project/showfiles.p...group_id=165460

Cool!. Could you clarify the "Notes" paragraph in the frame header section, please? What blocksizes are allowed if it's a variable length block stream? I'd use "1000-1111 : 256 * (2^(n-8)) samples" but it looks like they are not allowed.
all that convoluted logic will be going away with the next version of FLAC thankfully.  I'll publish the details with the next release of FLAC (hopefully no later than july)

  • goodnews
  • [*][*][*]
  • Banned
Near-lossless / lossy FLAC
Reply #27
all that convoluted logic will be going away with the next version of FLAC thankfully.  I'll publish the details with the next release of FLAC (hopefully no later than july)

Slightly OT: Josh, is there a Intel Mac OS X version of FLAC 1.1.4 for download yet? All is still see is the PPC Mac version. Thanks!

What features will be included in next version of FLAC that you mentioned?
  • Last Edit: 13 June, 2007, 02:37:09 PM by goodnews

  • halb27
  • [*][*][*][*][*]
Near-lossless / lossy FLAC
Reply #28
I tried all the samples you provided, and couldn't abx them except for the second half of furious:
With my first trial I got at 4/4 with furious, but I missed several times with the following guesses.
With my second trial I got at 6/6, then 7/8, finally 8/10. Not a totally convincing result but maybe enough to show that furious lossy isn't totally transparent.
This is pretty similar to what I have learnt from wavPack lossy behavior for furious.
Anyway the difference is so subtle I can't really describe it. Just a minimal lack of precision may be. Not serious at all.

Can you provide Atem-lied, herding_calls, trumpet, harp40_1 please? These and all the other samples in the presumably more efficient short block version?

ADDED:
I forgot badvilbel. Can you provide badvilbel please?
  • Last Edit: 14 June, 2007, 03:09:27 AM by halb27
lame3995o -Q1

  • Bourne
  • [*][*][*][*][*]
  • Banned
Near-lossless / lossy FLAC
Reply #29
I kinda talked about this once before... I called it Virtual Lossless...
But unfortunately someone cut me out saying: Lossy is virtual lossless...

Thanks for your detailed explanation.
And I see a diference between LOSSY and VIRTUAL LOSSLESS.

  • Mitch 1 2
  • [*]
Near-lossless / lossy FLAC
Reply #30
Using a two-part file extension (e.g. .lossy.flac) should solve the compatibility problem, at the expense of longer filenames. Proper tagging is needed, of course, as filenames alone cannot be trusted.
lossyFLAC (lossyWAV -q 0; FLAC -b 512 -e)

  • 2Bdecided
  • [*][*][*][*][*]
  • Developer
Near-lossless / lossy FLAC
Reply #31
I tried all the samples you provided, and couldn't abx them except for the second half of furious:
With my first trial I got at 4/4 with furious, but I missed several times with the following guesses.
With my second trial I got at 6/6, then 7/8, finally 8/10. Not a totally convincing result but maybe enough to show that furious lossy isn't totally transparent.
This is pretty similar to what I have learnt from wavPack lossy behavior for furious.
Anyway the difference is so subtle I can't really describe it. Just a minimal lack of precision may be. Not serious at all.


Thank you for ABXing halb27.

I thought I could ABX the background noise at the end of Furious, but then failed. I'm not sure if I'm imagining it, or if it's nearly audible.


If you're in the mood to play, please try the attached files. I've played with the thresholding. I've also (intentionally) broken the lossy part by dithering the LSB itself so you can't cheat and look at the FLAC bitrate!

At least one of these files has more noise (so it's not as hard a job as it looks). At least one has less noise. So you should be able to ABX at least one, and maybe cannot ABX at least one. See what you think.

If you do have time to ABX, please decide upon the number of ABX tests before you start, and stick to that. As you probably know, selecting results or re-starting messes up the statistics.

Of course anyone is free to try.


Quote
Can you provide Atem-lied, herding_calls, trumpet, harp40_1 please? These and all the other samples in the presumably more efficient short block version?

ADDED:
I forgot badvilbel. Can you provide badvilbel please?


I'll do those as time permits. If I get chance before you've responded, I'll post them at the default settings. However, if your next response suggests I need to reduce the noise addition slightly, then I'll post them with a less aggressive setting.

Cheers,
David.

  • robert
  • [*][*][*][*][*]
  • Developer
Near-lossless / lossy FLAC
Reply #32
foobar has some problem with the sample 1_Furious:
Code: [Select]
Decoding failure at 0:01.486 (Unsupported format or corrupted file):
"F:\1_Furious.flac"


edit: Sorry, the downloaded file was broken on my side. I downloaded it again and Foobar plays it just fine.
  • Last Edit: 14 June, 2007, 08:17:30 AM by robert

  • halb27
  • [*][*][*][*][*]
Near-lossless / lossy FLAC
Reply #33
Will try them tonight.

BTW I don't concentrate on the background noise but on the accuracy of the 'main signal' in the second half of the track.

As for abxing if the question is 'is track X transparent?' I always allow for a second trial in case I have the impression that there is a difference and get at a result like 4/4 before I start to go wrong (due to possibly fatigueness). The number of guesses is fixed before a test (usually 10 guesses, sometimes 8), but in case I don't see what to concentrate on I often give up within the first guesses (usually with several wrong guesses at that time).
Of course my insisting on going through the test also depends on previous experience with the sample. From furious I know it's a serious problem for wavPack lossy and I have an idea what to look for. I'm also more emotionally engaged cause this is one of the more serious problems to wavPack lossy - I can imagine to get a similar kind of music in real life encoding, and it's not just a tiny amount of altered or increased hiss/noise but inaccuracy - though in your case very tiny as well.
lame3995o -Q1

  • 2Bdecided
  • [*][*][*][*][*]
  • Developer
Near-lossless / lossy FLAC
Reply #34
As for abxing if the question is 'is track X transparent?' I always allow for a second trial in case I have the impression that there is a difference and get at a result like 4/4 before I start to go wrong (due to possibly fatigueness). The number of guesses is fixed before a test (usually 10 guesses, sometimes 8), but in case I don't see what to concentrate on I often give up within the first guesses (usually with several wrong guesses at that time).
I'm sure an ABX / statistics guru will be along in a minute to explain exactly why that alters the statistics. I just recall that it does, and in a way that makes it much harder to hit a given level of confidence.

I think you could say something like "I will do 8, and they will not count, then I will do 16, and they will count" if you stuck to it.

Of course, you can always listen carefully, and A/B (not X!) until you believe you hear something. Then, and only then, do the pre-decided number of ABX trials. Then there's no question.

Cheers,
David.

  • Mark0
  • [*][*]
Near-lossless / lossy FLAC
Reply #35
I may be misunderstanding something, but: why linking this to FLAC at all?
I mean: this is a "sound simplifier", so to speak, so it's output could be very well fed to pretty any lossless (or even non lossless, even if this does a lot of less sense) coder, right?

Bye!

  • Ariakis
  • [*][*][*]
  • Members (Donating)
Near-lossless / lossy FLAC
Reply #36
It was originally designed to specifically exploit FLAC's wasted_bits handling for compression gain.

  • naturfreak
  • [*][*][*]
Near-lossless / lossy FLAC
Reply #37
My suggestion to further prevent confusion whether a FLAC file is from a lossless or lossy source:
Introduce a flag inside the FLAC (meta)data that indicate whether a file has a lossy or lossless source.

An user should be able to set that flag at encoding time only. It should be noneditable and unerasable inside the FLAC file (hex editor might be an execption).
  • Last Edit: 14 June, 2007, 09:59:58 AM by naturfreak

  • 2Bdecided
  • [*][*][*][*][*]
  • Developer
Near-lossless / lossy FLAC
Reply #38
It was originally designed to specifically exploit FLAC's wasted_bits handling for compression gain.


Exactly.

However, it turns out you can use it with wavpack lossless too, by using the --blocksize=1024 switch to force wavpack to use 1024 sample blocks. (or 4096 sample blocks for the examples I provided on the previous page). Thanks to David Bryant for providing this, and other useful tips via email (I will reply properly David).

Cheers,
David.
  • Last Edit: 14 June, 2007, 10:13:35 AM by 2Bdecided

  • halb27
  • [*][*][*][*][*]
Near-lossless / lossy FLAC
Reply #39
... I think you could say something like "I will do 8, and they will not count, then I will do 16, and they will count" if you stuck to it. ...

Hm... it's not like this: I say in advance I'll do 10 guesses with each trial in order to call two tracks abxable.
What shall I do in a situation when I have the impression (which doesn't count in the end but I can't ignore it) that there are audible differences, and this is backed up by the first guesses where I score 4/4? If after that I fail what does that mean? Failure can be due to the tracks not being able to abx, but also due to fatigueness or overconfidence according to the first results. Certainly this means I'm not very good at abxing, but with tracks hard to abx it happens to be like this - I can't change it. So what shall I do in such a situation? My solution is: I do a second trial and try harder. Can't see a better procedure. Taking the result of the first trial as the abxing result isn't the better alternative to me in case there's a suspicion that the tracks are abxable.
Sure if I allow for a second trial I could also allow for a third one, and so on. I see the point. But that's theory cause things are clear after the second trial.
  • Last Edit: 14 June, 2007, 03:05:05 PM by halb27
lame3995o -Q1

  • Mark0
  • [*][*]
Near-lossless / lossy FLAC
Reply #40

It was originally designed to specifically exploit FLAC's wasted_bits handling for compression gain.


Exactly.

However, it turns out you can use it with wavpack lossless too, by using the --blocksize=1024 switch to force wavpack to use 1024 sample blocks. (or 4096 sample blocks for the examples I provided on the previous page). Thanks to David Bryant for providing this, and other useful tips via email (I will reply properly David).

Cheers,
David.

Exactly. What I was meaning (sorry, my English isn't very good) is that the "simplification" made by your tool can enable similar gain in compression ratio with other similar tools. For example (using one of the posted sample):

Code: [Select]
                      WAV     FLAC      TAK       RAR
01_41_30sec_lossy   5.168KB  1.957KB  2.119KB   2.755KB
02_41_30sec         5.168KB  3.473KB  3.284KB   3.633KB


Maybe two hours from now a new Lossless compressor will surface that enjoy an even higher gain than FLAC. In the end, I don't see the link between a generic tool that "simplify" a sound file, and a particular encoder (for witch the action of the tool may result especially beneficial).

On the same way, I don't see the point about flaggin in some way a lossless encoded file that had as a input a WAV file altered in some way.
Even without using this tool, original files can be altered in a number of ways (badly equalized, or they can contains click/pop, can have some noise reduction effects applied, etc.)...

Bye!
  • Last Edit: 14 June, 2007, 10:49:27 AM by Mark0

  • halb27
  • [*][*][*][*][*]
Near-lossless / lossy FLAC
Reply #41
... However, it turns out you can use it with wavpack lossless too, by using the --blocksize=1024 switch to force wavpack to use 1024 sample blocks. ...

Wonderful.
Your idea is getting even more useful. Congratulations.
lame3995o -Q1

  • 2Bdecided
  • [*][*][*][*][*]
  • Developer
Near-lossless / lossy FLAC
Reply #42
Can you provide Atem-lied
I can't find it. Can you upload it please?

Cheers,
David.

  • Nick.C
  • [*][*][*][*][*]
  • Developer
Near-lossless / lossy FLAC
Reply #43
So, how soon before an executable version of SoundSimplifierâ„¢ (  ) is released to an expectant HA community? Inquiring (impatient) minds want to know!
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848| FLAC -5 -e -p -b 512 -P=4096 -S-

  • 2Bdecided
  • [*][*][*][*][*]
  • Developer
Near-lossless / lossy FLAC
Reply #44
I've uploaded herding_calls, trumpet, harp40_1, and badvilbel.


Here the lossy versions, and also links to the originals:

badvilbel:
[ Specified attachment is not available ]http://www.ff123.net/samples/badvilbel.flac

harp40_1:
[ Specified attachment is not available ]ftp://ftp.tnt.uni-hannover.de/pub/MPEG/au...am/harp40_1.wav

herding calls:
[ Specified attachment is not available ]http://www.hydrogenaudio.org/forums/index....showtopic=37002

trumpet:
[ Specified attachment is not available ]http://www.hydrogenaudio.org/forums/index....showtopic=39334


(I miss the HA problem samples library!)

Cheers,
David.
  • Last Edit: 14 June, 2007, 11:45:08 AM by 2Bdecided

  • TBeck
  • [*][*][*][*][*]
  • Developer
Near-lossless / lossy FLAC
Reply #45
...
Exactly. What I was meaning (sorry, my English isn't very good) is that the "simplification" made by your tool can enable similar gain in compression ratio with other similar tools. For example (using one of the posted sample):

Code: [Select]
                      WAV     FLAC      TAK       RAR
01_41_30sec_lossy   5.168KB  1.957KB  2.119KB   2.755KB
02_41_30sec         5.168KB  3.473KB  3.284KB   3.633KB


Maybe two hours from now a new Lossless compressor will surface that enjoy an even higher gain than FLAC. In the end, I don't see the link between a generic tool that "simplify" a sound file, and a particular encoder (for witch the action of the tool may result especially beneficial).
...

For optimum performance the intenal frame size of the encoder has to be taken into account when using the preprocessor. I suppose you haven't done this for TAK?

Possibly i will have to add an option to manually set the frame size for TAK files to get most out of the preprocessor. While TAK partitions each of it's fixed size frames into up to 5 variable size sub frames to adapt to signal changes, the "wasted bits" options works on the whole frame, which is too big (more than 4000 samples) to work well with the preprocessor.

  Thomas
  • Last Edit: 14 June, 2007, 11:58:29 AM by TBeck

  • 2Bdecided
  • [*][*][*][*][*]
  • Developer
Near-lossless / lossy FLAC
Reply #46
So, how soon before an executable version of SoundSimplifierâ„¢ (  ) is released to an expectant HA community? Inquiring (impatient) minds want to know!


I like the name!

I'm not keeping it back. I've attached the latest MATLAB script which I'm using to generate these samples. It executes very well if you have MATLAB!
(Though you need lots of memory for normal sized audio files, since there's no buffering, and you'll need to change waveread to wavread and wavewrite to wavwrite).

As for an efficient C/C++ implementation which could be compiled - that's beyond me.

I think we need some more listening tests before anyone puts that much effort in, but the job is open to anyone who wants it!

Cheers,
David.

  • smok3
  • [*][*][*][*][*]
  • Moderator
Near-lossless / lossy FLAC
Reply #47
i couldnt reliably abx the 1st set of samples, but there were some weird negative results on some like 1/8 or 2/8.
  • Last Edit: 14 June, 2007, 01:13:06 PM by smok3
PANIC: CPU 1: Cache Error (unrecoverable - dcache data) Eframe = 0x90000000208cf3b8
NOTICE - cpu 0 didn't dump TLB, may be hung

  • TBeck
  • [*][*][*][*][*]
  • Developer
Near-lossless / lossy FLAC
Reply #48

...
Exactly. What I was meaning (sorry, my English isn't very good) is that the "simplification" made by your tool can enable similar gain in compression ratio with other similar tools. For example (using one of the posted sample):

Code: [Select]
                      WAV     FLAC      TAK       RAR
01_41_30sec_lossy   5.168KB  1.957KB  2.119KB   2.755KB
02_41_30sec         5.168KB  3.473KB  3.284KB   3.633KB


Maybe two hours from now a new Lossless compressor will surface that enjoy an even higher gain than FLAC. In the end, I don't see the link between a generic tool that "simplify" a sound file, and a particular encoder (for witch the action of the tool may result especially beneficial).
...

For optimum performance the intenal frame size of the encoder has to be taken into account when using the preprocessor. I suppose you haven't done this for TAK?
...


I was curious...
I dowloaded "01_41_30sec_lossy" from here. Then i compressed it with TAK's default frame size and then with a frame size of 4096 Bytes. Results:

Code: [Select]
FLAC                2,004,157  Bytes
TAK Normal Default  2,023,188  Bytes
TAK Normal 4096     1,809,281  Bytes
TAK Turbo  4096     1,846,469  Bytes
  • Last Edit: 14 June, 2007, 01:19:58 PM by TBeck

  • Mark0
  • [*][*]
Near-lossless / lossy FLAC
Reply #49
For optimum performance the intenal frame size of the encoder has to be taken into account when using the preprocessor. I suppose you haven't done this for TAK?

Right. I have just compressed the two files "on a rush" to check if - as I supposed - the action of the SoundSimplifierâ„¢  would be interesting also for other encoders.
And it was, so much, as your results show even more. Thanks for taking the time for the "optimized" encoding.

Bye!
  • Last Edit: 14 June, 2007, 01:37:20 PM by Mark0