Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Near-lossless / lossy FLAC (Read 176364 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Near-lossless / lossy FLAC

Reply #25
SebG,

Ah, I see. Well, it might be interesting to try. It sounds like you're tempted to do it!


Josh PM'd me to point out that the FLAC frame/block size can be set from the command line using the -b command. I've just tried it, and it works as expected: with the MATLAB code changed to use 1024 sample blocks, it seems I get better compression, but I need to try more samples. On the one I tried (41_30sec) this shaved another 20% off, though that sample encodes 3% more efficiently in 1024 sample blocks than 4096 blocks anyway.

Cheers,
David.

Near-lossless / lossy FLAC

Reply #26
Slightly offtopic: Where'd you get the gspflac.dll??? I wanna!11!!! 
https://sourceforge.net/project/showfiles.p...group_id=165460

Cool!. Could you clarify the "Notes" paragraph in the frame header section, please? What blocksizes are allowed if it's a variable length block stream? I'd use "1000-1111 : 256 * (2^(n-8)) samples" but it looks like they are not allowed.
all that convoluted logic will be going away with the next version of FLAC thankfully.  I'll publish the details with the next release of FLAC (hopefully no later than july)

Near-lossless / lossy FLAC

Reply #27
all that convoluted logic will be going away with the next version of FLAC thankfully.  I'll publish the details with the next release of FLAC (hopefully no later than july)

Slightly OT: Josh, is there a Intel Mac OS X version of FLAC 1.1.4 for download yet? All is still see is the PPC Mac version. Thanks!

What features will be included in next version of FLAC that you mentioned?

Near-lossless / lossy FLAC

Reply #28
I tried all the samples you provided, and couldn't abx them except for the second half of furious:
With my first trial I got at 4/4 with furious, but I missed several times with the following guesses.
With my second trial I got at 6/6, then 7/8, finally 8/10. Not a totally convincing result but maybe enough to show that furious lossy isn't totally transparent.
This is pretty similar to what I have learnt from wavPack lossy behavior for furious.
Anyway the difference is so subtle I can't really describe it. Just a minimal lack of precision may be. Not serious at all.

Can you provide Atem-lied, herding_calls, trumpet, harp40_1 please? These and all the other samples in the presumably more efficient short block version?

ADDED:
I forgot badvilbel. Can you provide badvilbel please?
lame3995o -Q1.7 --lowpass 17

Near-lossless / lossy FLAC

Reply #29
I kinda talked about this once before... I called it Virtual Lossless...
But unfortunately someone cut me out saying: Lossy is virtual lossless...

Thanks for your detailed explanation.
And I see a diference between LOSSY and VIRTUAL LOSSLESS.

Near-lossless / lossy FLAC

Reply #30
Using a two-part file extension (e.g. .lossy.flac) should solve the compatibility problem, at the expense of longer filenames. Proper tagging is needed, of course, as filenames alone cannot be trusted.
lossyFLAC (lossyWAV -q 0; FLAC -b 512 -e)

Near-lossless / lossy FLAC

Reply #31
I tried all the samples you provided, and couldn't abx them except for the second half of furious:
With my first trial I got at 4/4 with furious, but I missed several times with the following guesses.
With my second trial I got at 6/6, then 7/8, finally 8/10. Not a totally convincing result but maybe enough to show that furious lossy isn't totally transparent.
This is pretty similar to what I have learnt from wavPack lossy behavior for furious.
Anyway the difference is so subtle I can't really describe it. Just a minimal lack of precision may be. Not serious at all.


Thank you for ABXing halb27.

I thought I could ABX the background noise at the end of Furious, but then failed. I'm not sure if I'm imagining it, or if it's nearly audible.


If you're in the mood to play, please try the attached files. I've played with the thresholding. I've also (intentionally) broken the lossy part by dithering the LSB itself so you can't cheat and look at the FLAC bitrate!

At least one of these files has more noise (so it's not as hard a job as it looks). At least one has less noise. So you should be able to ABX at least one, and maybe cannot ABX at least one. See what you think.

If you do have time to ABX, please decide upon the number of ABX tests before you start, and stick to that. As you probably know, selecting results or re-starting messes up the statistics.

Of course anyone is free to try.


Quote
Can you provide Atem-lied, herding_calls, trumpet, harp40_1 please? These and all the other samples in the presumably more efficient short block version?

ADDED:
I forgot badvilbel. Can you provide badvilbel please?


I'll do those as time permits. If I get chance before you've responded, I'll post them at the default settings. However, if your next response suggests I need to reduce the noise addition slightly, then I'll post them with a less aggressive setting.

Cheers,
David.

Near-lossless / lossy FLAC

Reply #32
foobar has some problem with the sample 1_Furious:
Code: [Select]
Decoding failure at 0:01.486 (Unsupported format or corrupted file):
"F:\1_Furious.flac"


edit: Sorry, the downloaded file was broken on my side. I downloaded it again and Foobar plays it just fine.

Near-lossless / lossy FLAC

Reply #33
Will try them tonight.

BTW I don't concentrate on the background noise but on the accuracy of the 'main signal' in the second half of the track.

As for abxing if the question is 'is track X transparent?' I always allow for a second trial in case I have the impression that there is a difference and get at a result like 4/4 before I start to go wrong (due to possibly fatigueness). The number of guesses is fixed before a test (usually 10 guesses, sometimes 8), but in case I don't see what to concentrate on I often give up within the first guesses (usually with several wrong guesses at that time).
Of course my insisting on going through the test also depends on previous experience with the sample. From furious I know it's a serious problem for wavPack lossy and I have an idea what to look for. I'm also more emotionally engaged cause this is one of the more serious problems to wavPack lossy - I can imagine to get a similar kind of music in real life encoding, and it's not just a tiny amount of altered or increased hiss/noise but inaccuracy - though in your case very tiny as well.
lame3995o -Q1.7 --lowpass 17

Near-lossless / lossy FLAC

Reply #34
As for abxing if the question is 'is track X transparent?' I always allow for a second trial in case I have the impression that there is a difference and get at a result like 4/4 before I start to go wrong (due to possibly fatigueness). The number of guesses is fixed before a test (usually 10 guesses, sometimes 8), but in case I don't see what to concentrate on I often give up within the first guesses (usually with several wrong guesses at that time).
I'm sure an ABX / statistics guru will be along in a minute to explain exactly why that alters the statistics. I just recall that it does, and in a way that makes it much harder to hit a given level of confidence.

I think you could say something like "I will do 8, and they will not count, then I will do 16, and they will count" if you stuck to it.

Of course, you can always listen carefully, and A/B (not X!) until you believe you hear something. Then, and only then, do the pre-decided number of ABX trials. Then there's no question.

Cheers,
David.

Near-lossless / lossy FLAC

Reply #35
I may be misunderstanding something, but: why linking this to FLAC at all?
I mean: this is a "sound simplifier", so to speak, so it's output could be very well fed to pretty any lossless (or even non lossless, even if this does a lot of less sense) coder, right?

Bye!

Near-lossless / lossy FLAC

Reply #36
It was originally designed to specifically exploit FLAC's wasted_bits handling for compression gain.

Near-lossless / lossy FLAC

Reply #37
My suggestion to further prevent confusion whether a FLAC file is from a lossless or lossy source:
Introduce a flag inside the FLAC (meta)data that indicate whether a file has a lossy or lossless source.

An user should be able to set that flag at encoding time only. It should be noneditable and unerasable inside the FLAC file (hex editor might be an execption).

Near-lossless / lossy FLAC

Reply #38
It was originally designed to specifically exploit FLAC's wasted_bits handling for compression gain.


Exactly.

However, it turns out you can use it with wavpack lossless too, by using the --blocksize=1024 switch to force wavpack to use 1024 sample blocks. (or 4096 sample blocks for the examples I provided on the previous page). Thanks to David Bryant for providing this, and other useful tips via email (I will reply properly David).

Cheers,
David.

Near-lossless / lossy FLAC

Reply #39
... I think you could say something like "I will do 8, and they will not count, then I will do 16, and they will count" if you stuck to it. ...

Hm... it's not like this: I say in advance I'll do 10 guesses with each trial in order to call two tracks abxable.
What shall I do in a situation when I have the impression (which doesn't count in the end but I can't ignore it) that there are audible differences, and this is backed up by the first guesses where I score 4/4? If after that I fail what does that mean? Failure can be due to the tracks not being able to abx, but also due to fatigueness or overconfidence according to the first results. Certainly this means I'm not very good at abxing, but with tracks hard to abx it happens to be like this - I can't change it. So what shall I do in such a situation? My solution is: I do a second trial and try harder. Can't see a better procedure. Taking the result of the first trial as the abxing result isn't the better alternative to me in case there's a suspicion that the tracks are abxable.
Sure if I allow for a second trial I could also allow for a third one, and so on. I see the point. But that's theory cause things are clear after the second trial.
lame3995o -Q1.7 --lowpass 17

Near-lossless / lossy FLAC

Reply #40

It was originally designed to specifically exploit FLAC's wasted_bits handling for compression gain.


Exactly.

However, it turns out you can use it with wavpack lossless too, by using the --blocksize=1024 switch to force wavpack to use 1024 sample blocks. (or 4096 sample blocks for the examples I provided on the previous page). Thanks to David Bryant for providing this, and other useful tips via email (I will reply properly David).

Cheers,
David.

Exactly. What I was meaning (sorry, my English isn't very good) is that the "simplification" made by your tool can enable similar gain in compression ratio with other similar tools. For example (using one of the posted sample):

Code: [Select]
                      WAV     FLAC      TAK       RAR
01_41_30sec_lossy   5.168KB  1.957KB  2.119KB   2.755KB
02_41_30sec         5.168KB  3.473KB  3.284KB   3.633KB


Maybe two hours from now a new Lossless compressor will surface that enjoy an even higher gain than FLAC. In the end, I don't see the link between a generic tool that "simplify" a sound file, and a particular encoder (for witch the action of the tool may result especially beneficial).

On the same way, I don't see the point about flaggin in some way a lossless encoded file that had as a input a WAV file altered in some way.
Even without using this tool, original files can be altered in a number of ways (badly equalized, or they can contains click/pop, can have some noise reduction effects applied, etc.)...

Bye!

Near-lossless / lossy FLAC

Reply #41
... However, it turns out you can use it with wavpack lossless too, by using the --blocksize=1024 switch to force wavpack to use 1024 sample blocks. ...

Wonderful.
Your idea is getting even more useful. Congratulations.
lame3995o -Q1.7 --lowpass 17


Near-lossless / lossy FLAC

Reply #43
So, how soon before an executable version of SoundSimplifier™ (  ) is released to an expectant HA community? Inquiring (impatient) minds want to know!
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

Near-lossless / lossy FLAC

Reply #44
I've uploaded herding_calls, trumpet, harp40_1, and badvilbel.


Here the lossy versions, and also links to the originals:

badvilbel:
[attachment=3324:attachment]http://www.ff123.net/samples/badvilbel.flac

harp40_1:
[attachment=3325:attachment]ftp://ftp.tnt.uni-hannover.de/pub/MPEG/au...am/harp40_1.wav

herding calls:
[attachment=3326:attachment]http://www.hydrogenaudio.org/forums/index....showtopic=37002

trumpet:
[attachment=3327:attachment]http://www.hydrogenaudio.org/forums/index....showtopic=39334


(I miss the HA problem samples library!)

Cheers,
David.

Near-lossless / lossy FLAC

Reply #45
...
Exactly. What I was meaning (sorry, my English isn't very good) is that the "simplification" made by your tool can enable similar gain in compression ratio with other similar tools. For example (using one of the posted sample):

Code: [Select]
                      WAV     FLAC      TAK       RAR
01_41_30sec_lossy   5.168KB  1.957KB  2.119KB   2.755KB
02_41_30sec         5.168KB  3.473KB  3.284KB   3.633KB


Maybe two hours from now a new Lossless compressor will surface that enjoy an even higher gain than FLAC. In the end, I don't see the link between a generic tool that "simplify" a sound file, and a particular encoder (for witch the action of the tool may result especially beneficial).
...

For optimum performance the intenal frame size of the encoder has to be taken into account when using the preprocessor. I suppose you haven't done this for TAK?

Possibly i will have to add an option to manually set the frame size for TAK files to get most out of the preprocessor. While TAK partitions each of it's fixed size frames into up to 5 variable size sub frames to adapt to signal changes, the "wasted bits" options works on the whole frame, which is too big (more than 4000 samples) to work well with the preprocessor.

  Thomas

Near-lossless / lossy FLAC

Reply #46
So, how soon before an executable version of SoundSimplifier™ (  ) is released to an expectant HA community? Inquiring (impatient) minds want to know!


I like the name!

I'm not keeping it back. I've attached the latest MATLAB script which I'm using to generate these samples. It executes very well if you have MATLAB!
(Though you need lots of memory for normal sized audio files, since there's no buffering, and you'll need to change waveread to wavread and wavewrite to wavwrite).

As for an efficient C/C++ implementation which could be compiled - that's beyond me.

I think we need some more listening tests before anyone puts that much effort in, but the job is open to anyone who wants it!

Cheers,
David.

 

Near-lossless / lossy FLAC

Reply #47
i couldnt reliably abx the 1st set of samples, but there were some weird negative results on some like 1/8 or 2/8.
PANIC: CPU 1: Cache Error (unrecoverable - dcache data) Eframe = 0x90000000208cf3b8
NOTICE - cpu 0 didn't dump TLB, may be hung

Near-lossless / lossy FLAC

Reply #48

...
Exactly. What I was meaning (sorry, my English isn't very good) is that the "simplification" made by your tool can enable similar gain in compression ratio with other similar tools. For example (using one of the posted sample):

Code: [Select]
                      WAV     FLAC      TAK       RAR
01_41_30sec_lossy   5.168KB  1.957KB  2.119KB   2.755KB
02_41_30sec         5.168KB  3.473KB  3.284KB   3.633KB


Maybe two hours from now a new Lossless compressor will surface that enjoy an even higher gain than FLAC. In the end, I don't see the link between a generic tool that "simplify" a sound file, and a particular encoder (for witch the action of the tool may result especially beneficial).
...

For optimum performance the intenal frame size of the encoder has to be taken into account when using the preprocessor. I suppose you haven't done this for TAK?
...


I was curious...
I dowloaded "01_41_30sec_lossy" from here. Then i compressed it with TAK's default frame size and then with a frame size of 4096 Bytes. Results:

Code: [Select]
FLAC                2,004,157  Bytes
TAK Normal Default  2,023,188  Bytes
TAK Normal 4096     1,809,281  Bytes
TAK Turbo  4096     1,846,469  Bytes

Near-lossless / lossy FLAC

Reply #49
For optimum performance the intenal frame size of the encoder has to be taken into account when using the preprocessor. I suppose you haven't done this for TAK?

Right. I have just compressed the two files "on a rush" to check if - as I supposed - the action of the SoundSimplifier™  would be interesting also for other encoders.
And it was, so much, as your results show even more. Thanks for taking the time for the "optimized" encoding.

Bye!