Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Yalac - Comparisons (Read 214518 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Yalac - Comparisons

Reply #175 –
Great stuff.  It's exciting to hear that you've got most of the codec worked out.  Do you think that you're just about ready to work on the other features, like error correction, seeking and tagging?

Would it make any sense at this juncture to port it to C or something so that it could be made cross-platform compatible?  I thought perhaps it would be less work in the long run to do it now, rather than writing the whole program, then having to do it over again.

I'm glad that you're thinking about compatibility on other platforms, and hardware, because that's one of Monkey's Audio's shortcomings.

Yalac - Comparisons

Reply #176 –
Quote from: Supacon on
Great stuff.  It's exciting to hear that you've got most of the codec worked out.  Do you think that you're just about ready to work on the other features, like error correction, seeking and tagging?

Probably. But i just found some new optimization i want to evaluate first. It compresses some files more than 1 percent better, too much to be ignored.

Afterwards i will care for error correction and seeking. Then for tagging.

But it is quite probable, that the progress will slow down. Since my first post at April 1st i have spent nearly my whole time on YALAC! But soon i will have to go back to some work, i get paid for!

Quote from: Supacon on
Would it make any sense at this juncture to port it to C or something so that it could be made cross-platform compatible?  I thought perhaps it would be less work in the long run to do it now, rather than writing the whole program, then having to do it over again.

The allready very compact source of the encoder core (without any file handling or interface) is currently more than 700 KByte big! Very much work to translate it. Possibly i will first translate the far less complex decoder source to C.

Quote from: Supacon on
I'm glad that you're thinking about compatibility on other platforms, and hardware, because that's one of Monkey's Audio's shortcomings.

But it is really a long way to go... Currently i am only preparing Yalac for such later extensions (of it's possibilities).

Yalac - Comparisons

Reply #177 –
Current progress (V0.10)

Evaluation of compression efficiency and speed of the new Hardware profile presets of V0.10 and comparison to preset FAST of V0.09.

Performed on my primary test sets "rw" and "songs".

Test system: P3-800

All tests performed without file output.

Code: [Select]
         | Compression             | Encoding speed          | Decoding speed  |
         | Evaluation              | Evaluation              | Evaluation High |
Preset   | Fast    Normal  High    | Fast    Normal  High    | MMX     Pascal  |
---------+-------------------------+-------------------------+-----------------+
rw       |                         |                         |                 |
Level 0  |  58.34   58.11   58.06  |  53.32   35.54   19.30  |  74.80   66.51  |
Level 1  |  58.00   57.71   57.66  |  51.77   28.10   15.75  |  74.06   61.24  |
Level 2  |  57.75   57.43   57.36  |  43.07   23.11   12.96  |  71.72   53.32  |
Fast     |  57.30                  |  38.65                  |  69.51   50.31  |
---------+-------------------------+-------------------------+-----------------+
songs    |                         |                         |                 |
Level 0  |  49.70   49.49   49.45  |  56.59   35.86   19.54  |  75.84   67.64  |
Level 1  |  49.07   48.81   48.75  |  53.13   28.55   15.92  |  74.70   60.84  |
Level 2  |  48.68   48.38   48.29  |  44.39   23.63   13.11  |  69.38   52.14  |
Fast     |  48.51                  |  37.54                  |  67.42   48.88  |
---------+-------------------------+-------------------------+-----------------+

Preset

Hardware profiles Level0/1/2 with 8/16/32 predictors and frame durations 63/100/131 ms. Preset Fast taken from V0.09.

Evaluation

Fast/Normal/High determines the evaluation depth of the encoder. This should only affect encoding but not decoding speed.

Compression

Compression ratio in percent.

Encoding / Decoding speed

Multiple of real time. Because decoding speed should not depend on the Evaluation level, i only tested High. MMX is the speed of the optimized assembler code, Pascal the speed of pure Pascal code.

Short summary

Level 2 with Evaluation High is about on par with preset FAST although it's implementation is far less complex.

Yalac - Comparisons

Reply #178 –
I must admit that I am finding this new system quite confusing.  I'm not very familiar with OptimFROG but it is reminding me of its --mode and --optimize switches.

One suggestion?  Perhaps you could find the best balances in that matrix and allow users to use presets to access them?  E.g.:

YALACC.EXE --level 0 --evaluation fast c:\path\to\myfile.wav

... becomes:

YALACC.EXE --preset turbo c:\path\to\myfile.wav

I have just realised that this isn't a new idea - FLAC, LAME, etc. all do this already - and you were probably already planning to. ... ah, well; I'll leave it here anyway.

Edit:  Hang on, have I got the wrong end of the stick?
Quote from: TBeck on
Evaluation of compression efficiency and speed of the new Hardware profile presets of V0.10 and comparison to preset FAST of V0.09.
Sorry, does this mean that the above are additional to the existing presets, or replacements?
I'm on a horse.

Yalac - Comparisons

Reply #179 –
Quote from: Synthetic Soul on
Edit:  Hang on, have I got the wrong end of the stick?
Quote from: TBeck on
Evaluation of compression efficiency and speed of the new Hardware profile presets of V0.10 and comparison to preset FAST of V0.09.
Sorry, does this mean that the above are additional to the existing presets, or replacements?

As I understand it, they are different sets of presets. The normal presets are for playback on a computer, the new hardware profile presets are for playback on devices with limited hardware resources (low complexity).

The hardware presets can be played on a computer of course, only they may not be as optimal as the normal presets in terms of speed and compression ratio. So normally you would use the normal presets, and convert to a hardware preset when you want to play on a hardware device (with Yalac impressive speed that should not be that painful!)

Yalac - Comparisons

Reply #180 –
Thanks for your input jido.  On reflection I believe that you are right.

Hmm... I am still confused though.  I wouldn't want to have to decide on which of the nine combinations would fit my particular hardware (if I had some).  I assume things will become clearer to me as they become more relevant (i.e.: when there is hardware support, perhaps a combo will be issued a preset name, like --rockbox or something...).
I'm on a horse.

Yalac - Comparisons

Reply #181 –
Quote from: Synthetic Soul on
I must admit that I am finding this new system quite confusing.  I'm not very familiar with OptimFROG but it is reminding me of its --mode and --optimize switches.

One suggestion?  Perhaps you could find the best balances in that matrix and allow users to use presets to access them?  E.g.:

YALACC.EXE --level 0 --evaluation fast c:\path\to\myfile.wav

... becomes:

YALACC.EXE --preset turbo c:\path\to\myfile.wav

Jido is right:
Quote from: jido on
As I understand it, they are different sets of presets. The normal presets are for playback on a computer, the new hardware profile presets are for playback on devices with limited hardware resources (low complexity).

The hardware presets can be played on a computer of course, only they may not be as optimal as the normal presets in terms of speed and compression ratio. So normally you would use the normal presets, and convert to a hardware preset when you want to play on a hardware device (with Yalac impressive speed that should not be that painful!)

Thanks for making it clear! I should have written some more explaination for the new presets...

But:
Quote from: Synthetic Soul on
Hmm... I am still confused though.  I wouldn't want to have to decide on which of the nine combinations would fit my particular hardware (if I had some).  I assume things will become clearer to me as they become more relevant (i.e.: when there is hardware support, perhaps a combo will be issued a preset name, like --rockbox or something...).

Yes, there are 9 possible combinations: 3 Profiles * 3 Evaluation levels.

How to make your choice:

1) Select the maximum hardware profile your hardware can support.

If you don't know or want highest compatibility to other devices, you may select 0 or 1.

I like your idea to select the profile by specifying it's name. On the other hand, it would not be too difficult to look into a list, which contains the maximum possible level for each hardware. And you anyway may have to do this, even with your named switch option, because it often will not be clear, how exactly the name of your hardware has to be specified. What if there are different models: Rockbox A3-B and Rockbox A4-9?

2) Select the evaluation level based upon your time constraints for encoding.

The hardware compatibility of a profile does not depend on the evaluation level. Select a higher evaluation level, if you are willing to wait longer for a bit more compression.

Quote from: Synthetic Soul on
One suggestion?  Perhaps you could find the best balances in that matrix and allow users to use presets to access them?  E.g.:

You know, that we all together have done this for the old PC-Presets. There is now a very good balance between encoding speed and compression ratio. But here we never really cared about the effect on decoding speed and hence the hardware requirements!

But we have to to this for the hardware profiles. The most important factor determining the (minimum) decoding speed is the predictor order. Not surprising: this one is the main difference between our well balanced PC-Presets.

But with the hardware profiles we are restricted to a certain predictor order. Within a profile we can only vary those encoder options, which have no effect on the decoding speed, and this is, what the evaluation level does. And because of the constraints of a particular profile, there is far less variation possible.

If you find three evaluation levels too confusing: Just ignore them and use the default (Normal). It anyhow provides the best speed compression ratio. Fast is only there to make the good old TURBO-Preset vailable.

I would be happy with less than 3 profiles. I would like to remove Level 0. But there is some conversation with Josh Coalson, and he told me, that he had good reasons to restrict FLAC -8 to 12 predictors for maximum hardware compatibility. Currently it is not clear, if Yalac would achieve the same decoding speed with 8 or 16 predictors.

Sorry for the confusion! But all this (thinking about hardware support) is very new for me.

But it's good, that i have written about it. Your feedback shows me possible failures and shortcomings of my concept.

Thanks

  Thomas

Yalac - Comparisons

Reply #182 –
Hmm... haven't heard of any new developments lately.  Hopefully that doesn't mean that TBeck couldn't sustain this project.  I'd hate to see this be relegated to vaporware after getting so close!

Hopefully something can be released soon enough that will demonstrate YALAC's capabilities well, but be able to maintain future compatibility with upcoming versions and revisions.  I'm no expert on the subject, but I thought that implementing seeking, error correction, and tagging would be fairly easy because good existing algorithms can be used.

Yalac - Comparisons

Reply #183 –
Quote from: Supacon on
Hmm... haven't heard of any new developments lately.  Hopefully that doesn't mean that TBeck couldn't sustain this project.  I'd hate to see this be relegated to vaporware after getting so close!

Hopefully something can be released soon enough that will demonstrate YALAC's capabilities well, but be able to maintain future compatibility with upcoming versions and revisions.  I'm no expert on the subject, but I thought that implementing seeking, error correction, and tagging would be fairly easy because good existing algorithms can be used.

After the many years of work something really strange had to happen to prevent me from finishing and releasing it. But i allready wrote, that i will not always be able to spend most of my time (for the last 3 month i did hardly anything else...) for yalac, hence development will progress a bit slower.

And i want a robust and extensible public release. I don't want to have to change the file format after a public release. The design takes some time.

Some things seem to be easier than they really are. For instance it is easy to add some error recognition code to yalac to detect damaged frames. But this check is usually beeing performed, when the decoder allready has tried to decode the whole frame. Damaged data can create invalid states within the decoder, which have to be detected and handled. Because decoding speed is very important for me, i am currently trying to design the decoder for maximum error tolerance without the need to frequently check the input data for invalid states, which could take a significant amount of time.

I am still working hard on Yalac, but currently there is not much that would be worth to be posted.

  Thomas

Yalac - Comparisons

Reply #184 –
Current progress (V0.10)

Well, this time it will take more than my usual 2 weeks to get the next version done, because i have to make more fundamental deceisions, for instance about the file format. But now it's at least clear, what the next version will bring.

Done:

- Slightly better compression for preset EXTRA.
- 2 hardware profiles.
- Each frame contains a CRC32 Checksum for error recognition.
- Each frame starts with a sync code, to make streaming possible and data recovery of damaged files easier.
- Further preparations for later streaming support.
- More error tolerance within the decoder.

Preset Turbo will be a bit slower, because the checksum generation takes some time and because i have increased the frame size from 63 to 100 ms to compensate the space requirements of sync code and checksum.

To do:

My primary goal for V0.10 is far more error tolerance and a function to repair damaged files. I had to deal a bit earlier than i wanted with the streaming implementation, because some of the features needed for streaming are also important for the error resistance.

V0.10 will bring a new GUI function to manually repair damaged files. Later versions will support some automatic recovery while decoding. It's anyhow needed for streaming.

I would like the testers to damage files (be bad!) and then try the new repair function. Probably i will provide a little tool, that will randomly replace some bits of the files.

Unfortunately i can not predict, how long the implementation of the repair function  will take. It's very probable, that i will find, that many parts of the decoder have to be rewritten for more error tolerance, when i start the testing of damaged files. Maybe a week...


BTW: Thanks to Joseph Pohm and Synthetic Soul for updating Joseph Pohm's comparison.

Warning: A comparison performed on a limited file set is nearly never representative for the average performance of a codec! For instance this set is the only one so far, where Yalac High comes so close to OptimFrog High. These files are really something special!

  Thomas


Edit: Warning added.

Yalac - Comparisons

Reply #185 –
Quote
Each frame contains a CRC32 Checksum for error recognition

Does this make sense?

Yalac - Comparisons

Reply #186 –
Quote from: LaserSokrates on
Quote
Each frame contains a CRC32 Checksum for error recognition

Does this make sense?

Good question!

I wanted to ask the same in the Yalac-format-development-thread...

It's very possible, that the file header will contain a MD5 check sum for the whole files. This should allready provide some safety.

The frame crc's should be able to detect more errors and their location (I hope so). What is it worth?

If the errors are so minimal, that you wouldn't notice them, you will anyhow want to ignore the check sum errors.

If you can hear them, you don't need a check sum to verify this.

They may be useful, if you don't want to listen to the file to check for errors...

And it's a question of the error probability; if there are not many, one MD5 check sum should be sufficient.

To be honest, i myself could live witout frame checksums. But most other audio compressors are using them, and i would feel bad, if users would not trust yalac because it does not use them...

  Thomas

Yalac - Comparisons

Reply #187 –
A CRC for each frame is possible in LAME, but with a penalty of 16 bits per frame. I'd go with MD5 for the raw audio data like in FLAC and WavPack. In this case, you either know if a file is corrupt or not, but that's sufficient IMHO.

Edit: Why do you "expect" data loss? I never had any problem with data corruption with my lossless files.

Yalac - Comparisons

Reply #188 –
Or, you could implement a CRC-16 code, maybe, in each frame.  I don't think the system needs to be _that_ sturdy...
 

Yalac - Comparisons

Reply #189 –
Quote from: TBeck on
To be honest, i myself could live witout frame checksums. But most other audio compressors are using them, and i would feel bad, if users would not trust yalac because it does not use them...
The advantage of per-frame error checking is that you can mute the audio when you detect an error, thereby avoiding loud bursts of noise. Also, if you are recovering a corrupt file it's nice to have the option of throwing away (or muting) the bad blocks. If you somehow lost the global MD5 you would have no idea whether what you had was mostly garbage or perfectly fine.

Yalac - Comparisons

Reply #190 –
How much would this actually impact the compression ratio?

I'm not really sure in what context you'd get so many bad frames... usenet posting of lossless image files, maybe?

Yalac - Comparisons

Reply #191 –
Quote from: bryant on
The advantage of per-frame error checking is that you can mute the audio when you detect an error, thereby avoiding loud bursts of noise. Also, if you are recovering a corrupt file it's nice to have the option of throwing away (or muting) the bad blocks. If you somehow lost the global MD5 you would have no idea whether what you had was mostly garbage or perfectly fine.

Thanks!

Yalac - Comparisons

Reply #192 –
Current progress (V0.10)

And again a report... The main reason for the increased report frequency: I am now working on features which are more important for the practical usability for average users than some specific encoder options. I want to make sure, that i don't miss necessary features.

Decompression and error recovery

Now you can deceide, if you want to decompress files or only test their integrity.

Errors can be corrected on the fly while decompressing.

You can choose, how damaged files should be handled:

1) Skip: Don't decompress them.

2) Repair: Make a new file from the valid frames.

You can choose, how damaged frames within each file should be handled:

1) Skip: The damaged data is beeing cut out.

2) Mute: Replace the damaged data with a frame of silence. This can be useful, if you want to perform manual corrections with an audio editor on the recovered file.

For each damaged file an error log is beeing created:

Code: [Select]
--- 41_30sec_err.yaa ---

Result: Frames damaged

Header frames:       477
Valid  frames:       474
Skipped blocks:        3
Skipped end:           0
Crc errors:            1

Skipped data blocks

No       Source pos  Size        Sample pos  Count
       1      979895       12617      394476        2778
       2     1445939       12924      575046        2778
       3     2637911       14806     1033416        2778

It contains:

- The frame count the file header reported.
- The count of valid frames.
- The count of damaged data blocks which have been skipped.
- Skipped end is 1, if there was a damaged data block after the last recovered frame.
- The count of crc errors. Most errors may be detected by the decoder, but some will go through and than be detected by the checksum.

Skipped data blocks contains a list of the defective data blocks:

- The position and size of the block within the compressed source file in bytes.
- The position and size of the block within the decompressed file in samples. With this info it is easy to find the affected file parts within an audio editor.

Yalac - Comparisons

Reply #193 –
Looks great.  In a software decoder implementation, ideal fuction would be play and silence bad frames.  It's nice to check for errors too, though, with a skip function -- if compressing a CD, for example.

Yalac - Comparisons

Reply #194 –
Quote
' date='Jul 5 2006, 05:01' post='408985']
Looks great.  In a software decoder implementation, ideal fuction would be play and silence bad frames.  It's nice to check for errors too, though, with a skip function -- if compressing a CD, for example.

Thanks.

But i forgot to implement two other useful options to handle corrupted frames:

3) Insert a frame of silence, but let it perform a fadeout (go down to 0) for the previous frame and a fadein for the next frame to avoid clicks.

4) Insert no frame of silence (skip), but instead perform a fadeout for the previous frame and a fadein for the next frame within their data to avoid clicks.

Yalac - Comparisons

Reply #195 –
Are these things really functions of the codec?  I thought it would be the decoder, or even the player that is responsible for behavior like this.

Yalac - Comparisons

Reply #196 –
Quote from: Supacon on
Are these things really functions of the codec?  I thought it would be the decoder, or even the player that is responsible for behavior like this.

Surely not.

The interface to the codec is simple: Here are n bytes of data representing one frame of audio data of format x. Decompress it. The codec knows nothing about sync codes, crc, streaming, metadata, files etc.

But woudn't users like to compress and decompress files? 

I am not sure, if i have understood your question...

Yalac - Comparisons

Reply #197 –
Don't you think you are paying a bit too much work into error correction? I think I never had a file with corrupt frames, and the only scenario I could imagine would be an audio stream over WLAN, not the best way to transport lossless files to be saved on another computer.

Yalac - Comparisons

Reply #198 –
Quote from: LaserSokrates on
Don't you think you are paying a bit too much work into error correction? I think I never had a file with corrupt frames, and the only scenario I could imagine would be an audio stream over WLAN, not the best way to transport lossless files to be saved on another computer.

It's possibly a matter of personal taste. Some people are being eased by error recognition and recovery. That's one reason for me. Others:

- I have read many of the older threads of this forum and found many posts regarding verifying and error correction.
- I anyhow needed sync codes and checksums for the preparation of streaming.
- I knew, that my decoder could crash when encountering data errors. I had to test and fix it. Not to much additional work to implement the whole thing.
- The comparioson table in the wiki contains a field for error recovery. I wouldn't like it to be red, if yalac should be drawn up (right?).
- Some trauma from the release of the first evaluation version: one tester had nothing better to do than damaging the files and reporting errors and crashes...
- I never before had used checksums and wanted to learn a bit more about them.

Yalac - Comparisons

Reply #199 –
Quote from: LaserSokrates on
Don't you think you are paying a bit too much work into error correction? I think I never had a file with corrupt frames, and the only scenario I could imagine would be an audio stream over WLAN, not the best way to transport lossless files to be saved on another computer.


I store my lossless "archive" on CDs. I use Monkey's Audio for this, because it compresses good enough so I can always find two albums together to put on one CD. However, when I can I put the files in a rar archive with recovery information to fill up the rest of the available space on the CD, because I would hate to lose my lossless copied due to holes in the CD. If this protection can be provided by the audio codec itself, I'd consider this a point to chose that format.