HydrogenAudio

Lossless Audio Compression => WavPack => Topic started by: aabxx on 2003-11-09 19:31:34

Title: why isn't wavpack more popular?
Post by: aabxx on 2003-11-09 19:31:34
If you look at the FLAC comparison page, http://flac.sourceforge.net/comparison.html (http://flac.sourceforge.net/comparison.html), you'll see that wavpack normal compresses a few MB better than FLAC maximum, encodes in half the time, and decodes not much slower...

Wavpack is also free and open-source, just like FLAC... and it has an excellent feature that FLAC lacks: hybrid mode ("lossy" + correction file).

FLAC's advantages are that it is streamable, is more well-known (under the xiph umbrella now as well) and has better support (on a few hardware players as well).

Now there isn't perhaps that much between them, but surely wavpack deserves more recognition than it gets?
Title: why isn't wavpack more popular?
Post by: atici on 2003-11-09 19:48:27
I don't think the difference is significant. FLAC is more established as a standard as well as Monkey's Audio. Optimfrog and WavPack is cool with hybrid mode. But if one is planning to store the loss file somewhere he can store it in lossless instead which makes more sense to me because the loss file is almost as large as the lossless encode in another format and is useless without the lossy part.

I think the two most important usability points that would make me adopt one lossless codec over another is Monkey's Audio's APL feature (bookmarking) and ReplayGaining which is not supported by either Monkey's Audio or FLAC.

If WavPack does these than I am in...
Title: why isn't wavpack more popular?
Post by: dev0 on 2003-11-09 20:10:39
Both FLAC and MonkeysAudio support ReplayGain.
WavPack does too, if you use APEv2 tags and a player, which supports ReplayGain.
Title: why isn't wavpack more popular?
Post by: rjamorim on 2003-11-09 20:24:54
IMO, teh biggest problems with WavPack are seeking speed, error robustness and multichannel capabilities.

Besides that (and all those issues are being addressed in WavPack 4), it has potential to smoke FLAC in almost every aspect: encoding efficiency (speed vs. compression), features (lossy and hybrid encoding), decent library API... :B
Title: why isn't wavpack more popular?
Post by: ChristianHJW on 2003-11-10 13:07:09
We are looking forward to Wavpack4 a lot .... hopefully David will write the lib such that en/decoder and the internal framing are separated from each other, as this will make the implementation job a lot easier.

I love the idea to be able to go from lossless to lossy mode and back in one and the same track, this sounds pretty interesting indeed. A superb feature was to make the encoding mode depnding from the actual spare CPU power, that way Wavpack could become the nr. 1 codec for analog capturing. Depending on the CPU power necessary for the encoding of the video, the codec could switch to lossless or lossy mode, to avoid frame drops.

Unfortunately, none of the emails sent to me from David ( and then forwarded to our lists ), asking about what we'd expect for best usability, were answered by our lazy devs  ..... grmbl .... /me goes waking them up now ....
Title: why isn't wavpack more popular?
Post by: The Belgain on 2003-11-10 22:31:14
Just out of interest, is there any news on when to expect WavPack4? IIRC the first alpha versions were supposed to be coming out about now?

I currently have most of my collection in FLAC format, and am thinking about going over to Wavpack once it has instant seeking...
Title: why isn't wavpack more popular?
Post by: tcmjr on 2003-11-10 22:58:32
E ai Roberto , tudo bom ?
hehehe

Well , I see you talking a lot about wavpack v4 , I could not found any info about it at wavpack homepage.
I am starting to rip my collection to lossless and would like to try wavpack , but I am waiting for this so talked wavpack 4.
Any guess when its coming ?
Title: why isn't wavpack more popular?
Post by: rjamorim on 2003-11-10 23:22:19
Quote
E ai Roberto , tudo bom ?
hehehe

OMG! Another brazilian.

Quote
Well , I see you talking a lot about wavpack v4 , I could not found any info about it at wavpack homepage.
I am starting to rip my collection to lossless and would like to try wavpack , but I am waiting for this so talked wavpack 4.
Any guess when its coming ?


Well, I am probably partially to blame for the delay. David says that I always call him at his programming time, and since our phone calls last one hour at least, he wastes too much time talking to me

So now I'm calling less...

Anyway, he told me that, feature-wise, he plans to launch an alpha that will almost be a beta - I.E, almost all fundamental features are already implemented, and later improvements wouldn't break backwards compatibility.

And that's the reason it's taking so long for an alpha (besides my calls, of course)

Hopefully he'll have it before christmas, but I really have no idea.

Regards;

Roberto.
Title: why isn't wavpack more popular?
Post by: danchr on 2003-11-10 23:51:09
One thing FLAC has that neither Monkey's Audio nor WavPack seem to have is portability. None of them seem to be written with portability in mind like FLAC obviously was. I once tried to compile Monkey's Audio on my mac; I wouldn't say it went smoothly...
Title: why isn't wavpack more popular?
Post by: rjamorim on 2003-11-11 00:35:26
Quote
One thing FLAC has that neither Monkey's Audio nor WavPack seem to have is portability. None of them seem to be written with portability in mind like FLAC obviously was. I once tried to compile Monkey's Audio on my mac; I wouldn't say it went smoothly...

WavPack is 100% ANSI C! (wavpack 3 at least, not sure about 4)
So, porting it to other systems and platforms should be quite easy.

Monkey's , on the other hand, is indeed not very portable, since it relies heavily on x86 ASM.
Title: why isn't wavpack more popular?
Post by: zokik on 2003-11-11 02:02:04
Quote
Quote
One thing FLAC has that neither Monkey's Audio nor WavPack seem to have is portability. None of them seem to be written with portability in mind like FLAC obviously was. I once tried to compile Monkey's Audio on my mac; I wouldn't say it went smoothly...

WavPack is 100% ANSI C! (wavpack 3 at least, not sure about 4)
So, porting it to other systems and platforms should be quite easy.

Does "#include <windows.h>" comply with ANSI C?

This software probably works on all windows systems, but that's it. What about gnu/linux, mac os, or even standalone hw players?

Quote
So, porting it to other systems and platforms should be quite easy.

If it is easy, why isn't there an xmms plugin or something similar?

One of the major disadvantages of wavpack is lack of proper portability.
Title: why isn't wavpack more popular?
Post by: rjamorim on 2003-11-11 03:19:55
Quote
Does "#include <windows.h>" comply with ANSI C?

Argh. I don't deserve that shit.

I'm talking about the encoding routines here, smarty guy. Not the frontend.

Quote
This software probably works on all windows systems, but that's it. What about gnu/linux, mac os, or even standalone hw players?


The sources have been released few time ago. And nobody is interested in working on properly porting 3.x, as well as creating plugins and what not, since 4.0 is around the corner.

Quote
One of the major disadvantages of wavpack is lack of proper portability.


OK, define "proper portability".

Besides, I suggest you actually study the sources before running around making wild claims.

Also, a hint, since you obviously doesn't seem to know a shit of what I'm talking about: check the pack.c and unpack.c sources. There's where compression and decompression actually occurs.

Also, please learn about the usage of #ifdef / #ifndef

Kind regards;

Roberto.
Title: why isn't wavpack more popular?
Post by: den on 2003-11-11 08:39:56
Funnily enough every time I ask David about Wavpack 4 he blames Roberto for his lateness also...

;-)
Title: why isn't wavpack more popular?
Post by: rjamorim on 2003-11-11 11:45:18
Quote
Funnily enough every time I ask David about Wavpack 4 he blames Roberto for his lateness also...

Shame on me
Title: why isn't wavpack more popular?
Post by: zokik on 2003-11-11 13:42:37
Quote
OK, define "proper portability".

ok, let's say a user would be able to use the software under other os/platform without having to do any significant work or have any knowledge & experience with programming. e.g. you just run an install program or script, or some usual routine, like ./configure, make, make install.

Quote
Besides, I suggest you actually study the sources before running around making wild claims.

ok, I really did just a very qiuck glance at the source, you're right about that. About windows.h... I just checked for files with that string, because that is quite common header people use. I don't have time or interest to do an extensive (re)search. If I can't get something working simply and quickly enough, I'll just look for other software that does the same.

I just made a glance at unpack.c and pack.c you mentioned. The first line is
#include "wavpack.h"
wavpack.h uses __fastcall (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vccore98/HTML/_core___fastcall.asp) which I can't compile with gcc (I wonder why  )

I am not some wavpack enemy, I just didn't find any way to get it all working outside windows (that includes also a player plugin, not just encoder and decoder). If wp4 will be fully supported under my os of choice, I will surely give it a try.

And in future please be more polite. You don't need to be rude just because someone disagrees with you.
Title: why isn't wavpack more popular?
Post by: seanyseansean on 2003-11-11 14:47:56
Quote
I don't have time or interest to do an extensive (re)search.

... in which case don't make remarks about things you have no interest in finding out the facts about.
Title: why isn't wavpack more popular?
Post by: joeg on 2003-11-11 17:04:58
flac is supported by hardware players such as the kenwood keg (Car audio) which i'm hopefully buying this week...

i have quite a bit of my collection in flac, so i can't wait to see how it goes...
Title: why isn't wavpack more popular?
Post by: bryant on 2003-11-11 18:35:09
Before everyone gets their panties in a bunch, let me make a few comments about portability. 

WavPack was written to be "easily" ported to other platforms. If you don't define the macro __WIN32__ you don't get "windows.h" and it won't use the Windows versions of most system calls. This is how the real DOS mode version is compiled because when DOS was invented Microsoft had not yet learned the value of just being different for different's sake. 

However, one of the things I wanted for WavPack was to have an intuitive interface (for a command-line program, that is) because during development I have to use it a lot. So, it will accept wildcards in filenames; it displays the progress on the console title bar; it displays the elapsed time when completed; it warns about overwriting and asks for confirmation; when you ^C during a conversion it deletes the partially created file before exiting; you can preserve the input file's time in the output; etc. There is also quite a bit of ugly code in there whose only purpose is to make sure that WavPack runs as fast as possible on various flavors of  Windows. The problem with these kind of features is that there is no portable way of implementing them, so all the work has to be (at least partially) duplicated for each platform (and it also helps to be familiar with the platform before starting).

When I released the source code for WavPack I supposed that some Linux programmer would jump on this and we'd have a Linux port in a few months (I knew that the big-endian/little-endian thing would make a MAC port less likely). Well, at least 3 people have created Linux port "hacks" for their own use by simply taking out all the bells and whistles that don't compile (and this takes just a few hours), but to date there has not been an effort to do a "real" port (which would probably take a few days). And now, of course, anyone who wants to do this should wait until WavPack 4.0 is ready.

As for WavPack 4.0 status, it is in alpha test now and I believe that we are still on schedule for release in January, although multiplatform (and hardware) support will certainly be later. The code for WavPack 4.0 will be designed for even easier portability, but if a Linux version doesn't appear I will probably put together a Linux system and do it myself (because I want to have a Linux system anyway). But I am not buying a MAC... 

Thanks for the support, guys!
Title: why isn't wavpack more popular?
Post by: torok on 2003-11-11 21:45:05
Baleeted!!!!
Title: why isn't wavpack more popular?
Post by: rjamorim on 2003-11-11 22:31:00
Quote
ok, let's say a user would be able to use the software under other os/platform without having to do any significant work or have any knowledge & experience with programming. e.g. you just run an install program or script, or some usual routine, like ./configure, make, make install.

OK, please let me try to make myself clear this time.

I said that wavpack is portable, not that it has already been ported.

Also, I said that "porting it to other systems and platforms should be quite easy". I never said it has already been easy.

But, comparing to other formats that never had porting in mind (like Monkey's Audio), that heavily rely on the x86 architecture or something like that, porting it should be quite fast.

Besides, I agree with seanyseansean's post. Why are you criticizing it if you didn't even had teh time to research properly first?

Regards;

Roberto.
Title: why isn't wavpack more popular?
Post by: Peter on 2003-11-12 17:33:07
FLAC API discussion moved here (http://www.hydrogenaudio.org/show.php/showtopic/15198).
Title: why isn't wavpack more popular?
Post by: glauco on 2003-11-12 18:08:20
I'm interested in having a Wavpack Linux version (at least a x86 version). I'm not a really great coder or programmer, but i work as one  , and i have ported a couple of programs between Linux and Windows.

I'm sure Wavpack is not so difficult to port. I'll try to do a complete (how risky sounds that...) Wavpack Linux version as soon as some alpha code gets released. Well, as soon as I have some free time, and that means probably Christmas.

Looks like it's time to PM bryant. 
Title: why isn't wavpack more popular?
Post by: rjamorim on 2003-11-12 18:42:24
Quote
I'm sure Wavpack is not so difficult to port. I'll try to do a complete (how risky sounds that...) Wavpack Linux version as soon as some alpha code gets released. Well, as soon as I have some free time, and that means probably Christmas.

Great, thanks for your interest

BTW: if you want to contact David by e-mail, you can find his address at http://www.wavpack.com/ (http://www.wavpack.com/)
Title: why isn't wavpack more popular?
Post by: glauco on 2003-11-12 18:53:00
I will.

Thanks.
Title: why isn't wavpack more popular?
Post by: Daybreak on 2003-11-12 19:29:50
I think basically WavPack needs more advertising of some sort.. I mean, you can have the greatest product in the world, but if no one uses it, who's going to know?

Perhaps a Slashdot frontpage announcement or something of the like would do just fine when v4 comes out
Title: why isn't wavpack more popular?
Post by: cabbagerat on 2003-11-12 19:32:49
Quote
I'm sure Wavpack is not so difficult to port. I'll try to do a complete (how risky sounds that...) Wavpack Linux version as soon as some alpha code gets released. Well, as soon as I have some free time, and that means probably Christmas.


glauco, I would be very interested in helping you with this endevour. Wavpack is a nice tool and I have wanted a Linux version of it for a while. I am fairly experienced in porting stuff (normally the other way - from Linux to Windows).

It would be nice, however, if whatever changes were made to the source made it properly portable (rather than just a Linux version) and were merged with the main tree. This is obviously the call of the wavpack developers though. Even if it doesn't go back in to the main tree, this would be good for wavpack. There are few better ways to find bugs than by porting a piece of code to a different compiler.

I would also be willing to write a simple GTK or KDE (QT) based frontend to replace the windows one in the Linux port.

glauco, contact me at mbrooker at smuts uct ac za if you are interested in getting some help with this project.
Title: why isn't wavpack more popular?
Post by: rjamorim on 2003-11-12 19:45:37
I have created a sourceforge account some months ago - nothing very interesting there so far, just some 3.x sources and binaries - and I think that would be the ideal environment to coordinate the porting efforts.
Title: why isn't wavpack more popular?
Post by: den on 2003-11-13 06:30:46
I was also considering doing a Linux port, but seeing as I've only just returned to Linux after a break of three years, and it's about the same time since I last looked at a C program, I think I will leave to you guys. You sound like you know what you are doing. I'll concentrate my time helping in other ways, like listening tests.

I would however suggest that you hang back from bugging David just yet, unless you want to swing Wavpack 3.97/3.98 across to Linux, as I'm guessing that Wavpack4 will remain in alpha for a little while yet. 

It would make sense to ensure everything is sweet before you spend too much time on it.

B)
Title: why isn't wavpack more popular?
Post by: rjamorim on 2003-11-13 15:00:17
Quote
I'm guessing that Wavpack4 will remain in alpha for a little while yet. 

As I understood, all that is left to implement to freeze the stream specs is the lossy VBR mode and multichannel support.

Did David tell you something different? (since I expect you also got an alpha  )

quoting from the readme:

Quote
                                4. What's Next
                               --------------

o create VBR "quality-based" version of hybrid compression
o add multichannel support and finalize format specification (with beta)
Title: why isn't wavpack more popular?
Post by: rjamorim on 2003-11-13 18:33:43
OK, so as some of you already noticed by my previous post, WavPack4 alpha is already out.

David didn't make this release public yet. His intention is to make it public only once he has a player plugin working, so that users can see for themselves that yes, seeking is now much faster.

Also, he didn't implement yet multichannel coding. So, the only new feature left to be tested in this version is error robustness.  And so I did.

The procedure:

-Encode a song in WavPack4
-Open it in a Hex editor (in my case, WinHex) and delete some bytes, replace others.
-Try to decode with wvunpack

wvunpack decoded OK until the part where I deleted some bytes. The part where I changed some bytes decoded OK, although there was an awful sound coming from my speakers.  Maybe it would be interesting if wvunpack output a message saying that sample #X is corrupt.

When it reached the part where I deleted some bytes, it output the error message "incorrect number of samples!" and exit. Maybe it would be possible to add a -f (orce) switch that would skip the wrong block(s) and resync?


About compression*:

From my small test (one sample, 46.524Kb), wavpack4 high seems quite faster than3, but compresses a little less. That's probably because of the new block-based bitstream (block header overhead, etc.)

WavPack4a - high - 23723Kb - 14.39s
WavPack3  - high - 23624Kb - 18.30s
WavPack4a - normal - 24171Kb - 8.30s
WavPack3  - normal - 24160Kb - 8.07s

WavPack4 features a new improved compression method that does exhaustive compression on each block searching for the best ratio. As I understand it, it's like pngcrush -brute. Of course, encoding is much slower in this mode, but decoding as about just as fast. It is probably useful if you need to compress just those extra few bits to make 2 albums fit one CDr.

I tested the default compression method with -x5 (the assymetrical methods go from x1 to x6)

WavPack4a - default x5 - 23.889Kb - 225.37s

I ran some tests with Flac and Monkey's Audio for comparision.

Flac 1.1.0 - 8 - 24140Kb - 58s
Flac 1.1.0 - 5 - 24198Kb - 12s
Monkey's Audio 3.97 - high - 23390Kb - 12s
Monkey's Audio 3.97 - normal - 23483Kb - 10.2s


Also, according to the readme, the lossy mode got improved, the new minimum bitrate is 192kbps and reasonable quality is obtained with 256kbps. Also, VBR is going to be implemented.

There are two forms of operation in the hybrid mode now: focus on quality (at expense of lossless overall size), or focus on compression (at expense of lossy file quality)

Well, that's about it regarding what I tested and noticed so far.

All in all, things are progressing well and there are some new features that make WavPack stand out from the competition. Looking forward to new alphas/betas...

Best regards;

Roberto.


*AMD XP 1500+, 384Mb RAM, Windows 2000.
Title: why isn't wavpack more popular?
Post by: glauco on 2003-11-13 18:35:51
Quote
cabbagerat : glauco, I would be very interested in helping you with this endevour


Having help is always better than doing something alone. Thank a lot. 

Quote
It would be nice, however, if whatever changes were made to the source made it properly portable (rather than just a Linux version) and were merged with the main tree


Sure. That was also my idea. It shouldn't be so difficult to do.

Quote
rjamorim :
I have created a sourceforge account some months ago


That would be a great way to coordinate the efforts. Even if I don't think the porting project's going to be big enough to need it, sure it will improve it's popularity.

Quote
den: I was also considering doing a Linux port, but seeing as I've only just returned to Linux after a break of three years, and it's about the same time since I last looked at a C program, I think I will leave to you guys. You sound like you know what you are doing. I'll concentrate my time helping in other ways, like listening tests.


Well, any help will surely be appreciated. Even if you have forgotten your C skills, I'm sure your ears are willing to test the new release.

By the way, you can contact me at gonzalomonte AT eresmas DOT net
Title: why isn't wavpack more popular?
Post by: glauco on 2003-11-13 18:38:09
Quote
WavPack4 alpha is already out.


Coooooooool.  B)
Title: why isn't wavpack more popular?
Post by: bryant on 2003-11-13 23:53:48
Roberto:

Thanks for trying out the alpha!

On the error robustness issue, the format is resistant to errors, but the alpha version of Wvunpack cannot yet take advantage of it. It currently expects everything to be in place or it fails. Needless to say, it's on my todo list, but is less important than just getting the spec complete.

It looks to me like your test of the -X switch was actually on the default mode rather than the "high" mode. First of all, -X5 in "high" mode would have taken a lot longer than that (I recommend -X3 in high, -X4 in default), and would not have resulted in worse compression than "high" without -X. What I see there is what I've seen myself: that the -X can make the default mode compress almost as well as "high" with very little (if any) affect on the decode speed. (After I wrote this I see you already caught and changed that...)

Thanks again.
Title: why isn't wavpack more popular?
Post by: den on 2003-11-14 00:39:22
@Roberto

I was trying to subtly drop hints without giving away that I have an alpha... 

But now that it's out in the open...

My focus (as you can imagine form my previous Wavpack threads) is on the hybrid and lossy side. My early impressions are that the 256kbit bitrate is very nice indeed, especially compared to Wavpack 3.98 in this bitrate range (or as close to it as it can get).

I'm in the throws of setting up a battery of listening tests once I settle on what quallity settings, bitrates etc I'm interested in. (After I finally get around to completing a certain trancoding test that is still outstanding...)

 

I'm still getting my head around the new extra compression assymetiric encoding options, and whether they deliver sufficient extra bang for the extra wait, but from a placebo point of view, it certainlly feels a lot better. 

This new encode of Blue Monday is taking longer than it takes to play in realtime on my PC, it has to sound better!!!
Title: why isn't wavpack more popular?
Post by: bryant on 2003-11-29 06:07:40
For anyone interested in playing around with the alpha release of WavPack 4.0, I have now got a winamp2 plugin working (with fast seeking). I also created a document giving a technical overview of the new WavPack format. The whole package is here:

http://www.wavpack.com/alpha1.zip (http://www.wavpack.com/alpha1.zip)

Note that this is very early and for testing purposes only, so don't archive with it. Also, the wvunpack program and the winamp plugin will not deal with regular WavPack files.

Any testing results, comments or suggestions are very much appreciated.

Happy Thanksgiving to the locals!
Title: why isn't wavpack more popular?
Post by: ChristianHJW on 2003-11-29 09:04:55
Quote
I have created a sourceforge account some months ago

Please allow me to invite you and David to move the project to corecodec.org instead. We are a small family, but we support each other best we can.

Also, on sourceforge wavpack is one project amongst many others, on corecodec its something special. A dedicated webforum, an IRC server and much support from the organization side are additional benefits. Please think about it before going opensource with Wavpack4 ....
Title: why isn't wavpack more popular?
Post by: B on 2003-11-29 15:22:14
Thanks for sharing Bryant!

I did some testing, just to see how the filesize/encoding time compared to each other. Encoding done with a p4 @ 2ghz, 512mb ram. first in the list is the encoding option, then filesize and then encoding time.

Test-track:
The Police - Walking on the moon  52,025 kb (just picked some random track)

No options | 33,478 kb (10.19)

-h | 32,598 kb (13.08)

-x1 | 33,284 kb (16.09)
-x2 | 33,240 kb (29.86)
-x3 | 33,196 kb (49.55)
-x4 | 33,064 kb (106.53)
-x5 | 33,052 kb (158,48)
-x6 | 33,001 kb (326,17)

-fx1 | 34,843 kb (13.89)
-fx2 | 34,821 kb (17.67)
-fx3 | 34,821 kb (18.36)
-fx4 | 34,112 kb (35.14)
-fx5 | 33,982 kb (46.02)
-fx6 | 33,924 kb (55.56)

-ffx1 | 35.010 kb (12.48)
-ffx2 | 34,984 kb (14.59)
-ffx3 | 34,984 kb (14.79)
-ffx4 | 34,699 kb (20.70)
-ffx5 | 34,699 kb (20.78)
-ffx6 | 34,699 kb (20.81)

-hx1 | 32,601 kb (31.55)
-hx2 | 32,585 kb (56.66)
-hx3 | 32,516 kb (230.17)
-hx4 | 32,492 kb (433.58)
-hx5 | 32,492 kb (596.64)
-hx6 | 32,480 kb (1276.63)

All decoding was somewhere between 5 and 10 seconds per track.
Verifying was somewhere between 3 and 4 seconds per track.

I'm on a old and slow hd for the moment, because my main hd crashed on me, so decoding time depended too much on hd writing speed. I could almost hear it choke on writing sometimes, thats why i didn't include the exact decoding times (too flawed IMO).

Seeking is instant with all tracks (nice!).

Below are the results on the same track with monkey's audio v3.97.

Just did these to have some comparison because i use monkey's right now for personal use.

-c1000 | 33,056 kb (7.1)
-c2000 | 32,531 kb (10.5)
-c3000 | 32,364 kb (15.1)
-c4000 | 32,034 kb (25.5)
Title: why isn't wavpack more popular?
Post by: danchr on 2003-11-29 16:56:59
Quote
WavPack is 100% ANSI C! (wavpack 3 at least, not sure about 4)
So, porting it to other systems and platforms should be quite easy.

I just a peek at code, and I wouldn't say it's ANSI C. Most people mean C89 when they talk about ANSI C, and something as basic as the // comment syntax is invalid in C89. It is, however, a part of C99. Granted, most of the code compiles, but functions like read(), write(), close() (all used in bits.c), tell() & lseek() (used in wavpack.c) aren't part of ANSI C. __fastcall and O_BINARY are Windows-specific and should be defined to nothing and 0, respectively. I guess it wouldn't be difficult to port the code, but that doesn't make it "100% ANSI C".

Sorry for the nitpicking (and the delay in the reply), I just couldn't resist
Title: why isn't wavpack more popular?
Post by: bryant on 2003-11-29 21:49:20
ChristianHJW:
Thanks, Christian! When the time comes we can discuss this. Roberto has done a pretty good job of selling me on SourceForge (those compile farms sound handy!) but if WavPack is going to wind up in Matroska in some form then corecodec makes sense too.

Benny.X:
Wow, thanks for your patient testing! (Of course, your computer is almost an order of magnitude faster than mine; I can't even think about -hx6!).

Your results pretty much match what I expect, and back up what I recommend in the readme. It's nice to see that the -X switch is able to get WavPack's "high" mode to beat Monkey's "normal" mode in compression (if not encode speed, obviously).

The poor encode and decode speed results that you got (without -X) are probably a result of some missing optimizations as much as your slower hard drive. When I first tested the old WavPack on Win2K I was amazed to see it run much slower than it ran on Win95! I finally figured out that the different flavors of Windows require all sorts of different "tweeks" for disk I/O management to perform optimally, and that in disk intensive applications (like WavPack's faster modes) these tweeks can make a 2:1 (or greater) difference in performance. In fact, on the XP system I tested this alpha version on, the "fast" mode was actually slower than the default mode! I plan on putting the Win2K+XP optimzations in there one of these days (probably the same time as Roberto's error resistance) and this should make 4.0 match the speed of 3.97 in the default and "high" modes because the overall complexity of the algorithms has not changed significantly. However, the optimization for Win95/98 involved multithreading and was so ugly that it will never be seen again...

Thanks again!

danchr:
Okay, okay, it's not 100% ANSI C! 

But read (), write (), open () and close () were described in the first K&R book and are pretty much universally available for non-stream I/O, right!?
Title: why isn't wavpack more popular?
Post by: Skymmer on 2003-12-02 01:37:43
Damn ! Why I always miss the best goodies ... 
Testing ... Testing ...
Title: why isn't wavpack more popular?
Post by: smok3 on 2003-12-02 14:00:02
so why isnt optimfrog more popular? 
Title: why isn't wavpack more popular?
Post by: PoisonDan on 2003-12-02 15:39:52
Quote
so why isnt optimfrog more popular? 

Well, it should be. 
Title: why isn't wavpack more popular?
Post by: Dologan on 2003-12-16 06:29:02
Any news on the development of Wavpack 4 in these past two weeks? I am eagerly awaiting a moderately-safe beta of it to try it out on some albums I want to fit into a CDR at high lossy bitrates.  Any gross estimation of when such a beta might be available?
OK, two weeks is probably too short a time to have any interesting news, so I am just checking whether David hasn't been squashed by a piece of masonry yet.

EDIT: Is there any chance we will see any improvement at lossy mode's decoding speed with version 4? A fast decoding speed would be important for transcoding purposes, and a quick test revealed that Wavpack lossy (@500 kbps, -h) decodes around 1/4-1/3 as fast as FLAC...
Title: why isn't wavpack more popular?
Post by: rjamorim on 2003-12-16 07:26:45
Quote
OK, two weeks is probably too short a time to have any interesting news, so I am just checking whether David hasn't been squashed by a piece of masonry yet.

He was still alive some days ago when I phoned him


IIRC, he's now working on psychoacoustic for implementation of the lossy VBR mode. Hopefully he'll have another alpha with this feature as christmas gift for you guys
Title: why isn't wavpack more popular?
Post by: danchr on 2003-12-16 08:57:08
Quote
danchr:
Okay, okay, it's not 100% ANSI C! 

But read (), write (), open () and close () were described in the first K&R book and are pretty much universally available for non-stream I/O, right!?

Nope, they are POSIX calls, and as such not related to any of the C standards. fread(), fwrite(), fopen() and fclose() are the ones mentioned in second edition of K&R. I don't know about the first edition of K&R, though, I never saw it

What I have seen of K&R C tells me that it, if possible, should be avoided.

(I seem to be particularly predisposed for missing posts in this thread...)
Title: why isn't wavpack more popular?
Post by: bryant on 2004-01-04 08:02:45
I have posted a new WavPack 4.0 alpha that includes my first cut at a "quality" mode using psycho-acoustic analysis. The package includes a readme with all the details:

http://www.wavpack.com/alpha2.zip (http://www.wavpack.com/alpha2.zip)

Any comments or suggestions are (as usual) very much appreciated... 
Title: why isn't wavpack more popular?
Post by: den on 2004-01-04 15:25:21
David you legend...

Will comment in due course...

Den.
Title: why isn't wavpack more popular?
Post by: ChristianHJW on 2004-01-05 15:17:59
David,

dont forget you wanted to ping me if you feel our devs can have a look at your sources, so we can comment about matroska compatibility .....
Title: why isn't wavpack more popular?
Post by: glauco on 2004-01-05 17:33:01
Cool !!!

Cristmas gift just in time. A new toy to play with... can't wait to test that new VBR stuff.

Great work David. 
Title: why isn't wavpack more popular?
Post by: glauco on 2004-01-05 18:17:02
Testing right now ...

WOW !!!

The VBR mode is great.

I have tested a bunch of 100 seconds music clips that i have for, well, testing purposes.

The VBR mode is much slower than the normal mode, but as fast as many other codecs. It's around 6x realtime (10 seconds encoding time/ 1 minute song time) in my 2.4GHz Pentium4, and that's with unoptimized code.

The bitrates, between 263 kbps (Babyface - Change the world) and 353 kbps (Queen - A kind of magic), averaging 297 kbps in 40 music clips tested.

The quality seems very good. For the moment I haven't detected any hissing sound, although I haven't been listening very carefully. I also believe my ears are not as good as Den's 

But what i like most is the idea. No target bitrate, no quality settings. Just an standard quality level and an (optional) offset in dB. Simple and effective. 

It's a great idea David. It's a great work.

Perhaps introducing a psycho-acoustic model has been a risky move. I don't know, maybe it can need some tunning in the future, but that remains to be seen... I believe it's been a wise move. ¿How conservative / aggressive have you been with that ?

Edit : I almost forgot : Seeking is as fast as possible. 
Title: why isn't wavpack more popular?
Post by: rjamorim on 2004-01-05 18:36:11
David rocks 
Title: why isn't wavpack more popular?
Post by: den on 2004-01-05 22:30:26
More detailed testing is to follow, but so far in casual listening, I am not picking up any obvious noise/hiss using the default q setting. I even did some stuff at -q-5 which dropped the bitrate down to 250 kbits or less in many cases. I can hear some hiss albeit at still quite low levels considering the bitrate.

Very cool, and yes, it's official. David does seriously rock.  This is EXACTLY what I have been looking for. My wishlist for Wavpack is now nearly complete, apart from foobar support with 4.

 

We just now need to do a few checks to see if anything can fool this puppy!

Den.
Title: why isn't wavpack more popular?
Post by: Dologan on 2004-01-06 06:37:17
Woah! Excellent work, David! I'm really looking forward to fb2k support for it! Given the current state of things we are not too far away from a beta, are we?

A couple of small requests, both for the encoder and the decoder:
1. Decoder: A umm... decoding speed ratio indicator (if that is how it is called ) e.g. 52.2x, 22.3x, etc. It's much more comfortable to have it automatically than to divide the track length by the decoding time. I must say I am quite impressed with the decoding speeds of the -ffq modes reaching ~50x (on my Mobile AthlonXP 1700+)! Keep it up and it will very likely replace MPC insane as my quasi-lossless-for-transcoding-and-archival format of choice!
2. Encoder: Average bitrate display. While the % lossy compression does provide what is needed to get it (100 - % * 1411), again for convenience's sake it would be nifty to have it automatically.

On another issue, the -x appears not to work as explained on the readme for the quality modes. Using -ffq9x6 and -ffq9 for one test song (3:06) yields exactly the same file size with 55s vs 36s encoding time, respectively. Decoding time are also practically the same: 4.07 vs. 4.13. Whether there are quality differences, I don't know (certainly inaudible to me), but if I understood correctly there shouldn't be any, only an impact on bitrate, which stays the same (around 460 kbps). Where is the extra encoding time going? A bug, perhaps?
Title: why isn't wavpack more popular?
Post by: bryant on 2004-01-07 00:33:42
ChristianHJW:
After we communicated last I created a document describing (in general terms) how the future API would look, and that file is also now included in the alpha package. I believe that this is a more useful picture of the Matroska integration issues than the source code because there is no API in the source code. If your devs look over that document and have questions or suggestions then we can go back and forth and they can influence how the API might look. If they still want to see the source after that I can provide something, but it's still kind of a mess and I really don't think they could find anything useful in there.

I do want to make sure that I don't do anything that makes Matroska difficult...

glauco:
Thanks for the testing! The model I used was from a 1988 paper, so there's nothing really new in there. The only real modification I made was having the ATH (absolute threshold of hearing) float relative to the audio level rather than be absolute so that when Den turns the volume way up in a quite part he still won't be able to hear hiss (I say with fingers crossed)! Anyway, when the bitrate came out to about the level I expected, I left it alone. We'll see whether people can find any problem samples...

As for tuning, there's probably not a lot to do because the model is so simple. But I think I will be able to reduce the bitrate some while staying within the model, but not until I finalize the format. My goal here was not to get transparent sound at the lowest possible bitrate, but to work toward a codec that is transparent in all circumstances.

Dologan:
I believe that I mentioned somewhere in the readme that the -x option works best with the default and the -f mode, and not so good with the -ff and -h modes (or maybe I thought about putting that in there). Anyway, the problem is that the -ff mode is so simple there's not much for the -x switch to do; it looks but can't find any improvement unless the signal is very atypical. I actually considered not including that mode at all. I recommend using the -f mode if decode speed is important because it will definitely benefit from -x and decodes almost as fast. Also note that some disk handling optimizations should speed up the "fast" modes somewhat on some systems (on one computer I tested "fast" was actually slower than normal!).

I agree that having average bitrate displayed would be good, maybe even the default for lossy modes? As for encoding/decoding speed as a realtime ratio...well, we'll see what I can do. 

My plan is to have a beta towards the end of this month, although it could easily slip into February as I have some paying work I need to get out of the way...    Anyway, thanks for the testing and suggestions!

rjamorim & den:
Thanks to you guys also! 
Title: why isn't wavpack more popular?
Post by: rinseaid on 2004-01-07 06:40:21
Hi,
After a few quick tests, I just have a quick bug report. When I answered to not overwrite an existing file using wvunpack, it deleted the file instead.

Code: [Select]
D:\wavpack>wvunpack.exe "01-Love Me Do.wv"

WVUNPACK  Hybrid Lossless Wavefile Decompressor  Win32 Version 4.0a1  11-6-03
Copyright (c) 1998 - 2003 Conifer Software.  All Rights Reserved.

overwrite 01-Love Me Do.wav (yes/no/all)? n


I'll report more when I get a chance to do some proper testing.

Thanks David,
rinseaid.
Title: why isn't wavpack more popular?
Post by: den on 2004-01-07 08:56:28
Still early days, but I have now encoded a couple of CDs through it and spent some time doing close attentitive listening, not ABX style though. The CDs I listened to were Seal IV and Street Fighting Years - Simple Minds. The latter one is particularly suitable for picking up hiss in the quiet sections with Wavpack 3.97.

I used the bog standard -q, no x, no f, no h, no nothing.

Very impressive. The bitrates for each track are in the 300 to 320 kbit range, and even at extreme volumes I am not hearing obvious hiss. -hb320 with 3.97 results in files where I can easily hear some hiss even at only moderate high volumes.

I am yet to try some decent ABX testing, but this lossy quality mode seems very reasonable so far.

After an email from David, I added x4 to the command line, just to see how much this squeezes the bitrate. (Quality doesn't change, it justs uses a few less bits.) So far, for the noticeable slowdown in encoding speed, I am only noticing a 5 kbit saving in most cases, so not enough to convince me, but for others where every bit counts, or if you can leave it encoding overnight. I'm trying it again with x6, just to see what happens, and now it is very slow encoding wise as you'd expect.

I have also done a couple of quick listens at -q5 and -q-5, and although -q-5 was hissy, it wasn't as bad as I was expecting, and it did shave another 20-25 kbits off in the couple of tracks I tested.

This is looking better and better everytime I play with it. 

Quote
rjamorim & den:
Thanks to you guys also!


No worries David. Me and rjamorim are just a couple of fanboys...

Later,

Den.
Title: why isn't wavpack more popular?
Post by: PoisonDan on 2004-01-07 09:23:24
Den, do you plan to do another transcoding test and/or a new comparison with Optimfrog Dualstram in the near future? It would be great to see the results of these tests.
Title: why isn't wavpack more popular?
Post by: den on 2004-01-07 11:12:38
Yes and yes PoisonDan.

I don't wish to pre-empt any such tests, but I don't expect the model that David has introduced to cause any significant problems with transcoding. I guess some will be worried that now that a model is involved, it could potentially cause problems, but if my understanding is correct, the model is more focused on using more or less bits in order to ensure that the level of quantisation noise(hiss) is perceptible, rather than deciding which frequencies in a signal will be missed if they are not encoded, etc. Alpha 2 still keeps all the frequencies, etc. The difference is it uses a model to determine whether the hiss can be detected, and if yes, use more bits, and if not, perhaps try a few less.

I'm certainly impressed with how it handles known problem samples from Wavpack 3.97. Where as before you had to make decisions as to whether to use joint stereo or not, or to use a higher bitrate when required for difficult samples, the quality mode just uses more bits when required, without the need for changing your command line. It even goes near lossless to resolve problem areas from what I've seen so far.

Sweet. This is just what I was after.

Anyway, I am still enjoying the last few days of holidays with my family, but after that, I am planning to do some reasonably serious tests and document them here.

Also, while I haven't posted much about alpha 1 on HA, it certainly achieves transparency at lower bit rates than 3.97 in my experience, and is now closer to Dualstream in terms of bitrate to get a given level of perceptible noise or lack thereof.

Later,

Den.
Title: why isn't wavpack more popular?
Post by: bryant on 2004-01-07 23:53:40
Quote
Hi,
After a few quick tests, I just have a quick bug report. When I answered to not overwrite an existing file using wvunpack, it deleted the file instead.

Code: [Select]
D:\wavpack>wvunpack.exe "01-Love Me Do.wv"

WVUNPACK  Hybrid Lossless Wavefile Decompressor  Win32 Version 4.0a1  11-6-03
Copyright (c) 1998 - 2003 Conifer Software.  All Rights Reserved.

overwrite 01-Love Me Do.wav (yes/no/all)? n


I'll report more when I get a chance to do some proper testing.

Thanks David,
rinseaid.


That was definitely not the intended behavior, although it did not actually overwrite the file! 

I have uploaded a new alpha2 with that fixed; hope there was no damage done.

Thanks a lot for letting me know! 
Title: why isn't wavpack more popular?
Post by: ATLien on 2004-01-08 00:31:17
Quote
If you look at the FLAC comparison page, http://flac.sourceforge.net/comparison.html (http://flac.sourceforge.net/comparison.html), you'll see that wavpack normal compresses a few MB better than FLAC maximum, encodes in half the time, and decodes not much slower...

Wavpack is also free and open-source, just like FLAC... and it has an excellent feature that FLAC lacks: hybrid mode ("lossy" + correction file).

FLAC's advantages are that it is stream able, is more well-known (under the xiph umbrella now as well) and has better support (on a few hardware players as well).

Now there isn't perhaps that much between them, but surely wavpack deserves more recognition than it gets?

I'm new to the whole lossless thing (but I know I don't care for WMA), and from what I see a lot here in HA, it seem people put speed before quality. I could care less for the speed of what I'm encoding, my "first" priority would be quality.
So I see a lot of people recommending FLAC or Monkey, etc. How come there aren't any forum listings on these formats? I know that MP3's, Ogg, and AAC are popular, but maybe if there were a little more emphasis on these, maybe, just maybe they can get a little more boost in popularity.
Again, I digress. From what I've been reading I think Flac would be a good choice, plus it seems like more are also jumping on the bandwagon(including me).
As for Wavepak, that's totally in left field and i have to continue may quest for knowledge of these lossless codecs.
Title: why isn't wavpack more popular?
Post by: Destroid on 2004-01-08 03:11:57
Quote
I could care less for the speed of what I'm encoding, my "first" priority would be quality.

Then any lossless compressor should interest you.

However, if you are looking for quality features (since all lossless compressors are 100% [digital] audio quality) then you will need to read up on them since I won't give you an unbiased opinion.

WAVPack seems to have better speed vs. size efficiency than other's recent versions of encoders. Hybrid mode should be interesting for those looking to have both a lossless backup and a PC-machine music collection with lossy without as much space reuqirements compared to something else like lossless + MP3. COol!

edit: much gooder rewording
Title: why isn't wavpack more popular?
Post by: Dologan on 2004-01-08 03:48:41
Quote
I'm new to the whole lossless thing (but I know I don't care for WMA), and from what I see a lot here in HA, it seem people put speed before quality. I could care less for the speed of what I'm encoding, my "first" priority would be quality.
So I see a lot of people recommending FLAC or Monkey, etc. How come there aren't any forum listings on these formats? I know that MP3's, Ogg, and AAC are popular, but maybe if there were a little more emphasis on these, maybe, just maybe they can get a little more boost in popularity.
Again, I digress. From what I've been reading I think Flac would be a good choice, plus it seems like more are also jumping on the bandwagon(including me).
As for Wavepak, that's totally in left field and i have to continue may quest for knowledge of these lossless codecs.

I don't know what you've been reading, pal, but most people here don't put speed before quality by any means. Proof of that would be that the --alt-preset standard is still the recommended setting for LAME, despite the fact that the fast variation of it yields almost the same quality and is significantly faster. On the other hand, why are you talking about quality if these are lossless codecs? They are all "perfect".
FLAC, Monkey, WavPack and the other lossless codecs have their place in the Lossless category. They don't have their own forums simply because there isn't so much to talk about each of them as there is for lossy codecs. Lossless codecs don't have problem samples and can't be tuned but for speed and a little extra compression. Lossless codecs also only come in one flavour, the original one, unlike AAC or MP3 for instance, which have several implementations each.
Anyway, all this is a little off-topic, since the topic has long wandered to another direction.
Title: why isn't wavpack more popular?
Post by: guruboolez on 2004-01-08 08:03:32
This encoder is exciting. I suppose that this new VBR/quality mode is able to keep plain transparency, and restitute noise free encoding, even with classical music.
I’ve played some hours with the –q mode with samples of mine. I’ve tried to find some flaws, and to make things easier, I’ve began to encode with –q-9 in order to select the most noisier sample. I’ve noticed that tonal instruments (organ, strings, wind...) suffers a lot at this low quality setting. But for most of them, the default setting (-q) was not strong enough, and the slight remaining noise was still ABXable. I have even ABXed some samples at -q5, and one at -q9. This last one is a part of an organ sample (bruhns.wav), and the funniest thing is that some lossless encoders are more efficient than the -q9 lossy mode (488 kbps vs 431 kbps for optimFROG -extranew).


Then, I’ve tried to compare wavpack4 alpha (WV) with the other hybrid encoder (dualstream or DS), which is for my very limited experience better than wavpack 3.9 lossy. DS -q3 gave me approx. the same size than WV -q mode. I’ve compared both format with three samples :

- Bruhns.wav (organ)
- Candide.wav (orchestral, strings part)
- Corelli.wav (chamber, strings)




BRUHNS.WAV (organ)

Whereas DS -q3 was smaller than WV -q, the quality was better. With noise shaping, DS outperform WV VBR... More surprising, DS -q1 was similar to WV -q (on some parts noiser, on some other cleaner), with a big difference in bitrate (DS -q1 = 240 kbps // WV -q = 350 kbps). Wavpack -q suffered a lot from unstable noise. Maybe flaws in noise estimation model?
I noticed than noise shaping (DS) was very efficient with -q3, but lowered the quality at -q1
I’ve compared in the same test WV -q-8 and WV -hb230, and quality was really poor (blizzard noise) compared to DS at similar bitrate (-q1 and -q1 ans).




CANDIDE.WAV (orchestral)

This time, DS -q3 was clearly bigger than WV -q. I’ve therefore tested DS -q2 and DS -q3, and WS -q and -q2. I’ve introduced in the same test WV in -b mode (-hb340), in order to compare -q and -b at the same bitrate.

Again, DS -q3 ans was better than anything else (even better than reference: I’ve rated 4.7/5 the reference file...). Second one was WV -hb340, not so transparent. Last one (on 7 files) was WV -q !!! WV -q2 was inferior to DS -q3 (with and without noise shaping).
Again, I’ve noticed that noise shaping (DS) is a real advantage at -q3, but lowered the quality at -q2...
For this sample, WV -q and -q2 mode were clearly worse than -b340




CORELLI.WAV (chamber)

I’ve restricted the test for this sample, which is more difficult to ABX than the two previous one. I’ve tested WV -q and WV -hb320 to DS -q3 and -q3 ans.

Two groups appeared:
- close to transparency, DS -q3 ans and WV -hb320. I gave the best note to DS, though I didn’t tried to ABX the existing difference with WV.
- audible noise : DS -q3 without noise shaping and WV -q. For the first time, WV was superior to DS (but difference is very limited)




CONCLUSIONS

• I can’t say if WV is superior or inferior to DS. For two reasons. First reason: I’ve just tested three samples... Second reason: I used three samples with specific problems for WV4 -q mode.
• VBR or quality mode is not always better than CBR or -b mode. On the three samples, comparison between -b and -q at the same average bitrate was each time in favor of -b mode...
• If all hybrid encoders are suffering from noise, WV VBR is the one to generate unatural (fluctuant) noise, which is really annoying (but maybe very rare).
• (From Wavpack alpha2 readme.txt:) In the default version (-q) the psycho-acoustic model is used directly and should provide transparent operation with virtually all samples and listeners. This is not the case for the moment. Some kind of samples are really noisy with the -q setting (limited noise, but not acceptable IMO for 350 kbps encodings).
• DualStream with noise shaping (and sometimes without) was each time better than Wavpack -q. I suppose that progress are possible for WV4 final )
• DualStream noise shaping seems to have problems with -q1 and -q2 settings, but provide a very impressive quality jump with -q3 (at least, on these three samples).
• Last one : at this bitrate (>300 kbps), it seems to be easier to find problem samples for hybrid encoders than for traditionnal lossy encoders, so I’m not sure that this kind of encoder is really useful (except maybe in the case of correction files stored on CD/DVD or perhaps for loud mastered music).





OTHER SMALL TESTS

• MENDELSSOHN.wav
DS –q3--ans  >  WV –hb340  >  WV –q

• BINIOU.wav
DS –q3--ans > WV –q & WV –hb340 > WV –hb340-s-1



EDIT: samples are uploaded here:
http://www.hydrogenaudio.org/forums/index....ndpost&p=170894 (http://www.hydrogenaudio.org/forums/index.php?showtopic=17303&view=findpost&p=170894)
Title: why isn't wavpack more popular?
Post by: den on 2004-01-08 09:07:55
Thanks for the post guru. Facinating reading. I haven't checked out your samples yet and in fact I'm yet to do detailed ABX tests with Wavpack 4 a 2, but will be later next week.

It's interesting that you mention the fluctuating noise. I actually find the noise shaping in Dualstream quite annoying in some of the samples I have reported previously. I find I can hear subtle changes in the background moving around the notes, and while the volume needs to be high for me to hear it, it bugs me.

I don't think that it is easier or harder to find problem samples for these or the more traditional lossy encoders. They are simply different. If you put fatboy, castanets or some of the other well known problem samples through Wavpack or Dualstream, they handle them quite well. But then something that LAME or Vorbis finds easy to encode, can produce background hiss in Wavpack/DS. I think what is more important is what bugs you. Noticeable artifacts such as pre/post echo, smearing, etc or slight changes in the background hiss.

Also, with regards to your comments about how useful these codecs are, it depends on your needs. I still believe that the lossy files from Wavpack/DS leave vorbis/LAME/mpc/AAC for dead when it comes to transcoding.  My latest music storage system is to have all of my CDs, (~250) on my harddisks as Wavpack lossless as split wv+wvc files. When I am travelling, I take a cut down version of my favourites as the lossy files on a few DVD+RWs, which I can transcode to ATRAC3 and into my MD from my laptop when required with minimal (nil) transcoding artifacts. If I do this with mpc for example, I can clearly hear transcoding problems with some of my files. Not everytime, but enough to bug me.

I suspect that Wavpack 4 still has some work to do to match Dualstream in the quality:bitrate ratio stakes, but the gap is closing.

Den.
Title: why isn't wavpack more popular?
Post by: bryant on 2004-01-09 06:49:38
guruboolez:
Thanks so much for your (as usual) thorough testing! I have downloaded your samples and will be trying to duplicate some of your results and figure out what is going on. BTW, when I used the phrase "virtually...all listeners", I had you (and some aussie dude) in mind. 

There is one thing I would like to clarify about comparisons between WavPack hybrid and OptimFROG Dualstream. Using the command lines that you used and decoding on my 1.7 GHz P4, I measured a 4:1 difference in speed (in WavPack's favor). Even using OptimFROG's fastest mode and WavPack's slowest resulted in a difference of almost 2:1. Now, there is a direct relationship between the complexity of an algorithm and the compression performance in can achieve, and WavPack and DualStream are definitely at opposite sides of the spectrum. If quality at a given average bitrate is the only criteria, then WavPack will almost certainly never match DualStream. However, my emphasis has been on decoding simplicity and the possibility of hardware (portable?) playback.

Changing your WavPack command lines from -q to -qx4 reduced the bitrates on those 4 files by a little over 5 kbps, and switching DualStream to "fast" mode increased its bitrate by about 10 kbps, so there's 15 kbps that might narrow the quality difference somewhat.

Also with respect to bitrate, the current WavPack alpha does not perform optimally, mostly so I could stay compatible with the existing decoders. My emphasis was on finding out whether I had the quality right and then let the bitrate fall somewhat as I went to the beta version. Also, the "fluctuating" noise that you heard may be an artifact of trying to fit into the existing format (especially at negative q settings) and will go away in the beta.

However, this brings up the point that you can still hear noise at -q0. Since the psychoacoustic model I used is pretty old and well established, I can only think of two possibilities. One is that I have implemented something incorrectly, or the problem may be a binaural problem which is not covered by the model. Could I ask you to try an experiment? I would be very interested if you still hear the noise if both the reference and test files are converted into mono just before the ABX test:

A: original file ---> mix to mono
B: original file ---> encode ---> decode ---> mix to mono

If this eliminates the audibility of the noise then I have a very good idea of how to proceed. If not, then I'll quickly come up with another plausible explanation... 

Anyway, thanks again for taking the time to report your results.
Title: why isn't wavpack more popular?
Post by: guruboolez on 2004-01-09 07:53:53
Hello David, and thanks for your positive comments.

On my Duron 800, optimFROG DualStream --quality 3 --ans is really fast compared to wavepack -q (it's faster fast than Vorbis, or Lame preset abr/cbr). In comparison, wv4 -q is painfull (I havn't checked speed, but I can remember that I was bored).

I have compared average bitrate with 20 classical tracks (from 20 different discs), for 2 hours of music. Here are results :

Code: [Select]
           WVq  ofsQ3 mpc10 GT3q9 AACTr

track 01 : 367   345   349   330  +324+
track 02 : 328   346   355   319   292
track 03 : 353   339   388   334   312
track 04 : 316   343  -313-  254  -265-
track 05 : 345  +360+  320   279   306
track 06 : 336  -304-  344   277   289
track 07 :-304-  327   365   321   289
track 08 :+453+ +360+  331   286   303
track 09 : 324   344   328   288   284
track 10 : 348   336   374   304   295
track 11 : 354   352   334   297   308
track 12 : 329   335  +409+ +372+  290
track 13 : 322   353   341   305   298
track 14 : 326   343   347   345   310
Dixit    : 312   348   322   298   310
Wagner   : 332   341   340   296   308
Bayle    :+443+  350   320  +353+ +324+
Beethoven: 357   350   355  -284-  300
Rinaldo  : 352   351   330   298   315
Requiem  : 326   342  -314- -284-  311
---------------
1h'56'55"
7015 sec

wavpack -q..................  289 Mo (303 832 170 octets)   ~337 kbps
DualStream -q 3 --ans.......  287 Mo (301 479 022 octets)   ~335 kbps
musepack -q10...............  289 Mo (303 764 860 octets)   ~337 kbps
vorbis GT3b1 q9.............  250 Mo (262 740 196 octets)   ~292 kbps
AAC Nero ::transcoding::....  253 Mo (265 975 691 octets)   ~295 kbps

14 first tracks are extracted from an anthology (14 different CD, all stereo with one old/noisy piano track [tr#5]).
Track#8 is a two-hand piano, and I can't explain the blow for WV4 (453 kbps - OptimFROG lossless max is 436 kbps).
"Bayle" is electronic music, with micro-impulse, and bitrate explosion is easy to understand.


As you can see, for my purpose (I have only tried classical), both hybrid encoders are close to 335 kbps (with a small advantage for DualStream), and DualStream is much faster and really usable.



EDIT: I can test downix this wek-end, but probably not post results before some days.
EDIT2 : just a precision to avoid confusion, wv4 -qis really good, and I can hear noise on some samples only. The organ sample is the most annoying, because noise level is different for each note (and particulary audible for one - that's why I'm tempted to consider it as a problem sample)
Title: why isn't wavpack more popular?
Post by: bryant on 2004-01-10 19:19:24
guruboolez:

I played around with the organ sample and I agree that it's a problem (I ABX'd it at -q-3 and probably could have at -q if it hadn't been for my new computer's many noisey fans and open headphones). It is very tonal and the part of my model that estimates tonality does not seem to be registering it high enough. I'll have to figure out if it's a bad implementation or whether I need to go to a multi-frame tonal estimator.

I think there's no longer a need for the downmix test if you haven't already done it, although if you have done it I would still be interested in the general results.

BTW, when I mentioned speed I was refering to decode speed, not encode speed (which is done only once). The encode speed of the new -q mode is slower than OptimFROG, although there is probably some room for improvement once I settle on an algorithm.

Thanks again...
Title: why isn't wavpack more popular?
Post by: robUx4 on 2004-01-10 20:26:52
Quote
I have posted a new WavPack 4.0 alpha that includes my first cut at a "quality" mode using psycho-acoustic analysis. The package includes a readme with all the details:

http://www.wavpack.com/alpha2.zip (http://www.wavpack.com/alpha2.zip)

Any comments or suggestions are (as usual) very much appreciated... 

Code: [Select]
typedef struct {
   char ckID [4];  // "wvpk"
   long ckSize;  // size of entire frame (minus 8, of course)
   short version;  // 0x400 for now
   uchar track_no;  // track number (just an idea now)
   uchar index_no;  // remember these? (just an idea now)
   ulong total_samples;    // for entire file (-1 if unknown)
   ulong block_index;  // index of first sample in block (to file begin)
   ulong block_samples;    // # samples in this block
   ulong flags;  // various flags for id and decoding
   ulong crc;      // crc for actual decoded data
} WavpackHeader;


Does the CRC contain this header too ?

Otherwise from a Matroska point of view it all looks good to me. We'll probably strip some unnecessary data from this header when muxing in Matroska and reconstitute it on playback (if that's needed by the decoder).

BTW, can you mix mono and stereo (or more) packets in the same file (for the same stream) ? I guess not, as it's supposed to be lossless and there are no such source material, but I just want to make sure it's OK (not to have this).
Title: why isn't wavpack more popular?
Post by: bryant on 2004-01-11 04:43:27
robUx4:
The header is not included in the CRC; it is for the decoded audio samples only.

The only fields required to be correct by the decoder are the data size, version, block_samples, flags, and crc. I'm not too sure of the advantage of not just leaving the block alone considering how large they generally are (unless you are planning on really short blocks).

Each of these packets are completely stand-alone. They can be decoded back to mono or stereo audio without anything else. So I don't see any reason that you could not mix as many mono and stereo streams as you wanted as long as Matroska was keeping track of them.

The only issue about consecutive blocks is that during encoding it is more efficient if the encoder can start where it left off in the previous block. However, I have written code to encode a block without that (i.e. stand-alone) but it's a little slower because it has to scan the block to set some encoding parameters.

If you would like to discuss this more offline, feel free to e-mail me at the address on the WavPack website.
Title: why isn't wavpack more popular?
Post by: Pamel on 2004-01-11 07:38:03
While you are free to require a certain bitstream always be returned to the decoder, it would be nice if the headers could be stripped. 

As a high level container, the idea is to store only the RAW data inside of the Blocks and store all other information at the container level.  This way tools can be written that can read and manipulate a variety of information on any type of format. 

A good example of this would be subtitles.  For storage in Matroska, the display time and duration are stripped out of the data stored in the Block and stored at the container level.  Because of this it is possible to remove, move, and change the duration of subtitles no matter the format they are stored it (SRT, SSA, VobSub, etc), without the program needing to understand the format. 

Things like the CRC can be stored at the container level, allowing applications to instantly spot a Block with bad data.  Then you can decide exactly what you want to do with that Block.
Title: why isn't wavpack more popular?
Post by: robUx4 on 2004-01-11 09:20:14
Quote
If you would like to discuss this more offline, feel free to e-mail me at the address on the WavPack website.

I tried to email to you but my personal SMTP server was blocked...

You can find a copy of the email here (http://lists.matroska.org/pipermail/matroska-devel/2004-January/001316.html).
Title: why isn't wavpack more popular?
Post by: bryant on 2004-01-15 01:15:51
I figured out what was going on with my tonality estimation and fixed it. Unfortunately, this resulted in an unacceptable increase in average bitrate, so I biased up the noise threshold to get the average back to about 360 kbps. I am happier with this version than the previous (and it does much better on the organ sample from guruboolez), but I realize that this will have to wait for the beta version to really work well at reasonable bitrates.

The latest (and probably last) alpha is here:

http://wavpack.com/alpha3.zip (http://wavpack.com/alpha3.zip)

Thanks to everyone who tried it out!

robUx4 and Pamel:
I have responded by e-mail (to robUx4) regarding these issues. Please let me know if you don't receive something.
Title: why isn't wavpack more popular?
Post by: glauco on 2004-01-15 22:13:57
Nice.

Quality is, for me, far more important than bitrate.

And I was not so wrong about the tuning stuff 

Great work.
Title: why isn't wavpack more popular?
Post by: sshd on 2004-01-16 00:07:34
@bryant

Please consider adding the md5sum [of the datapart of the WAVE file] to the header. Just like FLAC and the newest OptimFROG.
Title: why isn't wavpack more popular?
Post by: NuclearFusi0n on 2004-01-16 17:55:34
Quote
@bryant

Please consider adding the md5sum [of the datapart of the WAVE file] to the header. Just like FLAC and the newest OptimFROG.

an md5sum of the wav file would make me feel a lot more comfortable archiving with wavpack, so i always know my decompressed files are not corrupt.

Also, is there any APL-like feature for wavpack in sight any time soon?
Title: why isn't wavpack more popular?
Post by: kaiwei on 2004-01-17 16:36:20
Pardon me but doesn't using FB2K grants you that ability no matter what the format?
Title: why isn't wavpack more popular?
Post by: bryant on 2004-01-17 21:49:31
Quote
Quote
@bryant

Please consider adding the md5sum [of the datapart of the WAVE file] to the header. Just like FLAC and the newest OptimFROG.

an md5sum of the wav file would make me feel a lot more comfortable archiving with wavpack, so i always know my decompressed files are not corrupt.

Also, is there any APL-like feature for wavpack in sight any time soon?

Md5sum and some sort of APL support are on my "to do" list for some time after the beta of 4.0. What's nice is that with the new WavPack format stuff like this can be easily added without breaking anything.

However, even without md5sum there has been error detection in WavPack since 3.0, so you really don't need to worry about a currupted file going unnoticed.
Title: why isn't wavpack more popular?
Post by: sshd on 2004-01-18 03:40:49
It would be very sweet if Wavpack4 would calculate the md5sum the same way as FLAC does. I believe it cuts the 40 byte RIFF header and md5s the rest. That way I could convert all my FLACs to Wavpack4 and still compare (using md5sum) my collection with my friend's FLAC collections.
Title: why isn't wavpack more popular?
Post by: den on 2004-01-19 04:58:56
For those who are interested, I had a closer listen to some samples which I tend to use when I play with Wavpack and Dualstream. From the Wavpack 4.0 alpha 3 side, it went something like this.

All tests were conducted with alpha 3 as posted above, using -q-2 in the commandline. This setting was used to keep the bitrates below 400 kbits/s, which is about where it needs to be for my needs.

Listening environment: a quiet corner of the house through some Senn headphones out of a Thinkpad laptop, using the onboard sound. All were decoded into wavs, and ABX'ed using the ABX tool in foobar.

blue_monday_intro.wv : 379 kbit/sec. ABX 10/10. Slight background hiss between the beats. No other obvious artifacts as such.
bitter_intro.wv : 368 kbit/sec. ABX 9/10. Very slight change in the background noise, mainly determined at high volumes.
rof_intro.wv : 341 kbit/sec. ABX 10/10. Very slight hiss. Some of the distortion at the peak of the distorted guitar also sounds slightly more distorted, but only right at the loudest peaks. Quite subtle, but there. I have noticed a variation of this effect previously with 3.97 and Dualstream.
badman_intro.wv : 307 kbit/sec. ABX 6/10.  Purely guessing. Transparent so far.

So in a nutshell, the hiss is still there. I have not yet done back to back against Wavpack 3.97 with or without shaping etc, but I am guessing that it will be similar to 3.97 at ~ 320 kbits/sec or so. It was impressive that at least for badman, the encoder used less bits and still sounded transparent, which is the idea behind these quality modes, using more or less bits to maintain quality across all samples, if possible.

I was planning to check this very closely against 3.97 in various forms, but David has suggested I hang back from doing too much as he prepares to go towards a beta release, so I will wait. I might just check a couple of the obvious ones, ie blue_monday_intro and rof_intro against 3.97 to make sure that it has not regressed at least.

Last, but not least, I also transcoded the above Wavpack file to ATRAC3 (132 kbits/sec) and checked it out on Minidisc, compared to the the original wav direct to ATRAC3. I could not detect any transcoding artifacts. I also did a quick check against the same sample as LAME 3.90 alt preset insane transcoded into ATRAC3, and the artifacts were clearly there, as has been reported in this forum previously.

So from testing so far, Wavpack's performance as a lossy transcoding source is still very good.

I'm looking forward to the next beta.

Den.
Title: why isn't wavpack more popular?
Post by: manusate on 2004-02-02 01:52:39
I'm getting anxious to try the WavPack 4.0. Any news? Still alpha stage?



Enjoy!
Title: why isn't wavpack more popular?
Post by: tcmjr on 2004-03-10 12:57:07
Quote
My plan is to have a beta towards the end of this month, although it could easily slip into February as I have some paying work I need to get out of the way...

That was at january , yes I did notice you said it could be delayed

We are entering mid-march ! Any news ? I am holding archiving my cd collection for wavpack 4 !!!!
If you need any help just drop me a line.
Title: why isn't wavpack more popular?
Post by: bryant on 2004-03-12 04:03:21
Well, as I explained in this thread (http://www.hydrogenaudio.org/forums/index.php?showtopic=9653 (http://www.hydrogenaudio.org/forums/index.php?showtopic=9653)), nobody should be waiting for WavPack 4.0... 

However, I am about to release the first beta of 4.0 and it's going to have several significant new features compared to the previous alpha, but most importantly the compressed files will be forward compatible. This should happen before the end of March.

Initially, this will be just the command-line encoder and decoder with a winamp2 plugin (like the alpha), but I will immediately start work on a Foobar plugin and get in the support for decoding existing WavPack files. Because of the problems with the "quality" mode discovered by guruboolez and den, this feature is on hold temporarily. I want to get back to this (and have done some work on it) but I didn't want progress on this mode to delay the initial release any longer.

Thanks to everyone for your support, and I promise to start a new thread when the beta is ready!
Title: why isn't wavpack more popular?
Post by: tcmjr on 2004-03-13 21:35:02
Thanks for the info