Skip to main content

Topic: WavPack 4.0 Beta Release 3 (Read 32102 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • Liisachan
  • [*][*][*]
  • Members (Donating)
WavPack 4.0 Beta Release 3
Reply #50
Quote
Quote
Actually, this same bug was first found in ttaenc 3.1, and I did the same testing in wavpack, and was embarrassed to learn that Wavpack has the same problem...
I hope this will be fixed in the final 4.0 release...

Hmm. This is a tough one because your test files work fine here, and I was not able to find any directly applicable information on the Internet. Was the problem fixed in TTA? Maybe I could look at what they did. It probably won't help, but you might try putting the whole filename in quotes.


Well, I think you can reproduce the problem by setting your locale (system langauge) = Japanese. In Windows XP,  control panel -> Regional and Language Options -> Advanced -> Language for non-unicode program.
You dont need to have Japanese version OS to reproduce this bug. My OS is not Japanese version either. And possibly the same problem may happen in another language using multibute char. such as Chinese.

problematic 2nd half of double-byte char is one of the most common problems in  i18n. There should be more than one solutions, one of which is to use Unicode internally, like many internationalized apps do.

A double-byte char uses 2 bytes to express 1 glyph. The problem is, the 2nd half of those 2 bytes might be confusing, like, it can be 0x5c and innocent parser thinks 0x5c as the path separator, or the letter for an escape sequence, and will be messed up. But it actually is not such a special (meta)character, but just the 2nd half of a common double-byte character.

The author of TTA, Alexander aka ald,  said "I think that I know the reason of this problem. Going to fix it now." a few days ago.
-o switch in ttaenc can already handle double-byte char decently, and WinTTA (tau producer) is more perfect. As for WavPack, this problem is not conspicuous, because it supports pipeline, so if the user uses foobar2000 as the front-end, then the filename handling will be done by foobar2000. TTAENC cannot do that, not supporting pipeline, so it should parse %s from foobar2000 by itself anyway. So you might want to check his source when ready, or someone else's who knows this problem well...

The reason of this problem is very simple, but possibly fixing it is not that easy.
I hope you will find an elegant way to solve it.....
tyia
  • Last Edit: 11 July, 2004, 11:11:15 AM by Liisachan

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
WavPack 4.0 Beta Release 3
Reply #51
Quote
Actually, this same bug was first found in ttaenc 3.1, and I did the same testing in wavpack, and was embarrassed to learn that Wavpack has the same problem...
I hope this will be fixed in the final 4.0 release...

Unfortunately, changing my language settings did not change how that test folder unpacked, but I think I figured out what was going on anyway. Please let me know if this helps:

http://wavpack.com/wp4-jis.zip

Thanks again, Case. 

  • Liisachan
  • [*][*][*]
  • Members (Donating)
WavPack 4.0 Beta Release 3
Reply #52
Quote
Unfortunately, changing my language settings did not change how that test folder unpacked, but I think I figured out what was going on anyway. Please let me know if this helps:
http://wavpack.com/wp4-jis.zip


Did you "install files for East Asian languages" (Language tab)? Maybe you have to change the "Text services and input languages" too.  I can reproduce this problem on my Windows XP English version...

Anyway, thanks a lot for your work.
I tested "Win32 Version 4.0b3x  11-7-04" and the bug related 0x5C has been fixed

Yes, fixed!  Thanx again!

But this fix has this terrible side effect
- now wavpack can not handle all the other double-byte chars (that is, all the double-byte chars that do not include 0x5c in the trailing byte)

Since its not that easy to describe what is happening by words, I upped this pic.
Could you please chack this one?
http://www.faireal.net/tmp/wv4b3x.png

wavpack [965C].wav
DOES work, while
wavpack [82A0].wav
fails.

The error message is: "file [82A0].wav.wav not found!" 

The original WavPack 4.0b3 can do:
wavpack [82A0].wav
successfully, so your last tweak includes both debugging and enbugging...

I hope you wont start hating double-byte chars for that.

  • Duble0Syx
  • [*][*][*][*]
WavPack 4.0 Beta Release 3
Reply #53
It may be a dumb question, but does wavpack 4.0 support adding tags while encoding?  For example when using Exact Audio Copy for extractio directly to wavpack.  Would be nice if the files could be tagged directly, kind of like flac, rather than with some other app.  If this is possible I'd be pleased to hear the command line string to do so.  That would be the only thing slowing my down from switching.  So far it looks like damn good work.

  • Faelix
  • [*][*]
WavPack 4.0 Beta Release 3
Reply #54
Quote
It may be a dumb question, but does wavpack 4.0 support adding tags while encoding?  For example when using Exact Audio Copy for extractio directly to wavpack.  Would be nice if the files could be tagged directly, kind of like flac, rather than with some other app.  If this is possible I'd be pleased to hear the command line string to do so.  That would be the only thing slowing my down from switching.  So far it looks like damn good work.
[{POST_SNAPBACK}][/a]


Well, actually it is through another little application, but in the end it is transparent when using EAC, so learn on this [a href="http://www.saunalahti.fi/cse/EAC/]page[/url] how to do it.

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
WavPack 4.0 Beta Release 3
Reply #55
Quote
But this fix has this terrible side effect
- now wavpack can not handle all the other double-byte chars (that is, all the double-byte chars that do not include 0x5c in the trailing byte)

Yeah, I was afraid that might happen. The problem is that shift-jis cannot be reliably parsed backward because the second byte of the two byte codes can also be above 0x80 (thanks, Bill). What I did should work in all cases except when the double byte code is the last character in the filename (I think).

I know how to fix this correctly, but it's a little more work. Thanks for helping me out on this.
  • Last Edit: 12 July, 2004, 01:16:39 PM by bryant

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
WavPack 4.0 Beta Release 3
Reply #56
Well, it wasn't as hard as I thought. I think I have got every case this time:

http://wavpack.com/wp4-jis.zip

Thanks very much for your assistance and patience on this.

  • Liisachan
  • [*][*][*]
  • Members (Donating)
WavPack 4.0 Beta Release 3
Reply #57
Quote
The problem is that shift-jis cannot be reliably parsed backward because the second byte of the two byte codes can also be above 0x80 (thanks, Bill).

Exactly.... You cannot parse it backward.
Quick hack would be like this:

Code: [Select]
    char szArg[] = "C:\\My Music\\[Some DB Charcters]\\foo.wav";
    char szBuf[ MAX_PATH + 1 ];
    char szPathSeparated[ MAX_PATH + 1 ][ MAX_PATH + 1 ];
    int iPathSeparated = 0;
    memset( szBuf, '\0', sizeof( szBuf ) );
    if( GetSystemDefaultLangID() == 0x411 ) // Japanese
    {
 int len = lstrlen( szArg );
 for( int i = 0, j = 0; i < len; )
 {
     if( (UINT) szArg[ i ] > 0x80 )
     {
   szBuf[ j++ ] = szArg[ i++ ];
   szBuf[ j++ ] = szArg[ i++ ];
     }
     else
     {
   if( szArg[ i ] == '\\' )
   {
       lstrcpy( szPathSeparated[ iPathSeparated++ ], szBuf );
       i++;
       j = 0;
       memset( szBuf, '\0', sizeof( szBuf ) );
   }
   else
   {
       szBuf[ j++ ] = szArg[ i++ ];
   }
     }
 }
    }


But this is ugly. You might want to process everything in Unicode internally....

  • Liisachan
  • [*][*][*]
  • Members (Donating)
WavPack 4.0 Beta Release 3
Reply #58
ADDED: Sorry, not yet fixed.

I forgot to test the "half-width" case.  In Shift_JIS Code Page, 0x80-FF can be used as a single-byte char too, like 0x80-FF used in French or German for special characters. Wavpack cannot handle this case yet.

Heres a demo result (pic):
http://www.faireal.net/tmp/0xb1.png

/ADDED




Quote
Well, it wasn't as hard as I thought. I think I have got every case this time:

http://wavpack.com/wp4-jis.zip

Thanks very much for your assistance and patience on this.
[{POST_SNAPBACK}][/a]


Thank you very much
You posted that one while I'm posting another one,
and YES, looks like this newest build finally fixes all the problems related to Shift_JIS. 

Im really happy for that, and at the same time, let me add this for the future development:
That build is not yet perfect, as many other same tools are not, when it should handle Unicode filenames.
For instance, it cannot handle the path which contains both Japanese and Latin special characters such as ä (a umlaut).

Youll get an error like this (yeah, this is Bill's fault, basically):
[a href="http://www.faireal.net/tmp/mixed_filename.png]http://www.faireal.net/tmp/mixed_filename.png[/url]

So, users would be in trouble if they had a folder in Japanese name and if they had some music files in German or French titles in it. Moreover, on win2k/xp, you can mix, say, Korean, Japanese, Arabic, and French in one filename. I guess that's why modern windows programs need to be Unicode-based.
It is very difficult to handle this gap for commandline tools because cmd.exe does not support Unicode properly while Windows per se uses Unicode. But ofcos normal Win32 GUI applications can manage unicode file names, like foobar2000 can open files in Japanese and files in Thai and files in Korean in the one same playlist.

Thanx again anyway
  • Last Edit: 12 July, 2004, 08:00:27 PM by Liisachan

  • bryant
  • [*][*][*][*][*]
  • Developer (Donating)
WavPack 4.0 Beta Release 3
Reply #59
Okay, this is obviously not the way to fix this problem because even if I get this language working right, there are hundreds of others. One of these days I will look into this fully.

In the meantime I have generated one final version which might just fix all cases for shift-jis because I now use only 81-9f and e0-fc for the 2-byte introduction and I query the default language.

http://wavpack.com/wp4-jis.zip

Thanks again for assistance... 

  • Liisachan
  • [*][*][*]
  • Members (Donating)
WavPack 4.0 Beta Release 3
Reply #60
Fixed anyway. Thank you, bryant

Quote
Okay, this is obviously not the way to fix this problem because even if I get this language working right, there are hundreds of others. One of these days I will look into this fully.


Thats exactly what I thought, as there are even "unicode-only" languages, like Tatar, now working on Windows somehow. But on the other hand, this confusing situation is fully understandable when I remember that WavPack has a long history, begining from the time when Unicode was not common at all.

Quote
Thanks again for assistance... 
[a href="index.php?act=findpost&pid=225459"][{POST_SNAPBACK}][/a]


You're welcome
I am the one who should be thanking you, for your cool freeware, for its high speed (one of the fastest as lossless), and for its imaginative hybrid mode.
Keep up a good work!

Greetings from the east end of Asia
- Liisachan

  • robUx4
  • [*][*][*][*]
  • Developer (Donating)
WavPack 4.0 Beta Release 3
Reply #61
Yeah Liisachan is always a precious help for developpers. You can ask Mosu about it
We even named a Matroska release "Liisachan / CHIP release" (as it was also mentioned in the CHIP german magazine) !

  • jcoalson
  • [*][*][*][*][*]
  • Developer
WavPack 4.0 Beta Release 3
Reply #62
Quote
Quote
Quote
Bryant - is there any chance you could add a seek function to wvunpack? I rip cds to single flacs now and use flac's seek function to extract individual tracks from the resulting file. I can't do this with wvunpack.
That is a nice feature of FLAC. I'll look into puting something like that in. (What would be nice is if could use the cuesheet and just decompress by track number...)

Automatically splitting from cuesheets would be excellent!  One vote from me![a href="index.php?act=findpost&pid=224482"][{POST_SNAPBACK}][/a]

yes, this is a useful feature.  some squeezebox guys have been asking for this too since they use the command-line to decode and pipe out data.  I just checked in just such a --cue option to CVS; will be in the next release pretty soon.

Josh

  • kuniklo
  • [*][*][*]
  • Developer (Donating)
WavPack 4.0 Beta Release 3
Reply #63
Quote
Quote
Quote
Quote
Bryant - is there any chance you could add a seek function to wvunpack? I rip cds to single flacs now and use flac's seek function to extract individual tracks from the resulting file. I can't do this with wvunpack.
That is a nice feature of FLAC. I'll look into puting something like that in. (What would be nice is if could use the cuesheet and just decompress by track number...)

Automatically splitting from cuesheets would be excellent!  One vote from me![a href="index.php?act=findpost&pid=224482"][{POST_SNAPBACK}][/a]

yes, this is a useful feature.  some squeezebox guys have been asking for this too since they use the command-line to decode and pipe out data.  I just checked in just such a --cue option to CVS; will be in the next release pretty soon.

Josh
[a href="index.php?act=findpost&pid=226622"][{POST_SNAPBACK}][/a]


I'll definitely be making use of this.  Thanks!

  • Duble0Syx
  • [*][*][*][*]
WavPack 4.0 Beta Release 3
Reply #64
I just wanted to comment on how nice wavpack is looking.  From what I can tell it seems overall better than most lossless compressors.  When compared with things like optimfrog and ape, the compression is bit less, but damn does it encode fast.  I've testing it using -hm for encoding.  I like how it stores the wav files md5 in the file.  Not found any bugs yet either.  Nice work.  Just waiting for the final release now.
  • Last Edit: 20 July, 2004, 04:28:05 AM by Duble0Syx

  • kuniklo
  • [*][*][*]
  • Developer (Donating)
WavPack 4.0 Beta Release 3
Reply #65
Quote
yes, this is a useful feature.  some squeezebox guys have been asking for this too since they use the command-line to decode and pipe out data.  I just checked in just such a --cue option to CVS; will be in the next release pretty soon.

Josh


Just tried this out.  Seems to be working great.  Thanks!