Skip to main content

Topic: lossyWAV 1.1.0 Development Thread. (Read 131222 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • Nick.C
  • [*][*][*][*][*]
  • Developer
lossyWAV 1.1.0 Development Thread.
Reply #225
Check the first post.
Quote
Change log 1.0.1q: 30/05/08
Maybe you should attach list of programs wich supporting lossywav?

p.S. GXTranscoder
The list of known supported codecs is in the wiki article linked from the first page in the thread.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848| FLAC -5 -e -p -b 512 -P=4096 -S-

  • Nick.C
  • [*][*][*][*][*]
  • Developer
lossyWAV 1.1.0 Development Thread.
Reply #226
lossyWAV beta 1.0.1s attached to post #1 in this thread.

[edit] The speed of transcoding in foobar2000 has been dramatically increased by the use of STDIN and STDOUT. I transcoded my Mike Oldfield collection (366 tracks, 33h29m) in 22m18s or approximately 90.09x overall {Intel E6600 C2D @ 3.0GHz, 4GB DDR2-800, Windows MCE2005 SP2, source on server, destination 160GB Maxtor SATA HDD}. This was definitely limited by the processor cores as the utilisation plot for the duration of the transcode was a virtual flat-line at 100%.

What this does mean though is that the 'fact' chunk will not be preserved - so be careful not to mix-up your archive FLAC files with your lossyFLAC files. [/edit]
  • Last Edit: 09 June, 2008, 02:59:56 PM by Nick.C
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848| FLAC -5 -e -p -b 512 -P=4096 -S-

  • GeSomeone
  • [*][*][*][*][*]
lossyWAV 1.1.0 Development Thread.
Reply #227
I will put in a check to see if STDIN is being used without STDOUT and if so seek to the beginning of the file and put sensible values in for the overall size and the 'data' chunk size.
Thanks Nick,

Something has changed with STDIN but it seems still not quite right.

When I feed the intermediate file lossywav.lossy.wav to STDIN of FLAC (without --keep.. ) says
-: WARNING: skipping unknown sub-chunk 'fact' (use --keep-foreign-metadata to keep)
-: WARNING: unexpected EOF; expected 5869416 samples, got 5868544 samples

BTW according to foobar2000 the number of samples of the source is also 5869416 (and the lossywav and resulting lossy.flac are 5869056)  I'm not sure foobar displays correct values.

When I let FLAC read the lossy file (with --keep.. )
lossywav.lossy.wav: ERROR reading foreign metadata: invalid WAVE file: unexpected EOF (010)

Is that FACT chunk becoming a nuisance now, or is it something else?
  • Last Edit: 10 June, 2008, 12:44:02 PM by GeSomeone
In theory, there is no difference between theory and practice. In practice there is.

  • Nick.C
  • [*][*][*][*][*]
  • Developer
lossyWAV 1.1.0 Development Thread.
Reply #228
I will put in a check to see if STDIN is being used without STDOUT and if so seek to the beginning of the file and put sensible values in for the overall size and the 'data' chunk size.
Thanks Nick,

Something has changed with STDIN but it seems still not quite right.

When I feed the intermediate file lossywav.lossy.wav to STDIN of FLAC (without --keep.. ) says
-: WARNING: skipping unknown sub-chunk 'fact' (use --keep-foreign-metadata to keep)
-: WARNING: unexpected EOF; expected 5869416 samples, got 5868544 samples

BTW according to foobar2000 the number of samples of the source is also 5869416 (and the lossywav and resulting lossy.flac are 5869056)  I'm not sure foobar displays correct values.

When I let FLAC read the lossy file (with --keep.. )
lossywav.lossy.wav: ERROR reading foreign metadata: invalid WAVE file: unexpected EOF (010)

Is that FACT chunk becoming a nuisance now, or is it something else?
Could you please post your converter settings so that I can duplicate the problem?

Many thanks.

[edit] Problem found here too - average of 249 samples dropped off the end of 85 album lossyFLAC files. It may be something to do with the last read (incomplete codec-block [probably]) - I'll get onto it and revert soon. Until such times, STDIN input is not recommended to be used (--stdout seems fine....) [/edit]
  • Last Edit: 10 June, 2008, 03:44:43 PM by Nick.C
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848| FLAC -5 -e -p -b 512 -P=4096 -S-

  • Nick.C
  • [*][*][*][*][*]
  • Developer
lossyWAV 1.1.0 Development Thread.
Reply #229
[edit] Problem found here too - average of 249 samples dropped off the end of 85 album lossyFLAC files. It may be something to do with the last read (incomplete codec-block [probably]) - I'll get onto it and revert soon. Until such times, STDIN input is not recommended to be used (--stdout seems fine....) [/edit]
Many thanks for the heads up GeSomeone - it was as I surmised and it now seems to work properly in beta 1.0.1t.

lossyWAV beta 1.0.1t attached to post #1 in this thread.
  • Last Edit: 11 June, 2008, 03:15:19 PM by Nick.C
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848| FLAC -5 -e -p -b 512 -P=4096 -S-

  • halb27
  • [*][*][*][*][*]
lossyWAV 1.1.0 Development Thread.
Reply #230
lossyWAV beta 1.0.1t attached to post #1 in this thread.

Thank you, Nick, for the new version.
I encoded my regular music test set with piping and with my bat file.
Speed is great when using piping (though it is good with the bat file).
I tested the musical content just to make sure, and with all the tracks content is identical no matter which encoding way.

Just one thing is strange: the lossy.flac files generated when using piping are 0.5...1.0 per cent bigger than those generated by the bat file. (I don't use meta information with the piped version, but I do so with the bat file solution).
  • Last Edit: 11 June, 2008, 05:02:38 PM by halb27
lame3995o -Q1

  • Nick.C
  • [*][*][*][*][*]
  • Developer
lossyWAV 1.1.0 Development Thread.
Reply #231
lossyWAV beta 1.0.1t attached to post #1 in this thread.
Thank you, Nick, for the new version.
I encoded my regular music test set with piping and with my bat file.
Speed is great when using piping (though it is good with the bat file).
I tested the musical content just to make sure, and with all the tracks content is identical no matter which encoding way.

Just one thing is strange: the lossy.flac files generated when using piping are 0.5...1.0 per cent bigger than those generated by the bat file. (I don't use meta information with the piped version, but I do so with the bat file solution).
I would guess that FLAC is inserting a larger number of seek points due to the fact that (in foobar2000) the assumed size of the piped file is maximum, i.e. 4,294,967,294 bytes.

Processing on my laptop, using two intermediate WAV files, used to be a real HDD thrashing exercise - 40 to 50 minutes for my 10 album test set (3.36GB, 9h22m57s total duration). Using piped input and output, the transcode now takes approximately 10 minutes - a really nice reduction in total processing time.

I've just finished a "stress test" of sorts: all of my FLAC'ed CD's on my server to lossyFLAC (-q 1.25). 12d4h24m45s (4064 tracks, 316 CD's, 108GB, 887kbit/s average) transcoded in 3h33m23s (40.5GB, 331 kbit/s average) - 82.22x overall and exactly the same number of samples out as went in . Bearing in mind that previously my 3686 track 100GB(ish) transcode took approximately 7h30m, that's an appreciable saving.

If there are no reported problems and if no-one has any further suggestions then now may be the time to move towards a release candidate for 1.0.2.
  • Last Edit: 11 June, 2008, 05:33:27 PM by Nick.C
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848| FLAC -5 -e -p -b 512 -P=4096 -S-

  • GeSomeone
  • [*][*][*][*][*]
lossyWAV 1.1.0 Development Thread.
Reply #232
lossyWAV beta 1.0.1t attached to post #1 in this thread.
Just one thing is strange: the lossy.flac files generated when using piping are 0.5...1.0 per cent bigger than those generated by the bat file. (I don't use meta information with the piped version, but I do so with the bat file solution).
Yes thanks a lot, this time my little script works as intended 

Except from the metadata it could be the larger seektable that FLAC allocates for a file of unknown length, as foobar2000's converter generates.

I'll show my script (batfile) that (with this fix)
  • handles either - or %s as input.
  • one intermediate file less than the "old" script
  • the FLAC files are a few kB smaller because no large seektables and 2kB padding (more than enough for me) instead of 8kB.
  • Foreign metadata is added to the FLAC file, though I doubt lossyWav will ever see it again when converting from STDIN (with foobar).
Cons
  • can run only 1 converter thread at once
  • takes less advantage of multi core/hyperthreading than piping directly from lossyWav into FLAC
foobar2000:
Encoder:    C:\WINDOWS\system32\cmd.exe
Extension:  lossy.flac
Parameters: /D /C C:\PROGRA~1\FLAC\lossyFlac.bat - %d -q 4.5  (or any lossyWav settings)

and my lossywav.bat
Code: [Select]
@echo off
set lossyWAV_path="C:\Progra~1\FLAC\lossyWAV.exe"
set FLAC_path="C:\Progra~1\FLAC\flac"
set Mylog="C:\Progra~1\FLAC\lossyFlac.log"

echo # lossyWAV.exe "%~N2.flac" %3 %4 %5 %6 %7 %8 --quiet >> %Mylog%
%lossyWAV_path% %1 %3 %4 %5 %6 %7 %8 --quiet --nowarnings --low --writetolog
type lossyWav.log >> %MyLog%
del lossyWav.log

set MyLossyFile="%~N1.lossy.wav"
if NOT %1 == - goto :FLAC
set MyLossyFile="lossywav.lossy.wav"

:FLAC
%FLAC_path% -5 -e -s -b 512 --force --padding=2048 --keep-foreign-metadata %MyLossyFile% -o%2 >>%Mylog%

del %MyLossyFile%

set MyLossyFile=
set Mylog=
set lossyWAV_path=
set flac_path=
Of course that logging stuff can be removed and on Vista it should be in another location.

[edit] some tidying up and removing a debug line from the script [/edit]
  • Last Edit: 11 June, 2008, 06:21:23 PM by GeSomeone
In theory, there is no difference between theory and practice. In practice there is.

  • halb27
  • [*][*][*][*][*]
lossyWAV 1.1.0 Development Thread.
Reply #233
I would guess that FLAC is inserting a larger number of seek points ...

Yes, it's the seektable. When using --no-seektable with FLAC, I arrive at an expected file size.

Is the seektable necessary? I had no problems skipping part of the track (using foobar) with the --no-seektable version.

What kind of problems can I encounter when using --no-seektable?
  • Last Edit: 12 June, 2008, 01:50:31 PM by halb27
lame3995o -Q1

  • Nick.C
  • [*][*][*][*][*]
  • Developer
lossyWAV 1.1.0 Development Thread.
Reply #234
What kind of problems can I encounter when using --no-seektable?
Did Josh not explain this once - I think that it only takes a bit longer to seek when moving about the file.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848| FLAC -5 -e -p -b 512 -P=4096 -S-

  • halb27
  • [*][*][*][*][*]
lossyWAV 1.1.0 Development Thread.
Reply #235
I don't encounter any disadvantage using --no-seektable when playing on my DAP.

So I'll use --no-seektable together with GeSomeone's --padding=2048 proposal and enjoy piping.

...If there are no reported problems and if no-one has any further suggestions then now may be the time to move towards a release candidate for 1.0.2.

I second this.
lame3995o -Q1

  • sidewalking
  • [*][*][*]
lossyWAV 1.1.0 Development Thread.
Reply #236
I don't know if anyone uses REACT to do their EAC tasks, but I have a rough (but working) model of hacking the track-config file to use lossyWAV/lossFlac.

Code: [Select]
IF NOT @lossyFlac@==1 GOTO end_lossyFlac_tracks
    IF NOT EXIST %TrackDir_lossyFlac% MKDIR %TrackDir_lossyFlac%
    PUSHD %TrackDir_lossyFlac%
        IF @various@==1 SET VA_tag=-T "album artist=@VA@"
        IF %embed_cover%==1 SET Cover_tag=--picture="|image/jpeg|||@cover@"
        ECHO ON
    @tools@\lossywav.exe "@source@" @Opt_lossyWAV@
    RENAME *lossy.wav "temp.lossy.wav"
    DEL "@source@"
        @tools@\flac.exe @Opt_lossyFlac@ %Cover_tag% %VA_tag% %Disc_Flac% -T artist="@artist@" -T album="@album@" -T tracknumber="@track@" -T title="@title@" -T date="@year@" -T genre="@genre@" -T comment="@comment@" -T encoding="Flac @Ver_Flac@ @Opt_lossyFlac@" "temp.lossy.wav" -o "%TrackName%.lossy.flac"
    DEL "temp.lossy.wav"
        @ECHO OFF
        IF %have_cover%==1 IF NOT EXIST folder.jpg COPY "@cover@" folder.jpg
    POPD
:end_lossyFlac_tracks


(and these were in the REACT.ini:)
Code: [Select]
Opt_lossyFlac=-5 -b 512 --keep-foreign-metadata
Opt_lossyWAV=-q 5



Not groundbreaking, I guess, but a start.  I don't know how to use the temp variables in batch files, so I use a static temp name and then delete it.  But, it works fine.  I get resulting lossyFlac files with proper tagging, album art embedded, and Replay Gain tagging.

  Scott

  • collector
  • [*][*][*]
lossyWAV 1.1.0 Development Thread.
Reply #237
What kind of problems can I encounter when using --no-seektable?
Did Josh not explain this once - I think that it only takes a bit longer to seek when moving about the file.

At first, when ripping to images, I added seekpoints for every 90 seconds. Now I use -S- and noticed that the  cue points from the embedded cue sheet are sufficient. And because I can use the cuesheet to play the disc image I can jump to any track or movement I like I like.

  • sauvage78
  • [*][*][*][*][*]
lossyWAV 1.1.0 Development Thread.
Reply #238
hi, today I played with lossywav again in order to have it working for good with F2K & tak, so I copy/pasted your command line from post N°1, & it worked like a charm with flac

/d /c c:\data_nic\bin\lossywav - --standard --silent --stdout|c:\data_nic\bin\flac - -b 512 -5 -f -o%d

then I changed the directory path & flac setting according to my own taste c:/z directory + Tak, & I ended with:

/d /c c:\z\lossywav - --portable --silent --stdout|c:\z\Takc -e -p3 -fsl512 -ihs - %d

it worked like a charm too even if I didn't tested expected size against actual size.

But my knowledge being very limited I don't undertand the use of /d /c ...
I understand that I call cmd.exe with c:\windows\system32\cmd.exe then lossywav with c:\z\lossywav - --portable --silent --stdout| then tak with c:\z\Takc -e -p3 -fsl512 -ihs - %d, but I don't get the use of /d /c, but if I delete it it doesn't work anymore so could someone explain the use of /d /c ? What is it needed for ? Also is there any mistake in my own lossyTak command line ? Thks
  • Last Edit: 14 June, 2008, 09:28:31 AM by sauvage78
Rip & Check: EAC Secure [Low/C2]+CUETools [AR Confidence 2+]
NAS (Backup): Flac -4 (for Speed) | CDImage+CUE with F2K
DAP (Playback): Opus 128Kbps | Tracks with F2KM on Android (LG G5)
Video: VP10 (2160p30@24Mbps/2160p60@48Mbps) Asap !!!

  • Nick.C
  • [*][*][*][*][*]
  • Developer
lossyWAV 1.1.0 Development Thread.
Reply #239
hi, today I played with lossywav again in order to have it working for good with F2K & tak, so I copy/pasted your command line from post N°1, & it worked like a charm with flac

/d /c c:\data_nic\bin\lossywav - --standard --silent --stdout|c:\data_nic\bin\flac - -b 512 -5 -f -o%d

then I changed the directory path & flac setting according to my own taste c:/z directory + Tak, & I ended with:

/d /c c:\z\lossywav - --portable --silent --stdout|c:\z\Takc -e -p3 -fsl512 -ihs - %d

it worked like a charm too even if I didn't tested expected size against actual size.

But my knowledge being very limited I don't undertand the use of /d /c ...
I understand that I call cmd.exe with c:\windows\system32\cmd.exe then lossywav with c:\z\lossywav - --portable --silent --stdout| then tak with c:\z\Takc -e -p3 -fsl512 -ihs - %d, but I don't get the use of /d /c, but if I delete it it doesn't work anymore so could someone explain the use of /d /c ? What is it needed for ? Also is there any mistake in my own lossyTak command line ? Thks
Code: [Select]
C:\Documents and Settings\User>cmd /?
Starts a new instance of the Windows XP command interpreter....

/C      Carries out the command specified by string and then terminates
/D      Disable execution of AutoRun commands from registry (see below)
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848| FLAC -5 -e -p -b 512 -P=4096 -S-

  • sauvage78
  • [*][*][*][*][*]
lossyWAV 1.1.0 Development Thread.
Reply #240
thks for cmd /? ... didn't knew about it 
Rip & Check: EAC Secure [Low/C2]+CUETools [AR Confidence 2+]
NAS (Backup): Flac -4 (for Speed) | CDImage+CUE with F2K
DAP (Playback): Opus 128Kbps | Tracks with F2KM on Android (LG G5)
Video: VP10 (2160p30@24Mbps/2160p60@48Mbps) Asap !!!

  • Nick.C
  • [*][*][*][*][*]
  • Developer
lossyWAV 1.1.0 Development Thread.
Reply #241
thks for cmd /? ... didn't knew about it 
No problem - I'm glad that the command line is working properly for you. Thanks for the Tak command line - I'll include it in the first post.

From what collector and halb27 have said, maybe --no-seektable -P=4096 would be a sensible way of reducing the size of the seektable when using STDIN / STDOUT in foobar2000. I processed my 10 Album test set and I think this saved about 1MB (in 1.37GB processed). Still, 1MB is 1MB....
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848| FLAC -5 -e -p -b 512 -P=4096 -S-

  • TBeck
  • [*][*][*][*][*]
  • Developer
lossyWAV 1.1.0 Development Thread.
Reply #242
then I changed the directory path & flac setting according to my own taste c:/z directory + Tak, & I ended with:

/d /c c:\z\lossywav - --portable --silent --stdout|c:\z\Takc -e -p3 -fsl512 -ihs - %d

it worked like a charm too even if I didn't tested expected size against actual size.

Also is there any mistake in my own lossyTak command line ? Thks

Looks good 

In my evaluations of LossyWav processed TAK test file sets there was no advantage of choosing a preset higher than -p2m. Some file sets even compressed about 0.05 percent worse with -p3 to -p5...

  Thomas

  • TBeck
  • [*][*][*][*][*]
  • Developer
lossyWAV 1.1.0 Development Thread.
Reply #243
edit: Content of duplicate post deleted...
  • Last Edit: 14 June, 2008, 12:59:20 PM by TBeck

  • sauvage78
  • [*][*][*][*][*]
lossyWAV 1.1.0 Development Thread.
Reply #244
Nick.C:
... well I haven't yet read all the discussion of collector and halb27 on seektable ... I only come to read this thread once a week/month to see how things go on ... I know I will not use lossywav for production as long as it's not stable ... so I am not in a hurry to read it all

TBeck:
Thks for the hint ... I was planning to test the efficiency of flac -q6 vs tak -p3 vs wv -hm on lossywav processed files as I think I read somewhere not all lossless codecs where equal when encoding from lossywav ... I think I read wavpack wasn't optimal compared to tak & flac ... will add tak at different settings when I will test ... post lossywav V1.2 ... with some luck maybe by then I will have a frozen tak too
Rip & Check: EAC Secure [Low/C2]+CUETools [AR Confidence 2+]
NAS (Backup): Flac -4 (for Speed) | CDImage+CUE with F2K
DAP (Playback): Opus 128Kbps | Tracks with F2KM on Android (LG G5)
Video: VP10 (2160p30@24Mbps/2160p60@48Mbps) Asap !!!

  • Nick.C
  • [*][*][*][*][*]
  • Developer
lossyWAV 1.1.0 Development Thread.
Reply #245
then I changed the directory path & flac setting according to my own taste c:/z directory + Tak, & I ended with:

/d /c c:\z\lossywav - --portable --silent --stdout|c:\z\Takc -e -p3 -fsl512 -ihs - %d

it worked like a charm too even if I didn't tested expected size against actual size.

Also is there any mistake in my own lossyTak command line ? Thks
Looks good 

In my evaluations of LossyWav processed TAK test file sets there was no advantage of choosing a preset higher than -p2m. Some file sets even compressed about 0.05 percent worse with -p3 to -p5...

  Thomas
Thanks for the clarification, Thomas.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848| FLAC -5 -e -p -b 512 -P=4096 -S-

  • sauvage78
  • [*][*][*][*][*]
lossyWAV 1.1.0 Development Thread.
Reply #246
Just tested with wavpack 4.5 & it worked too, here is my 3 settings for Tak/Flac/Wv:

Code: [Select]
Encoder:    c:\windows\system32\cmd.exe
Extension:  lossy.tak
Parameters: /d /c c:\z\lossywav - --portable --silent --stdout|c:\z\Takc -e -p3 -fsl512 -ihs - %d
Format is:  lossless or hybrid
Highest BPS mode supported: 24


Code: [Select]
Encoder:    c:\windows\system32\cmd.exe
Extension:  lossy.flac
Parameters: /d /c c:\z\lossywav - --portable --silent --stdout|c:\z\flac - -6 -b 512 -f -o%d
Format is:  lossless or hybrid
Highest BPS mode supported: 24


Code: [Select]
Encoder:    c:\windows\system32\cmd.exe
Extension:  lossy.wv
Parameters: /d /c c:\z\lossywav - --portable --silent --stdout|c:\z\wavpack -hm --blocksize=512 --merge-blocks -i - %d
Format is:  lossless or hybrid
Highest BPS mode supported: 24


wavpack works but even with --merge-blocks its efficiency seems reduced compared to real lossless source, for exemple on my single test album flac -6 compressed more than wavpack -h on this lossywav file while according to my experience wavpack -h usually beats flac -6 on real lossless music.

also when I tried to add --keep-foreign-metadata to the flac command line, it crashed.
  • Last Edit: 15 June, 2008, 10:39:44 AM by sauvage78
Rip & Check: EAC Secure [Low/C2]+CUETools [AR Confidence 2+]
NAS (Backup): Flac -4 (for Speed) | CDImage+CUE with F2K
DAP (Playback): Opus 128Kbps | Tracks with F2KM on Android (LG G5)
Video: VP10 (2160p30@24Mbps/2160p60@48Mbps) Asap !!!

  • GeSomeone
  • [*][*][*][*][*]
lossyWAV 1.1.0 Development Thread.
Reply #247
also when I tried to add --keep-foreign-metadata to the flac command line, it crashed.

That option can only be used when reading from a file (not from STDIN).
In theory, there is no difference between theory and practice. In practice there is.

  • Nick.C
  • [*][*][*][*][*]
  • Developer
lossyWAV 1.1.0 Development Thread.
Reply #248
lossyWAV 1.0.1u RC1 attached to post #1 in this thread.
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848| FLAC -5 -e -p -b 512 -P=4096 -S-

  • halb27
  • [*][*][*][*][*]
lossyWAV 1.1.0 Development Thread.
Reply #249
lossyWAV 1.0.1u RC1 attached to post #1 in this thread.

Thanks for the new RC.
I'm looking forward to the next release.
lame3995o -Q1