I know this question has been brought up before, most specifically on the Vorbis mailing list, but I still don't quite understand why 2-pass encoding isn't supported by Oggenc. Sure, the savings are minimal, but isn't that what lossy coding is all about? To make every bit count?
maybe in the future rehuff could be integrated, maybe john can create a special version
On a single example, rehuff compresses a 6016Kb file to 5925Kb. The saving is about 3kbps on this file encoded at -q 6.25. It would not be difficult to integrate this as an option into oggenc2.1 (my hybrid with lossless decoding support), but is there really a demand for it?
it would be a nice feature to play with and if its not too difficult go ahead, we would love it
Because of the sub-optimal one pass method, it is possible to gain some space by recompressing an Ogg Vorbis file, but space savings as dramatic as 10% could not be reproduced with the current version of the encoder. (For example, a quick test by myself resulted in a pitiful 0.2% gain on a randomly chosen file from my collection.)
Segher Boessenkool is currently developing a tool that optimizes the Huffman codes for a given Ogg Vorbis file in order to shave off a few percent of the file size. He reports that the resulting space savings are normally under 5%, and typically between 1% and 2%.
In video, double pass is not used for saving space, but to maximize quality for a given size/bitrate. I think it should be the same for audio.
I don't know if this is doable, but a 2nd pass could scan the compressed file for heavy compression damages and reencode the block with a higher bitrate. And do the same on highbitrate blocks, recompressing them on a lower bitrate.
How do you want to detect artefacts? There's no way to teach a computer, how an artefact "looks" like
I don't know if this is doable, but a 2nd pass could scan the compressed file for heavy compression damages and reencode the block with a higher bitrate. And do the same on highbitrate blocks, recompressing them on a lower bitrate.
No, I think the problem is more or less like this: The 2nd pass is really worth only with ABR-like bitrate managment. With video codecs like DivX or Xvid most of the times you're targeting for a certain bitrate, much like LAME ABR. So the problem is: if the codec, as it only 'knows' the complexity of the signal progressively, wastes too much bits with the first sections of the movie or the audio, there'll less space for the later part. And, if the codec is conservative at the first part 'waiting' for more complex signals, maybe at the end it'll never get that complex signals so, again, bits are wasted.
So, at the first pass the encoder only analyzes the complexity of the signal (therefore the first pass is much faster than the second) for a correct 'bit distribution' at the second pass.
I think the method discused here for Vorbis is other, that is in relation with Huffman compression (lossless).
Excuse me if this explanation is poor technically speaking, or incorrect. Maybe someone can explain it better than me .
@amp: Your explanation is perfectly understandable to me.
The other comment about 2 pass and quality, since most people concerned with quality use the quality settings, and these use VBR, the encoder grabs whatever bits it needs to achieve the desired quality level in the first place.
In video, double pass is not used for saving space, but to maximize quality for a given size/bitrate. I think it should be the same for audio.
Actually in video, the main purpose of 2-pass is to obtain a target output filesize after encoding while keeping quality consistent through the entire video. The one-pass method of obtaining a target output filesize (CBR) gives too many bits to the easy scenes and too few bits to the hard scenes therefore quality noticably suffers in the hard scenes. 2-pass allows target filesize to be reached while maintaining a consistent quality.
In video, a target output filesize is often desired because people often rip DVDs to CDRs and target the maximum capacity of 1 or 2 CDRs. In audio, 2-pass is not really in demand because it would be very rare that an audio ripper would require a target filesize for the encoding.
Ogg Vorbis constant quality single pass will already give you the best quality the codec can give you for its filesize (not counting the use of rehuff), it is just that the output filesize is unpredictable. Video encoding can also be done with a constant quality by encoding with a constant quantizer (option available for xvid and divx4/5), as usual, filesize is unpredictable, but this is the recommended method where you do not need a target filesize.
The other factor why you don't see 2-pass audio encoding is that it's harder to implement than for video. In video, there is only 1 variable (quantizer) which is varied between frames to affect the size of the frame towards a target bitrate. For audio, there are a lot more variables which the encoder will have to consider.
Audio encoding 2-pass has been asked many times before, it really should go into a FAQ. Pio2001, you there?
WMAv9 support 2-pass encoding,too
/me
Going back to john33, saving 3kbs is exactly what it's about You could do it like that, just shaving the file-size down.. you still have the quality defined by q6.25, but in a smaller file size.
If it was the size you were after, you can just encode slightly higher at q6.4, and get 3kbs more of quality added for the file you were expecting
Besides from the various problems reported about rehuff at the Doom9 forums, I'd definetly like tosee this function in oggenc2.1.
But don't call it 2 Pass! As explained above the meaning of 2pass is different from what rehuff does, so adding an option --rehuff might be just fine.
dev0
For those who are interested, oggenc2.2 is now available from Mirror 1.
Additional option is '--rehuff'. It processes quite quickly with a fair amount of info displayed to the console. The '-Q' (quiet) option from the command line will also kill the console output from the rehuff pass.
Have fun!!
it's a big improvement, i hope we'll see this 2 pass mode in oggmachine soon (for now i only use ogg to convert from ac3 files)
thanks
does the ogg vidoe container support the rehuff savings ?
if i use ogmmux to mux avi videi and some vorbis file with rehuff
is the audio still done with the rehuff savings ?
does the ogg vidoe container support the rehuff savings ?
if i use ogmmux to mux avi videi and some vorbis file with rehuff
is the audio still done with the rehuff savings ?
So far as I'm aware, the content and appearance of the rehuffed file to 'ogg aware' software should be no different than for any other ogg file. All the headers are correct, it simply has more efficient compression. I know that this is not an unequivocal answer, but I don't believe it should behave in any different way to any other ogg file.
OK, now I post it here.
I just tried the --rehuff option while ripping and encoding a whole audio CD - nothing special about that.
For ripping I use Exact Audio Copy in conjunction with OggEnc2, command line reads as follows:
-q 6 %s %d -t "%t" -a "%a" -l "%g" -d "%y" -N "%n" -G "%m" --rehuff
This has worked well so far except the --rehuff option.
Now, having tried to encode this CD, I have indeed received files which were smaller than they used to when using OggEnc(1), but unfortunately, seeking and jumping in files has been corrupted.
For playback I use Winamp 2.81 and in_vorbis.dll 1.2.9 latest build.
When trying to seek inside the file to any position after half of the whole file right after starting, playback instantly stops.
Did anybody else track this problem down? It would be interesting if it is an encoder or decoder problem ... and, strangely, it didn't occur on ALL files, just on about 20% ...
Bye
MS
EDIT: OK, now I've found out at the Doom9 Forum that this problem has been encountered in previous releases too ... unfortunately there seems to be no solution, right?
http://forum.doom9.org/showthread.php?thre...ighlight=rehuff (http://forum.doom9.org/showthread.php?threadid=30491&highlight=rehuff)
Now, having tried to encode this CD, I have indeed received files which were smaller than they used to when using OggEnc(1), but unfortunately, seeking and jumping in files has been corrupted.
Yes, I noticed the same problem with all players.
EDIT: OK, now I've found out at the Doom9 Forum that this problem has been encountered in previous releases too ... unfortunately there seems to be no solution, right?
I found a solution: remux the file (not reencode!) with Tobias DirectShow-filters, build a graph in GraghEdit: infile.ogg -> Ogg Splitter -> Ogg Multiplexer -> outfile.ogg
The file has nearly the same size (27 Byte bigger actually) but seeking works perfect again.
does the ogg vidoe container support the rehuff savings ? if i use ogmmux to mux avi videi and some vorbis file with rehuff is the audio still done with the rehuff savings ?
Yes. My test:
without rehuff: 1.458.758 Byte
with rehuff: 1.409.311 Byte (96,610% from the normal file) seeking is broken!
with rehuff and remuxed: 1.409.338 Byte (96,612% from the normal file) file perfectly seekable!
So this problem is not unfixable and not a general result of rehuffing.
Hmmm, I can't comment at this time other than to confirm the same behaviour. The files play fine from start to finish, it's the seeking that screws up.
Yeah, that GraphEdit-Thingo works indeed. Unfortunately, this procedure is way too complex to apply it to a whole bunch of .OGG files, so I do hope this issue can be handled very soon ....
Thanks John, we appreciate your effort!
BTW, I was just trawling through the vorbis-dev mailing lists and came across a comment by Segher Boessenkool, the author of rehuff, referring to a bug in Libogg as being the cause of the problems. The upcoming patch release relates primarily to the ogg side of things, ogg file I believe, so maybe this will enable a proper release of rehuff in the not too distant future.
John,
It seriously makes me bang my head against the wall - you are so bloody helpful!
It's very much a case of "It'd be nice if vorbis did...." and before I've finished the
sentence you've gone and done it. Superb.
I'll have to come to Berkshire and buy lots of beer for you.
Ruairi.
Yesterday I downloaded oggenc2.2 and tested it using the --rehuff switch; it seems that skipping and searching within a track is working correctly now.
My filesize savings due to using --rehuff came to about 1 - 2 percent.
Are there plans to make available an ICL7 compile optimised for Pentium 4?
My thanks to john33!
Yesterday I downloaded oggenc2.2 and tested it using the --rehuff switch; it seems that skipping and searching within a track is working correctly now.
My filesize savings due to using --rehuff came to about 1 - 2 percent.
Are there plans to make available an ICL7 compile optimised for Pentium 4?
My thanks to john33!
No sooner requested, than done! You'll find it at Mirror 1.
I see ICL7, optimised for Pentium 4 links for oggenc, oggenc2.1 but not for oggenc2.2.
I figured it out as being
http://homepage.ntlworld.com/jfe1205/oggenc2.2P4.zip (http://homepage.ntlworld.com/jfe1205/oggenc2.2P4.zip)
Thanks! You da man!
Great work! Now I can increase quality of my files
and my friends will still be able to stream from me!
Another question... are OggEnc daily and OggEnc 2.2 very different?
Yesterday I downloaded oggenc2.2 and tested it using the --rehuff switch; it seems that skipping and searching within a track is working correctly now.
You must be lucky and using special input which doesn't produce flaws in the resulting output file. filestamp still says 2003-01-01, and so my files still lack seekability.
try encoding a standard cd track @ 44.1 kHz, 16Bit stereo, and you will encounter the same problems as mentioned above.
therefore i don't quite see why it would make sense to provide a icl7 compiled version.
bye
MS
I see ICL7, optimised for Pentium 4 links for oggenc, oggenc2.1 but not for oggenc2.2.
I figured it out as being
http://homepage.ntlworld.com/jfe1205/oggenc2.2P4.zip (http://homepage.ntlworld.com/jfe1205/oggenc2.2P4.zip)
Thanks! You da man!
You're right, sorry, I was in too much of a hurry!! Now corrected.
it's a big improvement, i hope we'll see this 2 pass mode in oggmachine soon (for now i only use ogg to convert from ac3 files)
thanks
it used to be there long time ago (august 2002 (http://www.xiph.org/archives/vorbis-dev/200208/0037.html)), but Segher asked me to remove it.
here's a snap from our discussion :
Greetings Segher,
i've understood you're not satisfied with the fact that BeSweet offers ReHuffing of Ogg Vorbis streams.
:
:
i would like to keep offering this feature, but would remove it if you want me to. no problem.
Please remove it.
please clarify your position on this issue.
When i think the code is ready for public consumption, i'll release it for general use (and for free).
Cheers,
Segher
Great work! Now I can increase quality of my files
and my friends will still be able to stream from me!
Another question... are OggEnc daily and OggEnc 2.2 very different?
oggenc2.2 differs in 3 ways:
1. Providing you have the necessary decoders in 'the path', you can specify lossless encoded files for direct input.
2. You can specify 'padding' in the comment header to be used later for storing tags.
3. You can specify a second pass using the 'rehuff' code which will compress the encoded file with a little more efficiency, ie. make it smaller, without any effect on the quality. Beware, though, this is experimental code and no responsibility is taken for the results!!
Sorry folks, I must have been spacing out yesterday. Today the .ogg files encoded with the --rehuff option are definitely NOT responding well to search and skip within a file. I can succeed with short skips only, and then only about 3 or 4 skips; after that the search slider jumps back to the start and the track stops playing or the player moves to the next one on the playlist. This is with Winamp 2.81 with the 1.3 alpha input plugin. Yesterday, I could have sworn it was working...
Even worse results here... foobar2000 consumes 100% cpu when I try to seek or at the end.
Probably you should remove ALL Vorbis headers after rehuffing then add them again at correct positions.
This should fix seeking/positioning info. I think it breaks because frames change their size unevenly
and most players rely on frame size for seeking... if they get wrong enough info (eg. get wrong header)
they'll either skip the file or crash.
<edit>
This does NOT seems like a bug in OggEnc, but rather in players/libvorbis.
</edit>
This does NOT seems like a bug in OggEnc, but rather in players/libvorbis.
Sorry, but it indeed does look more like an encoder than an decoder problem. When applying the Graphedit-De-Remuxing procedure as described above, everything works fine and the headers seem to be written in the correct way. (filesize increases by some bytes)
Therefore I agree with you that some not direct-audiodata-related file information tags are not written correctly by oggenc2.2. probably frame/header information - maybe the dev-update also mentioned above will fix it.
bye
MS
Libvorbis should recover from such error properly...
Does anybody have a player using decoder other than libvorbis?
If something is not written properly, it could be only last frame.
So problem is in eof() function in libvorbis.
It seems to need padding present in the last frame...
Rehuffing compresses this information.
Remuxing helps because it pads the last frame.
thanks DSPguru, i hope we will see again this 2 pass mode in oggmachine soon
take care
<edit>
This does NOT seems like a bug in OggEnc, but rather in players/libvorbis.
</edit>
Rehuff writes incorrect files (a known problem), so it definetely is an encoder problem.
Yes, but are they REALLY incorrect?
I'll check OGG and Vorbis specs and see if this padding is needed
or only 'recommended' or 'might be implemented'.
If it's the second case, then decoder is wrong.
The only wrong call is eof() function, which seems to be used by seek()
<edit>
From Ogg Vorbis I format specification: bitpacking convention
end-of-packet alignment
The typical use of bitpacking is to produce many independent byte-aligned packets which are embedded into a larger byte-aligned container structure, such as an Ogg transport bitstream. Externally, each bytestream (encoded bitstream) must begin and end on a byte boundary. Often, the encoded bitstream is not an integer number of bytes, and so there is unused (uncoded) space in the last byte of a packet.
Unused space in the last byte of a bytestream is always zeroed during the coding process. Thus, should this unused space be read, it will return binary zeroes.
Attempting to read past the end of an encoded packet results in an 'end-of-packet' condition. End-of-packet is not to be considered an error; it is merely a state indicating that there is insufficient remaining data to fulfill the desired read size. Vorbis uses truncated packets as a normal mode of operation, and as such, decoders must handle reading past the end of a packet as a typical mode of operation. Any further read operations after an 'end-of-packet' condition shall also return 'end-of-packet'.
This is what's probably wrong... the data to skip is rehuffed and zeroes are compressed.
Remuxing restores them. That's why the filesize was different by few bytes.
Decoder (libvorbis) tries to read it and returns end-of-packet.
Winamp handles it correctly (skips), foobar starts to consume 100% CPU...
but after a LONG while it begins to play another file!
It's a bug in foobar, not libvorbis!!!
Nonetheless this data should be restored after rehuffing, it should be quite easy to do.
Just fill the end of the last packet with zeroes where needed after rehuffing.
</edit>
Comment from Segher, the author of the rehuff routines, extracted from the vorbis-dev lists:
The best way to speed up its release is fixing the bug in libogg that
prevents it from writing valid streams, btw...
Segher
This dated 08 Nov 2002.
Reread my post, I stated what's probably incorrect there.
<edit>
Reread it myself... I'm most probably wrong.
</edit>
What does remuxing exactly do to the track that makes it work?
Someone could duplicate it in oggenc if rehuff is used...
This IS NOT a fix, but rather a proposed workaround.
Oh hell, I don't have the tools and Tobias' page has exceeded bandwith limit...
Anybody can post here two very small files, one only rehuffed and second rehuffed then remuxed?
Look at this...
OGG Rehuffed (http://www.bedeox.devisland.com/OGGHuffed.PNG)
OGG NOT Rehuffed (http://www.bedeox.devisland.com/OGGNonHuffed.PNG)
This dot is FF... in Rehuffed there are 11 of them, in Non-rehuffed there are 14.
There's one more bug to report with oggenc 2.2:
when encoding with command line a standard 44.1 kHz, 16 bit, stereo file and "on the fly" downmixing it using the --downmix option, everything works just fine.
additionally using the --rehuff option makes the encoding process crash: the hufftmp-file remains and there is no data in the output file although no error reports.
bye
MS
Rehuff does not work with mono files IIRC.
Rehuff does not work with mono files IIRC.
It falls over if the 'residue type' is not = 2. Is that the same thing??? That's point at which it falls over in the rehuff code.
Rehuff does not work with mono files IIRC.
It falls over if the 'residue type' is not = 2. Is that the same thing???
Yes, because Vorbis 1.0 does not use residue2 for mono files.
(I think it can't handle floggy either for similar reasons)
Will somebody post here rehuffed and rehuffed/remuxed sample, preferably short?
Or do a binary compare... and give here the results!
<edit>
Just add a safeguard against residue 2 until it's fixed.
</edit>
I did follow the announcement about Rehuff back in 2002, but I think development was dropped because of 'flags' being set incorrectly (whatever that means ) and there are some bugs in one of the vorbis files that would have to be fixed before Rehuff could be released properly. I was reminded about rehuff when I was encoding a couple of files to vorbis earlier. I didn't think it was ever properly released still although vorbis 101 has been out for a while. I wonder if it ever will be?
I do have an early binary of rehuff on a CD that someone had put up back in 2002 but it is not an official release. It saves about 3% from what I worked out. I think it just looks at Huffy compression and reoptimises it (effectively losslessly to the sound I think)
If anyone knows more about Rehuff and whether it is still being worked on and will ever be released I'd be grateful to know, thanks
yeah, I'd like to know too
Please add 2-pass Vorbis support!
Up!
Ogg Vorbis constant quality single pass will already give you the best quality the codec can give you for its filesize
Yes, but if i want the best quality (in VBR) for a given filesize ?
I think that a bi-directional 2-pass MT encoder would be great.
2pass encoding is only usefull to speed up the process of getting the most quality out of a given file size (I say 'speed up' because trial & error is the alternative method).
I have yet to see any circumstance where 2pass audio encoding is more usefull than 1pass vbr
Does anybody know of any at all?
For video encoding 2pass is very usefull, since the trial & error method is extremely time consuming.
What about listening tests where a specific average bitrate for the collection of samples may be desired?
I have yet to see any circumstance where 2pass audio encoding is more usefull than 1pass vbr
Does anybody know of any at all?
Rehuff can be used only with a two pass encoding (altought this is not the same two pass of video encoding, this is really a two pass).
More infos on rehuff:
http://lists.xiph.org/pipermail/vorbis-dev...ust/018522.html (http://lists.xiph.org/pipermail/vorbis-dev/2006-August/018522.html)