Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Streaming Ogg Packets (Read 28678 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Streaming Ogg Packets

Reply #25 –
Quote from: chris319 on
That the code streams mp3 continuously tells me something. The mystery is why the same code (with the audio/opus mime type) doesn't stream opus continuously. I've tried encoding both wav and raw files with the same result.

If the MP3 stream generates all pages (or whatever chunks you are streaming with) smaller than 4500 bytes (eg 4096) and opusenc generates pages larger than 4500 bytes (which it can and does) and your code truncates chunks at 4500 bytes, then that would be perfectly reasonable.  Doesn't explain why Firefox can play anything at all from that particular file though (and it doesn't here, up to Firefox 29).  I suspect the process you used to save the stream into a file damaged it.

Streaming Ogg Packets

Reply #28 –
test.opus and Tone.opus are perfect so far as I can tell.  OpusTestStreamChris is badly damaged, full of holes, probably truncated on every page.

Streaming Ogg Packets

Reply #29 –
Quote from: lithopsian on
test.opus and Tone.opus are perfect so far as I can tell.  OpusTestStreamChris is badly damaged, full of holes, probably truncated on every page.

I should qualify that.  There seems to be some unusual padding on the OpusTags page.  None of the tools are complaining about it and the file plays OK, but it looks wrong to me.

Streaming Ogg Packets

Reply #30 –
Does this testing take into account that OpusTestStreamChris is not a contiguous ogg file, but rather several ogg files joined together? Did it play back just one second of tone or more than that? Is ogginfo able to discern valid and different serial numbers?

Here is another file to test. If all went well it should be an exact copy of tone.opus.

www.gameshowforum.org/audio/OpusTestFile

Thanks in advance for all of this testing, BTW.

Streaming Ogg Packets

Reply #31 –
Several Ogg files joined together *is* a contiguous Ogg file.  It is known as a chained file and is a perfectly legitimate Ogg container.  You don't have to have any special markers, separators, or structure.  Literally you just concatenate them together and it forms a file which can be played through exactly as if it was one Opus track, or you can treat it as a number of separate tracks since each link in the chain has its own header and can have different comment tags.

There are a couple of gotchas that I've run into, though.  If you concatenate together two files that contain Opus streams with the same serial number, for example two copies of the same file, then most tools will reject at least one of those links in the chain.  In my experience the first link with that serial number plays OK, but the second fails.  Streaming (ie. unsekkable) content tends to be a little more tolerant because it simply doesn't know what the serial number of every track in a (theoretically) infinite file was.  It can run into trouble at the boundary between two tracks with the same serial number though, because it can be tricky to know which page is from which track (not impossible though, and some tools will just play it regardless).

Streaming Ogg Packets

Reply #32 –
The first thing to note with your new file is that it clearly isn't an exact copy of Tone.opus.  It is a different size for a start

I notice there are no HTTP headers at the start now.  That is probably a good thing.

Side by side comparisons are so revealing.  The first difference I see is that the VorbisComment packet in OpusTestFile is corrupt.  The vendor length indicator following the OpusTags magic string is only 3 bytes and should be four.  In Tone.opus there are four bytes, the first of which is 0x0d indicating the 13 byte length of the vendor.  In addition the checksum for the page is identical in both files despite the content of the page being different.  This in itself would make most decoders reject the file, but lets assume they can recover ...

The first audio page has identical segment tables in each file but the contents of the page differs.  OpusTestFile is again missing apparently random bytes.  I haven't a clue how that would be happening but presumably something in your code.  Missing bytes would be reported as holes by most tools and that's what they report.  Although opusinfo is reporting the holes at exact multiples of 4500 bytes, that might just be an artefact of its analysis.  The missing bytes that I have spotted are randomly all over the page.

Streaming Ogg Packets

Reply #34 –
Same thing.  Random missing bytes including one in the OpusHead page that just invalidates the whole file.  Does Firefox actually play that file for you?  Firefox here says resource could not be decoded.

Streaming Ogg Packets

Reply #36 –
Quote from: chris319 on
Here we go again. No headers, just a copy of the tone.opus file generated by my program.

www.gameshowforum.org/audio/OpusTestFile


Worse.  It's tiny!  Are you sure this is supposed to be a copy of Tone.opus, a 17k file?  It is 1500 bytes exactly and the first audio page segment table is for a short page, not at all like Tone.opus which has fairly standard 4k-ish pages.  The OpusTags header is corrupt in the same place though.  Fewer errors are reported by opusinfo, but there in a lot less file to report them on.

A good sanity check would be that what gets streamed should be identical to the original file.  Literally, a bit-perfect copy.  If I was to stream an Opus file and save it then I should end up with exactly the same file that you started with.  However your code is sending Tone.opus, this is not happening.  HTTP headers should not affect this because they shouldn't be included in the file, rather they get sent as part of the transport.

Streaming Ogg Packets

Reply #37 –
Sorry about that. Tone.opus was shortened and is different from the file you have a link to. That's why you're seeing a shorter file.
 

Streaming Ogg Packets

Reply #38 –
I'm going to have to write a short program to compare the original ogg file with my copy of it.

Streaming Ogg Packets

Reply #39 –
OK, but still missing bytes even in the short file.  From streaming or from saving the stream to a file?  Chopping off the end of each chunk you stream or read?

Streaming Ogg Packets

Reply #40 –
The buffer that gets streamed is the same buffer that gets written to a file. What I need to do is make a faithful copy of a known good opus file. Why I am having so much difficulty doing this is the big mystery. Nothing is being chopped off by my code, in theory. There could be a bug in PureBasic, too. These missing bytes occur at random? Doing a straight read of the buffer and writing it to disk shouldn't result in random missing bytes.

It occurs to me that I may still have Linux on an old USB HD. If I do, I can try running ogginfo on these files myself.

Streaming Ogg Packets

Reply #41 –
hexdump can be fun too.  Or hexedit on Windows?  You can usually see the first missing byte in the headers, so in the first few hundred bytes.  Interesting to learn the Ogg page and Opus header formats.

Streaming Ogg Packets

Reply #42 –
OK, I got Linux up and running again. I ran ogginfo on an opus file encoded by opusenc. There was no modification to the file. This is the direct, unmodified output of opusenc:

Code: [Select]
Processing file "Tone.opus"...

New logical stream (#1, serial: 00007275): type unknown
Logical stream 1 ended


Now here is the output of ogginfo with an ogg vorbis file created by Goldwave:

Code: [Select]
New logical stream (#1, serial: 000010e9): type vorbis
Vorbis headers parsed for stream 1, information follows...
Version: 0
Vendor: BS; LancerMod(SSE) (based on aoTuV [20110424])
Channels: 1
Rate: 44100

Nominal bitrate: 239.919000 kb/s
Upper bitrate not set
Lower bitrate not set
Vorbis stream 1:
    Total data length: 3935733 bytes
    Playback length: 2m:12.199s
    Average bitrate: 238.168411 kb/s
Logical stream 1 ended


Here is OpusTestFile:

Code: [Select]
Processing file "OpusTestFile"...

New logical stream (#1, serial: 0000095a): type unknown
Logical stream 1 ended


ogginfo doesn't handle opus files correctly? I'm assuming the output of opusenc is valid.

Streaming Ogg Packets

Reply #43 –
Now I've got opus-tools installed. opusinfo doesn't report any problems. The data reported by opusinfo all looks reasonable.

Streaming Ogg Packets

Reply #44 –
opusinfo is correct.  Your ogginfo understands the file structure but doesn't recognise the Opus packet type.

Streaming Ogg Packets

Reply #45 –
Several opus packets joined together don't pass muster with opusinfo.

I'm stuck. I don't know what else to do to form a continuous stream.

I'm not modifying the opus files themselves, just writing/streaming them one after the other.

Streaming Ogg Packets

Reply #46 –
What does it say about the joined file?  Does it have HTTP headers at the start (I don't think it should).  Doeas it complain about holes?  Invalid headers?  Unrecognised format?

Streaming Ogg Packets

Reply #47 –
Individual packets don't give opusinfo any trouble. It's when I try to chain them together that the trouble begins. I can give you a more detailed report later.

There are no HTTP headers after the first packet. It's complaining of holes in the data, that kind of thing. I don't think you can simply butt opus packets together, because that's what I've been doing and opusinfo reports trouble with that. When I try it with HTTP headers between packets, still no joy.

The opus streams I've tried from xiph seem to use oggfwd and icecast. Oggfwd is the missing link but oggfwd is not available for Windows, so that's a problem. I'm just using PureBasic's networking functions, but apparently more is required than that. I'm not even sure this can be done in Windows.

Why don't I set up my streamer under Linux? Well, there's an added complication. My streaming program uses PortAudio. I found out the hard way about a year ago that PortAudio does not play well under Linux but plays much better under Windows. So Windows is the preferred platform.

Doing low-level manipulation of the opus packets myself a la oggfwd is not really viable.

Streaming Ogg Packets

Reply #48 –
You can't just butt packets from different streams together, that's true, but that's now what you're doing.  You're butting together entire Opus files, which is entirely legal.  In fact that's all a chain is.  If you take two Opus files and concatenate them together then you have a chain and it will play in most payers, including recent versions of Firefox.

You have some other issue.  I still haven't seen a file that you have streamed that looks anything except corrupt.  Many players will play past holes, and if they are small (like the one byte holes I see in your files!) it may well not be audible.  Holes are essentially a page number out of sequence in the Opus stream, which might be caused by a page number being out of sequence, but is more commonly caused by a small corruption which causes the stream to lose sync.  When sync is recovered one or two pages later, the decoder knows something has gone missing and reports a hole.

The streamer programs are convenient ways to manage whole bunches of files, insert your own station tags, etc. but don't really do a great deal more than send out a file with a mime type.  In Ogg file formats. the track tags are managed within the files so you don't even need s special streamer for that.

Streaming Ogg Packets

Reply #49 –
Quote
You can't just butt packets from different streams together, that's true, but that's now what you're doing.


Did you mean to use the word "not" in that sentence instead of "now"? If so, it changes the meaning entirely.

I have opusinfo running on Windows now so it's a lot easier. Here is what I get writing just one opus file. No warnings, no errors:

Code: [Select]
Processing file "OpusTestStreamChris.opus"...

New logical stream (#1, serial: 00000001): type opus
Encoded with libopus 1.1
User comments section follows...
        ENCODER=opusenc from opus-tools 0.1.8
        ENCODER_OPTIONS=--serial 1 --quiet --comp 0 --ignorelength
Opus stream 1:
        Pre-skip: 316
        Playback gain: 0 dB
        Channels: 1
        Original sample rate: 44100Hz
        Packet duration:   20.0ms (max),   20.0ms (avg),   20.0ms (min)
        Page duration:   1000.0ms (max),  510.0ms (avg),   20.0ms (min)
        Total data length: 9951 bytes (overhead: 9.51%)
        Playback length: 0m:00.999s
        Average bitrate: 79.61 kb/s, w/o overhead: 72.04 kb/s
Logical stream 1 ended


Here is what I get writing things twice to the same file. It's the same code executed twice, same input file and all. Just butting two files together. No HTTP headers anywhere.
Code: [Select]
Processing file "OpusTestStreamChris.opus"...

New logical stream (#1, serial: 00000001): type opus
Encoded with libopus 1.1
User comments section follows...
        ENCODER=opusenc from opus-tools 0.1.8
        ENCODER_OPTIONS=--serial 1 --quiet --comp 0 --ignorelength
Opus stream 1:
        Pre-skip: 316
        Playback gain: 0 dB
        Channels: 1
        Original sample rate: 44100Hz
        Packet duration:   20.0ms (max),   20.0ms (avg),   20.0ms (min)
        Page duration:   1000.0ms (max),  510.0ms (avg),   20.0ms (min)
        Total data length: 9951 bytes (overhead: 9.51%)
        Playback length: 0m:00.999s
        Average bitrate: 79.61 kb/s, w/o overhead: 72.04 kb/s
Logical stream 1 ended
WARNING: illegally placed page(s) for logical stream 1
This indicates a corrupt Ogg file: Page found for stream after EOS flag.
WARNING: sequence number gap in stream 1. Got page 1 when expecting page 4. Indi
cates missing data.