HydrogenAudio

Lossy Audio Compression => Ogg Vorbis => Ogg Vorbis - Tech => Topic started by: chris319 on 2014-04-20 15:05:01

Title: Streaming Ogg Packets
Post by: chris319 on 2014-04-20 15:05:01
I am writing a program to stream audio via TCP to a browser. I am encoding buffers of PCM audio to a compressed format and streaming the encoded files one by one to the browser which then plays them back one after the other. This scheme works fine if I use mp3 files. The browser plays back the individual packets one after the other for a continuous stream.

The problem arises when I try to stream ogg files, either ogg/vorbis or ogg/opus. The browser (Mozilla Firefox) plays back the first file and stops. It does not accept any more packets. It's behaving as if the network connection has been dropped. I am using HTML 1.1 so it is a keep-alive connection by default.

What's puzzling is that, using the exact same code, mp3 files are played back as a continuous stream while ogg files cause the browser to quit and possibly drop the connection after the first packet.

Why is this so? Am I taking the entirely wrong approach to streaming? Is there something about an ogg file that would trigger this behavior? Is it a bug in the browser? I have tried myriad combinations of header fields and have tried sending the data chunked, to no avail. I have examined the headers of an opus demo stream and can't find anything they're doing differently with the headers.
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-04-20 17:55:56
Have you tried any other browsers?  Also, which version of Firefox?

I think this used to be standard Firefox behaviour, but has been fixed since Firefox 20.  Ogg streams effectively are multiple files chained together continuously, whereas MP3s really are streaming continuously with the server perhaps throwing out different metadata from time to time.  I just tested and Firefox 17 stops after each track.  Firefox 29 plays continuously.

So your code may be fine.  Try streaming direct to a music player that does this properly.
Title: Streaming Ogg Packets
Post by: chris319 on 2014-04-21 09:51:50
This is Firefox v28.0.

I can't imagine any other way of doing LIVE streaming than by sending individual encoded buffers/packets, or what some call progressive download.

How are you playing back these tracks? Are they packets in a live stream or separate song tracks?
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-04-21 17:54:56
Firefox 28 should be able to play a chained Ogg stream continuously.  You can test your browser against one of these streams (http://dir.xiph.org/by_format/Opus).  Note that not all of them make track changes.  A track change in an Ogg stream is effectively a separate file, although the packets are sent continuously.  As you probably know if you wrote it  The track change interrupts the continuous flow of audio packets and send some header packets describing the new track, then back to the flow of audio packets.  I'm not sure what you would break doing this, but perhaps something is getting truncated (or duplicated?) at the file switchover.

Do you have an example I can connect to and examine?
Title: Streaming Ogg Packets
Post by: chris319 on 2014-04-21 19:19:50
The streams you linked to play fine on Firefox 28.0. The question is, what are they doing differently to make them stream continuously? As mentioned previously, I am able to stream mp3 files continuously but not ogg files. I have copied the HTTP headers from those streams into my code and it didn't solve the problem. The fact that those streams play fine suggests that there isn't a problem with the browser but perhaps with the way the files are encoded or the way I'm sending them?

I have tried sending header - file - header - file, and header - file - file - file but no matter -- I never make it past the first header and file pair. The browser plays the first ogg file and thinks it's done and accepts no more data, suggesting the connection may be broken.

By "header" I mean HTTP header.

Here are the HTTP headers from my code. They were copied from the headers linked to above. #CRLF$ is a carriage return/line feed string constant.

header$="HTTP/1.1 200 OK"+#CRLF$
header$+"Server: 127.0.0.1:8000" + #CRLF$
header$+"Content-Type: application/ogg"+#CRLF$
header$+"Cache-Control: no-cache"+#CRLF$
header$+"icy-pub: 1"+#CRLF$
header$ +#CRLF$
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-04-22 11:32:25
Correct content-type is audio/ogg.  No matter, almost everyone including Firefox accepts application/ogg.

I don't think the HTTP headers are the problem.  More likely you are transmitting something between the EOS packet and the BOS packet of the next link.  There isn't really anything fancy to worry about.  So long as Firefox (or any other player) gets a page marked EOS and immediately gets a page marked BOS then it should keep playing.

Serial numbers must be valid, but if you have a valid Ogg chain then that should be true already.  Are you streaming a bunch of separate files or from a single Ogg chain file?  Might be worth making a chain file and streaming that.  Just concatenate a bunch of individual tracks togther and that's a chain.  Make sure it plays as a file, then stream it.
Title: Streaming Ogg Packets
Post by: chris319 on 2014-04-22 12:01:13
I concur that it's probably not the headers. Again, it works fine with mp3 files. The application/ogg was copied directly from the HTML5 opus streams you linked to, but it really should be audio/ogg.

I emphasize that this is a LIVE stream so the individual audio buffers are encoded in real time on the fly. I don't see how you could concatenate them into a single chain file. You would have to keep building these concatenated files and that's more overhead. I am not familiar with ogg BOS and EOS so I will have to research those.
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-04-22 12:16:18
OK, LIVE stream.  Build straight from PCM?  Or equivalent?  Doesn't matter I suppose.

So what is your definition of the end of a track?  An Ogg stream consists of a BOS page, which is a header defining the stream, then a series of pages.  Or packets if you like.  The first packet must be a comment packet.  In Vorbis the second packet must be a codebooks packet, but Opus doesn't have that.  Then audio packets with no special markings until the last one, which is marked EOS (end of stream).

I assume you must be creating the header packets somehow if you are encoding live?  Are you doing this yourself or calling a library?  What happens when you decide it is time to change tracks?  Are you coding the EOS page correctly?  Are you then coding a new set of headers correctly?  You can stream standard audio packets forever if you want, but a track change must have the defined EOS and BOS and other header packets or it will be considered as an EOF/error by the receiver.  Without the correct header, which contains things like channel info and serial number for the stream, subsequent audio packets can't be understood.
Title: Streaming Ogg Packets
Post by: chris319 on 2014-04-22 16:47:41
Here is the basic scheme:

1. PCM data from sound card/interface

2. write PCM data to temporary raw file

3. encode raw file to opus using opusenc

4. send encoded file on network (localhost for testing)

Running opusenc with its options is the extent of my knowledge of opus encoding. Do I need to manipulate the serial number when encoding?

The opus documentation mentions oggfwd but this does not seem to be an option for non-Linux setups (running Windows 7). I am not using Icecast for this.

https://mf4.xiph.org/jenkins/view/opus/job/...an/opusenc.html (https://mf4.xiph.org/jenkins/view/opus/job/opus-tools/ws/man/opusenc.html)

The end of the stream would occur when either my program or the browser closes the network connection.
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-04-22 17:32:26
opusenc should take care of everything for you, including all the serial numbers.  It outputs a mass of data which can either be stored as a file or streamed directly, no difference in the format.  You might try outputting a few streams into files instead of sending them over the network.  Then I could examine the results.  Or if you could log the stream that you are receiving.

I'm not sure what tool might be available that would give you more information about what is happening on your stream when the track changes.  Firefox doesn't really say anything useful.  On Linux I could offer ways to get some information at least that might tell you why the stream breaks.

I'm still not clear what your program is going at a track change.  Does it do anything that just streaming the last byte opusenc sends from one track and then the first byte that it sends for the next track?  Is there a pause between availability of the TCP packets for the two streams?
Title: Streaming Ogg Packets
Post by: chris319 on 2014-04-22 22:02:58
Here is some actual code written in PureBasic. It is executed in a loop.

It works. The first packet plays OK, then the browser stops accepting data.

Code: [Select]
Procedure AudioLoop()
Pa_ReadStream(*my_stream, *writeBuffer, framesPerBuffer)
CreateFile(10, "temp.raw"):
WriteData(10, *writeBuffer, framesPerBuffer*bytesPerFrame): CloseFile(10)

RunProgram("\Programs\Opus\opusenc.exe" , "--max-delay 0 --quiet --comp 0 --raw-rate 44100 --raw-chan 1 temp.raw  temp.opus", "./", #PB_Program_Wait)

header$="HTTP/1.1 200 OK"+#CRLF$
header$+"Server: 127.0.0.1:8000" + #CRLF$
header$+"Content-Type: audio/ogg"+#CRLF$
header$+"Cache-Control: no-cache"+#CRLF$
header$ +#CRLF$

ReadFile(10, "temp.opus"): ReadData(10, *writeBuffer+Len(header$), leng): : CloseFile(10)

If ClientID <> 0

SendNetworkString(ClientID, header$)
result = SendNetworkData(ClientID, *tempBuffer, leng)

EndIf

EndProcedure


If I join the header and the data and send them together, the result is the same.
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-04-22 23:23:27
I can't imagine that ever working well.  opusenc takes many seconds, maybe even minutes, to run.  You will have that gap between your tracks and I suspect most players will just hang up.  Streams have to stream  opusenc can stream, or at least it can on Linux.  I imagine there is a similar solution where you can take the data direct without an intermediate file.  Or maybe the browser would actually wait patiently for the next track to arrive?

Seems like you also send the (HTTP) header before every file.  I don't think Firefox will like that.  You already told it what it is getting, now you just send data forever.  It would be fine the first time, but I think sending it again will break the playback.

Last thing I'm a little confused about.  You have tempbuffer.  Is that a typo or do I not understand the Basic syntax.  And you put the Ogg data into writebuffer after a gap (as long as the HTTP header, but is the HTTP header actually in there?)  Again I might be misunderstanding the language, but it seems like that is making a mess between tracks.

Probably some misunderstandings of the Basic from me, but that should give you a few ideas of where to look for issues.
Title: Streaming Ogg Packets
Post by: chris319 on 2014-04-23 01:22:17
writeBuffer is where PortAudio puts the PCM data. It gets written to a file which is the file opusenc encodes. The opus file is read into tempBuffer which is sent to the network.

I'm prepared for there to be gaps, but haven't gotten that far yet.

I don't know of a better way of doing this without using Linux and oggfwd. I don't know of a non-disk solution that would work with Windows.

I'll try again but I think the mp3 version needed a header before every packet.

For testing purposes, I can open a pre-made file containing a sine wave, read the data to writeBuffer, encode that and send it, rather than getting the audio data from PortAudio.
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-04-23 14:07:59
Is the opusenc call async?  Can it run in the background while the program streams from the file it is writing?  Maybe it actually does that already?  Maybe worry about that once other issues are addressed.  I think there are definite problems with the way the HTTP headers are being sent and the way the writebuffer is populated.  Garbage at the start of a stream is generally tolerated and receivers will just seek (or read) through until they find a valid Ogg stream, but between streaming tracks they aren't so tolerant.
Title: Streaming Ogg Packets
Post by: chris319 on 2014-04-23 15:25:37
You've seen the code and I've said the program works with mp3 files. Why a browser stops accepting data after the first buffer/packet is the mystery. I'm not going to worry about the rest of it until that mystery is solved.
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-04-23 16:40:05
You've seen the code and I've said the program works with mp3 files. Why a browser stops accepting data after the first buffer/packet is the mystery. I'm not going to worry about the rest of it until that mystery is solved.

Stop putting garbage between the end of one track and the start of the next.  Saying it works for MP3 is irrelevant.  Don't do it for Ogg, then see where to go from there.  If you get a significant pause between the last packet from one track and the first packet from the next, then that will also need to be addressed.
Title: Streaming Ogg Packets
Post by: chris319 on 2014-04-23 16:59:27
So you're saying to send the header only once. OK, I'll try it.
Title: Streaming Ogg Packets
Post by: chris319 on 2014-04-24 09:12:00
OK I tried sending the HTTP header only once, then sending successive packets. No change. Firefox plays through the first packet and quits.
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-04-25 11:42:24
OK I tried sending the HTTP header only once, then sending successive packets. No change. Firefox plays through the first packet and quits.

Seems like you want me to psychically spot the bug in your program.  I've offered you several possibilities and several avenues for investigation.  You ignored all that and didn't even try to fix any of the obvious bugs I pointed out until pressed.  Just what is it you want me to do if you won't do anything yourself?  Have you examined the generated stream to see if there is garbage between the tracks (still my #1 guess at the problem).  Have you even tried to log the stream so I can examine it and save you the trouble?  Have you tried feeding the output to a tool like ogginfo which will tell you what is wrong with it?
Title: Streaming Ogg Packets
Post by: chris319 on 2014-04-26 02:22:21
I wasn't familiar with ogginfo, which appears to be a Linux program.

Here is a link to a file with a sample stream if you want to examine it with ogginfo:

www.gameshowforum.org/audio/OpusTestStreamChris
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-04-26 20:47:43
ogginfo and opusinfo *really* don't like that file and neither does anything else.  I'm amazed Firefox would play it.  Are you sure that is an accurate reflection of what you're streaming?

The first obvious thing I see wrong is the serial number.  It is the same for every link in the stream.  That confuses decoders because they can't tell whether a packet is part of a new stream or not.  The tools are also complaining about holes all over the place, basically on every page.  That generally means your page structure is messed up, the segment table doesn't match the actual data segments.
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-04-27 11:46:00
I've looked at the file pages in a little more detail.  Looks like they're all truncated at 4500 bytes?  Does that ring any bells for you?  That doesn't seem like a normal fixed page size that oggenc would output.  I don't see anything in the code snippet that you post, but presumably there is more.  Is one of your buffers too small to contain a page?  An Ogg page can be up to 64k long although more typically it is 4k-16k.
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-04-27 12:10:48
Also checked the standards.  application/ogg used to be the mime type for all Ogg containers.  However the 2008 standards update reserved this for custom Ogg containers and created audio/ogg and video/ogg to be used in the vast majority of cases.  Obviously most tools will still support application/ogg, but in time new ones may not.
Title: Streaming Ogg Packets
Post by: chris319 on 2014-04-27 14:01:36
I've been using audio/opus as the mime type, which I believe is now the standard for opus.

I've reworked my code some and will have another capture of my stream shortly. If the opus file itself lacks integrity, that raises flags. I will also upload two opus files, one generated by Goldwave and the other generated by opusenc which is invoked from my program.

That the code streams mp3 continuously tells me something. The mystery is why the same code (with the audio/opus mime type) doesn't stream opus continuously. I've tried encoding both wav and raw files with the same result.
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-04-27 14:10:28
I've been using audio/opus as the mime type, which I believe is now the standard for opus.


Not really.  The standard says audio/ogg for all predominantly-audio Ogg files, or audio/ogg; codecs=opus if you want to make it obvious that the content is Opus.  Can't say I've seen audio/opus in the wild, but I wouldn't be surprised.  It seems a pretty likely answer if someone was to just guess at a mime type.  I notice that Wikipedia gives audio/opus as an alternative and references an obsolete draft specification.  I might have to get my wiki-hat out of the cupboard ...
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-04-27 14:15:45
That the code streams mp3 continuously tells me something. The mystery is why the same code (with the audio/opus mime type) doesn't stream opus continuously. I've tried encoding both wav and raw files with the same result.

If the MP3 stream generates all pages (or whatever chunks you are streaming with) smaller than 4500 bytes (eg 4096) and opusenc generates pages larger than 4500 bytes (which it can and does) and your code truncates chunks at 4500 bytes, then that would be perfectly reasonable.  Doesn't explain why Firefox can play anything at all from that particular file though (and it doesn't here, up to Firefox 29).  I suspect the process you used to save the stream into a file damaged it.
Title: Streaming Ogg Packets
Post by: chris319 on 2014-04-27 14:26:43
Then audio/ogg it shall be, per RFC 5334: http://www.ietf.org/rfc/rfc5334.txt (http://www.ietf.org/rfc/rfc5334.txt).
Title: Streaming Ogg Packets
Post by: chris319 on 2014-04-27 20:52:36
Here are some new files to look at:

www.gameshowforum.org/audio/OpusTestStreamChris

www.gameshowforum.org/audio/temp.opus

www.gameshowforum.org/audio/Tone.opus
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-04-28 17:09:46
test.opus and Tone.opus are perfect so far as I can tell.  OpusTestStreamChris is badly damaged, full of holes, probably truncated on every page.
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-04-28 19:32:34
test.opus and Tone.opus are perfect so far as I can tell.  OpusTestStreamChris is badly damaged, full of holes, probably truncated on every page.

I should qualify that.  There seems to be some unusual padding on the OpusTags page.  None of the tools are complaining about it and the file plays OK, but it looks wrong to me.
Title: Streaming Ogg Packets
Post by: chris319 on 2014-04-28 22:56:26
Does this testing take into account that OpusTestStreamChris is not a contiguous ogg file, but rather several ogg files joined together? Did it play back just one second of tone or more than that? Is ogginfo able to discern valid and different serial numbers?

Here is another file to test. If all went well it should be an exact copy of tone.opus.

www.gameshowforum.org/audio/OpusTestFile

Thanks in advance for all of this testing, BTW.
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-04-29 15:16:48
Several Ogg files joined together *is* a contiguous Ogg file.  It is known as a chained file and is a perfectly legitimate Ogg container.  You don't have to have any special markers, separators, or structure.  Literally you just concatenate them together and it forms a file which can be played through exactly as if it was one Opus track, or you can treat it as a number of separate tracks since each link in the chain has its own header and can have different comment tags.

There are a couple of gotchas that I've run into, though.  If you concatenate together two files that contain Opus streams with the same serial number, for example two copies of the same file, then most tools will reject at least one of those links in the chain.  In my experience the first link with that serial number plays OK, but the second fails.  Streaming (ie. unsekkable) content tends to be a little more tolerant because it simply doesn't know what the serial number of every track in a (theoretically) infinite file was.  It can run into trouble at the boundary between two tracks with the same serial number though, because it can be tricky to know which page is from which track (not impossible though, and some tools will just play it regardless).

Title: Streaming Ogg Packets
Post by: lithopsian on 2014-04-29 15:32:31
The first thing to note with your new file is that it clearly isn't an exact copy of Tone.opus.  It is a different size for a start

I notice there are no HTTP headers at the start now.  That is probably a good thing.

Side by side comparisons are so revealing.  The first difference I see is that the VorbisComment packet in OpusTestFile is corrupt.  The vendor length indicator following the OpusTags magic string is only 3 bytes and should be four.  In Tone.opus there are four bytes, the first of which is 0x0d indicating the 13 byte length of the vendor.  In addition the checksum for the page is identical in both files despite the content of the page being different.  This in itself would make most decoders reject the file, but lets assume they can recover ...

The first audio page has identical segment tables in each file but the contents of the page differs.  OpusTestFile is again missing apparently random bytes.  I haven't a clue how that would be happening but presumably something in your code.  Missing bytes would be reported as holes by most tools and that's what they report.  Although opusinfo is reporting the holes at exact multiples of 4500 bytes, that might just be an artefact of its analysis.  The missing bytes that I have spotted are randomly all over the page.
Title: Streaming Ogg Packets
Post by: chris319 on 2014-04-30 18:15:40
I found something in my code that didn't look right. I fixed it and the new test file is below. You should see one HTTP header in this one.

www.gameshowforum.org/audio/OpusTestFile
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-04-30 20:22:48
Same thing.  Random missing bytes including one in the OpusHead page that just invalidates the whole file.  Does Firefox actually play that file for you?  Firefox here says resource could not be decoded.
Title: Streaming Ogg Packets
Post by: chris319 on 2014-05-01 04:48:22
Here we go again. No headers, just a copy of the tone.opus file generated by my program.

www.gameshowforum.org/audio/OpusTestFile
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-05-01 14:13:09
Here we go again. No headers, just a copy of the tone.opus file generated by my program.

www.gameshowforum.org/audio/OpusTestFile


Worse.  It's tiny!  Are you sure this is supposed to be a copy of Tone.opus, a 17k file?  It is 1500 bytes exactly and the first audio page segment table is for a short page, not at all like Tone.opus which has fairly standard 4k-ish pages.  The OpusTags header is corrupt in the same place though.  Fewer errors are reported by opusinfo, but there in a lot less file to report them on.

A good sanity check would be that what gets streamed should be identical to the original file.  Literally, a bit-perfect copy.  If I was to stream an Opus file and save it then I should end up with exactly the same file that you started with.  However your code is sending Tone.opus, this is not happening.  HTTP headers should not affect this because they shouldn't be included in the file, rather they get sent as part of the transport.
Title: Streaming Ogg Packets
Post by: chris319 on 2014-05-01 16:30:06
Sorry about that. Tone.opus was shortened and is different from the file you have a link to. That's why you're seeing a shorter file.
Title: Streaming Ogg Packets
Post by: chris319 on 2014-05-01 17:55:50
I'm going to have to write a short program to compare the original ogg file with my copy of it.
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-05-02 01:00:56
OK, but still missing bytes even in the short file.  From streaming or from saving the stream to a file?  Chopping off the end of each chunk you stream or read?
Title: Streaming Ogg Packets
Post by: chris319 on 2014-05-02 10:33:52
The buffer that gets streamed is the same buffer that gets written to a file. What I need to do is make a faithful copy of a known good opus file. Why I am having so much difficulty doing this is the big mystery. Nothing is being chopped off by my code, in theory. There could be a bug in PureBasic, too. These missing bytes occur at random? Doing a straight read of the buffer and writing it to disk shouldn't result in random missing bytes.

It occurs to me that I may still have Linux on an old USB HD. If I do, I can try running ogginfo on these files myself.
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-05-02 12:56:49
hexdump can be fun too.  Or hexedit on Windows?  You can usually see the first missing byte in the headers, so in the first few hundred bytes.  Interesting to learn the Ogg page and Opus header formats.
Title: Streaming Ogg Packets
Post by: chris319 on 2014-05-03 06:14:49
OK, I got Linux up and running again. I ran ogginfo on an opus file encoded by opusenc. There was no modification to the file. This is the direct, unmodified output of opusenc:

Code: [Select]
Processing file "Tone.opus"...

New logical stream (#1, serial: 00007275): type unknown
Logical stream 1 ended


Now here is the output of ogginfo with an ogg vorbis file created by Goldwave:

Code: [Select]
New logical stream (#1, serial: 000010e9): type vorbis
Vorbis headers parsed for stream 1, information follows...
Version: 0
Vendor: BS; LancerMod(SSE) (based on aoTuV [20110424])
Channels: 1
Rate: 44100

Nominal bitrate: 239.919000 kb/s
Upper bitrate not set
Lower bitrate not set
Vorbis stream 1:
    Total data length: 3935733 bytes
    Playback length: 2m:12.199s
    Average bitrate: 238.168411 kb/s
Logical stream 1 ended


Here is OpusTestFile:

Code: [Select]
Processing file "OpusTestFile"...

New logical stream (#1, serial: 0000095a): type unknown
Logical stream 1 ended


ogginfo doesn't handle opus files correctly? I'm assuming the output of opusenc is valid.
Title: Streaming Ogg Packets
Post by: chris319 on 2014-05-03 08:28:43
Now I've got opus-tools installed. opusinfo doesn't report any problems. The data reported by opusinfo all looks reasonable.
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-05-03 13:53:39
opusinfo is correct.  Your ogginfo understands the file structure but doesn't recognise the Opus packet type.
Title: Streaming Ogg Packets
Post by: chris319 on 2014-05-03 14:26:58
Several opus packets joined together don't pass muster with opusinfo.

I'm stuck. I don't know what else to do to form a continuous stream.

I'm not modifying the opus files themselves, just writing/streaming them one after the other.
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-05-03 18:05:10
What does it say about the joined file?  Does it have HTTP headers at the start (I don't think it should).  Doeas it complain about holes?  Invalid headers?  Unrecognised format?
Title: Streaming Ogg Packets
Post by: chris319 on 2014-05-04 01:16:00
Individual packets don't give opusinfo any trouble. It's when I try to chain them together that the trouble begins. I can give you a more detailed report later.

There are no HTTP headers after the first packet. It's complaining of holes in the data, that kind of thing. I don't think you can simply butt opus packets together, because that's what I've been doing and opusinfo reports trouble with that. When I try it with HTTP headers between packets, still no joy.

The opus streams I've tried from xiph seem to use oggfwd and icecast. Oggfwd is the missing link but oggfwd is not available for Windows, so that's a problem. I'm just using PureBasic's networking functions, but apparently more is required than that. I'm not even sure this can be done in Windows.

Why don't I set up my streamer under Linux? Well, there's an added complication. My streaming program uses PortAudio. I found out the hard way about a year ago that PortAudio does not play well under Linux but plays much better under Windows. So Windows is the preferred platform.

Doing low-level manipulation of the opus packets myself a la oggfwd is not really viable.
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-05-04 19:53:36
You can't just butt packets from different streams together, that's true, but that's now what you're doing.  You're butting together entire Opus files, which is entirely legal.  In fact that's all a chain is.  If you take two Opus files and concatenate them together then you have a chain and it will play in most payers, including recent versions of Firefox.

You have some other issue.  I still haven't seen a file that you have streamed that looks anything except corrupt.  Many players will play past holes, and if they are small (like the one byte holes I see in your files!) it may well not be audible.  Holes are essentially a page number out of sequence in the Opus stream, which might be caused by a page number being out of sequence, but is more commonly caused by a small corruption which causes the stream to lose sync.  When sync is recovered one or two pages later, the decoder knows something has gone missing and reports a hole.

The streamer programs are convenient ways to manage whole bunches of files, insert your own station tags, etc. but don't really do a great deal more than send out a file with a mime type.  In Ogg file formats. the track tags are managed within the files so you don't even need s special streamer for that.
Title: Streaming Ogg Packets
Post by: chris319 on 2014-05-05 08:20:54
Quote
You can't just butt packets from different streams together, that's true, but that's now what you're doing.


Did you mean to use the word "not" in that sentence instead of "now"? If so, it changes the meaning entirely.

I have opusinfo running on Windows now so it's a lot easier. Here is what I get writing just one opus file. No warnings, no errors:

Code: [Select]
Processing file "OpusTestStreamChris.opus"...

New logical stream (#1, serial: 00000001): type opus
Encoded with libopus 1.1
User comments section follows...
        ENCODER=opusenc from opus-tools 0.1.8
        ENCODER_OPTIONS=--serial 1 --quiet --comp 0 --ignorelength
Opus stream 1:
        Pre-skip: 316
        Playback gain: 0 dB
        Channels: 1
        Original sample rate: 44100Hz
        Packet duration:   20.0ms (max),   20.0ms (avg),   20.0ms (min)
        Page duration:   1000.0ms (max),  510.0ms (avg),   20.0ms (min)
        Total data length: 9951 bytes (overhead: 9.51%)
        Playback length: 0m:00.999s
        Average bitrate: 79.61 kb/s, w/o overhead: 72.04 kb/s
Logical stream 1 ended


Here is what I get writing things twice to the same file. It's the same code executed twice, same input file and all. Just butting two files together. No HTTP headers anywhere.
Code: [Select]
Processing file "OpusTestStreamChris.opus"...

New logical stream (#1, serial: 00000001): type opus
Encoded with libopus 1.1
User comments section follows...
        ENCODER=opusenc from opus-tools 0.1.8
        ENCODER_OPTIONS=--serial 1 --quiet --comp 0 --ignorelength
Opus stream 1:
        Pre-skip: 316
        Playback gain: 0 dB
        Channels: 1
        Original sample rate: 44100Hz
        Packet duration:   20.0ms (max),   20.0ms (avg),   20.0ms (min)
        Page duration:   1000.0ms (max),  510.0ms (avg),   20.0ms (min)
        Total data length: 9951 bytes (overhead: 9.51%)
        Playback length: 0m:00.999s
        Average bitrate: 79.61 kb/s, w/o overhead: 72.04 kb/s
Logical stream 1 ended
WARNING: illegally placed page(s) for logical stream 1
This indicates a corrupt Ogg file: Page found for stream after EOS flag.
WARNING: sequence number gap in stream 1. Got page 1 when expecting page 4. Indi
cates missing data.
Title: Streaming Ogg Packets
Post by: chris319 on 2014-05-05 08:47:48
OK, now I think I have it. No warnings or errors:

Code: [Select]
Processing file "OpusTestStreamChris.opus"...

New logical stream (#1, serial: 00000001): type opus
Encoded with libopus 1.1
User comments section follows...
        ENCODER=opusenc from opus-tools 0.1.8
        ENCODER_OPTIONS=--serial 1 --quiet --comp 0 --ignorelength
Opus stream 1:
        Pre-skip: 316
        Playback gain: 0 dB
        Channels: 1
        Original sample rate: 44100Hz
        Packet duration:   20.0ms (max),   20.0ms (avg),   20.0ms (min)
        Page duration:   1000.0ms (max),  510.0ms (avg),   20.0ms (min)
        Total data length: 9951 bytes (overhead: 9.51%)
        Playback length: 0m:00.999s
        Average bitrate: 79.61 kb/s, w/o overhead: 72.04 kb/s
Logical stream 1 ended
New logical stream (#2, serial: 00000002): type opus
Encoded with libopus 1.1
User comments section follows...
        ENCODER=opusenc from opus-tools 0.1.8
        ENCODER_OPTIONS=--serial 2 --quiet --comp 0 --ignorelength
Opus stream 2:
        Pre-skip: 316
        Playback gain: 0 dB
        Channels: 1
        Original sample rate: 44100Hz
        Packet duration:   20.0ms (max),   20.0ms (avg),   20.0ms (min)
        Page duration:   1000.0ms (max),  510.0ms (avg),   20.0ms (min)
        Total data length: 9951 bytes (overhead: 9.51%)
        Playback length: 0m:00.999s
        Average bitrate: 79.61 kb/s, w/o overhead: 72.04 kb/s
Logical stream 2 ended
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-05-05 14:33:05
Beautiful!  Does it stream?
Title: Streaming Ogg Packets
Post by: chris319 on 2014-05-05 16:29:30
Yes it does. Looks like this is the winner.
Title: Streaming Ogg Packets
Post by: lithopsian on 2014-05-05 17:45:48
Yes it does. Looks like this is the winner.

Do you know what the problem was?  Just a bug in writing the data?  Or something basic to the Ogg format?
Title: Streaming Ogg Packets
Post by: chris319 on 2014-05-05 19:43:26
Basically it was not knowing how to handle the HTTP headers and chainig ogg files, and each file needing a different serial number.

Thanks again for your help.