rockus.at: Mediatomb and MP101

rockus.at / MP101 / mediatomb.html

 

The story so far

I have been using my Netgear MP101 Media Player together with TwonkyVision for a long time - and have been mostly happy with it: it played my mp3 files, it streamed net radio, and only occassionally locked up (that was annoying, but I couldn't find my way around it. Streams were fine for hours, but local files sometimes resulted in a sielnt timeout of several minutes. I wonder how I could live with it so long...).

But what you couldn't do, was listening to ogg/vorbis streams. And my favourite radio station is only available in ogg/vorbis (well, besides from FM, but that is now not receivable by the MP101 and I didn't want to put another piece of equipment into the kitchen...)

Hence, I was looking around for another UPnP server for my MP101 to connect to. And found a few, but only MediaTomb announced to be able of transcoding, which was my one central requirement for a new server.

So, now what?

What follows is a description of the setup of MediaTomb to enable ogg/vorbis streaming as well as mp3 streaming and playing local mp3 files.

Using debian etch, the setup was painless, just apt-get mediatomb-common and you're good to go. I haven't yet installed the daemon, rather am currently running the whole thing from shell.

The first hurdle is to write a config.xml. This is more or less straightforward for local files, since you can add repositories via the web interface, which is surprisingly useable. With that setup, you can listen to everything that's local and in a format the player understands, which in my case is mp3.

Hello, anybody out there?

To enable streaming of net radios, you need to configure items via web interface. This was actually my first real struggle to find out, since I was used to doing things twonkyvision-like and it's all file there. Not so here, you are actually using a database. I like that approach! And you don't even need to setup a full mysql server (although you can, it's supported, but I never tried it), since sqlite is compiled into the debian etch packages and sets itself up under the directory from which mediatomb is started. Either way, this is all rather straight forward in the config.xml. If there are questions nonetheless, feel free to ask me at rockus_at_rockus_dot_at.

For streaming, the basic issue to find out is if the player supports direct http access or if you need to route the data through the UPnp server (in this case Mediatomb). If the former, you can insert the item "External Link (URL)" into your database at an appropriate point and your player will give it to you.

On the MP101, however, things are a bit more, erm, sophisticated. First of all, there is a button on the MP101 remote for Internet Radio. You need to create a virtual container in your Mediatomb database with exactly the name "Internet Radio". That way you can use the Radio button on the remote to directly jump to that directory.

Inside this Virtual Container, as it is called in MediaTomb, you can now add items. For an mp3 stream you need the following:

type:External Link (URL)
Title:[a free text naming the stream]
URL:[the complete FQDN, i.e. including http://]
Protocol:http-get
Class:object.item
Description:[free text]
Mimetype:audio/mpeg

The thing with mimetypes

The mimetype you put in here (i.e. the profile section of the config.xml, not the item entry in the web interface) can serve two things:

  • tell the media player what type of data is coming
  • tell Mediatomb how to handle the file

If you do not enable transcoding at all, this Mimetype will be used by the media player to decide what to do with the data, in this case interpret it as mpeg audio and try to play it. It only depends on the player firmware on what it does with the data. It could very well be that the player ignores the mimetype and instead looks into the sent data to find out the content. But in all likeliness, the mimetype should be set for correct rendering.

Now, what mimetypes are understood by the Netgear MP101?

audio/mpegMPEG audio (mp3 only in the case of the MP101)
audio/L16raw audio data, in big endian format
audio/L16;rate=xxxxxsame as above, only that you can specify the sample rate in a subtype (e.g.: rate=44100)

There are probably others as well, as this player is also supposed to understand WMA files, but I haven't tried that. The above are confirmed working.

Enable streaming

THe MP101 is a bit reluctant to play entries directly via URL. Sometimes it works, mostly it doesn't. Therefore, my approach was to route the stream through Mediatomb and let the player think it's a local file. This is done using the utterly cool transcoding feature of Mediatomb. This way of doing things is also described in the Mediatomb documentation.

What is done here is to instruct Mediatomb to connect to the stream and pass on the data (unchanged in the case of mp3) to the player. The way Mediatomb knows how to process the data and what transcoding to use is now again based on what you entered in the Mimetype filed of the item in the database.

In the config.xml file you first need to enable transcoding:

  <transcoding enabled="yes">
Inside this transcoding section there are two sub-sections, the first, <mimetype-profile-mappings>, maps mimetypes (the ones you entered in the field in the database entry) to profiles. More than one mimetype can point to a single profile. The fun part here is that if you are transcoding, you are not limited to pre-defined mime-types. Whatever you enter into the field in the database item, can be mapped here. Just make sure your player never sees a self-made mimetype as it most probably doesn't know what to do with it.

In the mp3 stream entry above I entered audio/mpeg as a mimetype, therefore, in the <mimetype-profile-mappings> section, I need an entry to capture that and map it to a profile:

      <transcode mimetype="audio/mpeg" using="streammp3"/>

There now needs to be a profile named streammp3, which has to do several things:

  • define the mimetype the player will see
  • forward the data to the player
  • several other things that can be read-up in the Mediatomb documentation

The whole section for me looks like this:

      <profile name="streammp3" enabled="yes" type="external">
        <mimetype>audio/mpeg</mimetype>
        <accept-url>no</accept-url>
        <agent command="dd" arguments="if=%in of=%out"/>
        <hide-original-resource>no</hide-original-resource>
        <first-resource>yes</first-resource>
        <buffer size="600000" chunk-size="10000" fill-size="32000"/>
      </profile>
The trick here is the dd command, which just passes on all it receives. Since dd is not able to handle HTTP requests, we need to set accept-url to no, to instruct Mediatomb to take over connecting to the stream.

The buffer size works fine for the MP101 for now as I haven't yet had any disconnects when playing local mp3 files. I had them with Twonky rather reliably, unfortunately. Mediatomb is better, since you can play with buffer settings.

And this is all there is to playing mp3 streams, i.e. net radio, on the MP101 using the Mediatomb UPnP server.

Can we please get onto the really interesting stuff now?

Step by step, we came closer to reaching the goal of playing ogg/vorbis streams. The thing that remains to be done is creating proper transcoding sub-sections to enable actual transcoding. With the mp3 stream above, there wasn't any real transcoding involved, it is more like forwarding using the transcoding feature.

What we want now is that streams (and files as well, should work for both) get recognised as ogg/vorbis and data be sent to a different profile in the transcoding section. We therefore need an additional line in the transcoding mapping section:

      <transcode mimetype="application/ogg" using="ogg2s16be.ogg123"/>
This now tells Mediatomb that whenever the user selects an item that has application/ogg set as its mediatype, Mediatomb is supposed to handle the stream as specified in the ogg2s16be.ogg123 section. Again, the naming of the profile is not relevant, but I like speaking names, so in this case it tells us that the profile converts ogg/vorbis to raw audio data, 16 bits, big endian, using ogg123 as decoder.

And this is exactly what is done in the section:

      <profile name="ogg2s16be.ogg123" enabled="yes" type="external">
        <mimetype>audio/L16; rate=48000</mimetype>
        <accept-url>yes</accept-url>
        <agent command="ogg123" arguments="-d raw -o byteorder:big -f%out %in"/>
        <buffer size="1048576" chunk-size="131072" fill-size="524288"/>
      </profile>
We instruct the ogg123 tool to extract the incoming stream to raw data and send it onwards. Since the data is not ogg/vorbis anymore, but raw data, we therefore need to set the mimetype the player will see (i.e. the mimetype inside the profile section) to the correct format (in this case audio/L16;rate48000. Yes, the sub-type is honoured by the MP101 and it will really attempt to play the stream at the specified sample rate. Buffer size is set to accomodate the larger amounts of datam the rest is straight forward.

With this setting I have managed to get my favoruite radio station play reliably on the MP101. Yay! Mission accomplished!

The fine print...

Drawback of using raw data is that it is a network hog. It uses bandwidth. But in times of 100Mbps networks, this isn't really a problem, but could be an issue if networks get used intensively. It would be better to re-code the raw data into something less network intensive. More on that later.

The big advantage, though, of being able to write any text into the mimetype field in the database item, is that we can also map to anything we want in the transcoding section. This comes in handy if you have streams of same format (e.g. ogg/vorbis), but different sample rates (e.g. 44100 and 48000). This can only be handled with sub-types. Following are the relevant parts of the transcoding section in my config.xml.

In the mapping sub-section:

      <transcode mimetype="application/ogg;rate=48000" using="ogg2s16be.ogg123.48000"/>
      <transcode mimetype="application/ogg;rate=44100" using="ogg2s16be.ogg123.44100"/>
In the profile sub-section:
      <profile name="ogg2s16be.ogg123.48000" enabled="yes" type="external">
        <mimetype>audio/L16; rate=48000</mimetype>
        <accept-url>yes</accept-url>
        <agent command="ogg123" arguments="-d raw -o byteorder:big -f %out %in"/>
        <buffer size="1048576" chunk-size="131072" fill-size="524288"/>
      </profile>
      <profile name="ogg2s16be.ogg123.44100" enabled="yes" type="external">
        <mimetype>audio/L16; rate=44100</mimetype>
        <accept-url>yes</accept-url>
        <agent command="ogg123" arguments="-d raw -o byteorder:big -f %out %in"/>
        <buffer size="1048576" chunk-size="131072" fill-size="524288"/>
      </profile>
The only difference being the profile names and the sub-type in the mimetype entry. Of course, the sample rate cannot be detected dynamically, once you decode to raw audio data, so the player needs to be instructed, which is done per the mimetype sub-types. Luckily, the MP101 supports that. The drawback here is that you need to specify on the database item entry (the one you create via web interface) the sample rate of the stream and have a spearate profile section for each sample rate you encounter. I haven't yet found a way to dynamically create the mimetype.

What we loose here is the ability to dynamically display song titles or any other metadata the incoming stream contains. But since the MP101 never actually seemed to decode that additional data, it's not much of a loss for me. On the other hand, perhaps one of you readers knows how to do that?

bloody ogg

After having had some initial problems gettings things running, I started reading the ogg specification and found that the format is nice and clean and straight-forward, but, it has the feature of sub-streams, which are a very useful things, but bites us here.

I wanted to get re-coding running, using ffmpeg, which wasn't really a piece of cake, and didn't work at all in the end. Which is the reason I ended up using raw data transfers, as shown above.

The nice thing about ffmpeg is that it theoretically can convert between lots of different file formats, keeping all or most of all metadata available in the target format. apt-get'ing ffmpeg doesn't help much, though, since debian decided to get rid of all proprietary code (which also brings me into trouble with my Tokenring network cards, but this is a different story altogether), and also threw out mp3 support in ffmpeg. Ergo, download and compile the source, including liblame0-dev.

I have actually no idea which packages are needed to get ffmpeg compiled on a naked debian system. I've got a system that has lots of compiler and assorted stuff installed and configure/make was a painless thing there. The only thing I did was to configure only the relevant codecs into the ffmpeg I compiled. This binary then ran flawlessly on a different system where all the media stuff resides.

ffmpeg is a very powerful tool, but in that powerfulness lies its weak point. ogg streams can have several streams and sub-streams inside. Now when you use ffmpeg, it maps a particular sub-stream on the input to the output. Usually this is fine, but usually net radios send metadata like song titles along with their streams and in ogg they are mostly implemented as new sub-streams every couple minutes.

ffmpeg would be perfect to transcode ogg/vorbis to mp3 on the fly, but it stops after a while and waits for stuff to happen. Specifically, it waits for more packets for its particular sub-stream, but they will never again arrive, since the stream moved on to different sub-streams with a different id.

I have not found a way of telling ffmpeg that it should combine all input sub-streams into one output streams. If anybody of you out there knows how to do that, I would very much appreciate you dropping me an email at rockus_at_rockus_dot_at. I'd very much like to reduce the network load.

For the curious among you, this is the mapping entry and the profile to get ogg-to-mp3 transcoding working - as long as a substream continues.

The mapping entry:

      <transcode mimetype="application/ogg" using="ogg2mp3"/>
The profile sub-section:
      <profile name="ogg2mp3" enabled="no" type="external">
        <mimetype>audio/mpeg</mimetype>
        <accept-url>no</accept-url>
        <first-resource>yes</first-resource>
        <accept-ogg-theora>no</accept-ogg-theora>
        <hide-original-resource>yes</hide-original-resource>
        <agent command="/usr/local/sbin/ffmpeg" arguments="-y -i %in -f mp3 -ab 128k %out"/>
        <buffer size="600000" chunk-size="10000" fill-size="32000"/>
      </profile>