So, Apple announced yesterday that they’ll stream today’s special event live, and everyone immediately assumed the load would crash the stream, if not the whole internet, myself included. But then I got thinking: they wouldn’t even try it if they weren’t pretty damn sure it would work. So what makes them think this will work?
HTTP Live Streaming, that’s why. I banged out a series of tweets (1, 2, 3, 4, 5, 6, 7, 8, 9) spelling out why the nature of HTTP Live Streaming (which I worked with briefly on a fix-up job last year) makes it highly plausible for such a use.
To summarize the spec: a client retrieves a playlist (an .m3u8, which is basically a UTF-8′ed version of the old WinAmp playlist format) that lists segments of the stream as flat files (often .m4a’s for audio, and .ts for video, which is an MPEG-2 transport stream, though Apple’s payload is presumably H.264/AAC). The client downloads these flat files and sends them to its local media player, and refreshes the playlist periodically to see if there are new files to fetch. The sizing and timing is configurable, but I think the defaults are like a 60-second refresh cycle on the playlist, and segments of about 10 seconds each.
This can scale for a live broadcast by using edge servers, which Apple has long depended on Akamai (and others?) for. Apple vends you a playlist URL at a local edge server, and its contents are all on the edge server, so the millions of viewers don’t pound Apple with requests — the load is pushed out to the edge of the internet, and largely stays off the backbone. Also, all the local clients will be asking for the same handful of segment files at the same time, so these could be in in-memory caches on the edge servers (since they’re only 10 seconds of video each). All these are good things.
I do wonder if local 3G cells will be a point of failure, if the bandwidth on a cell gets saturated by iPhone clients receiving the files. But for wired internet and wifi LANs, I suspect this is highly viable.
One interesting point brought up by TUAW is the dearth of clients that can handle HTTP Live Streaming. So far, it’s iOS devices, and Macs with QuickTime X (i.e., running Snow Leopard). The windows version of QuickTime doesn’t support HTTP Live Streaming (being based on the “old” 32-bit QuickTime on Mac, it may effectively be in maintenance mode). Open standard or not, there are no handy HTTP Live Streaming clients for other OS’s, though MacRumors’ VNC-based workaround (which requires you to manually download the .m3u8 playlist and do the refresh yourself), suggests it would be pretty easy to get it running elsewhere, since you already have the ability to play a playlist of segments and just need to automate the playlist refresh.
Dan Leehr tweeted back that Apple has talked a good game on HTTP Live Streaming, but hasn’t really showed much. Maybe this event is meant to change that. Moreover, you can’t complain about the adoption — last December, the App Store terms added a new fiat that any streaming video app must use HTTP Live Streaming (although a February post seems to ratchet this back to apps that stream for more than 10 minutes over the cellular network), so any app you see with a video streaming feature almost certainly uses HLS. At WWDC, Apple boasted about the MLB app using HLS, and it’s a safe bet that most/all other iOS video streaming apps (Netflix, Crunchyroll, etc.) use it too.
And one more thing to think about… MLB and Netflix aren’t going to stream without DRM, right? That’s the other piece that nobody ever talks about with HTTP Live Streaming: the protocol allows for encrypting of the media files. See section 5 of the spec. As much as Apple and its fanboys talk up HTML5 as a rival to and replacement for Flash, this is the thing that should really worry Adobe: commoditizing DRM’ed video streaming.
Mike Shaver’s blog about Mozilla forgoing H.264 support in the HTML <video> tag is getting a lot of play, predictably from the /. crowd.
It’s a shame, because it’s a childish and facile argument. Here’s the gist of it.
For Mozilla, H.264 is not currently a suitable technology choice. In many countries, it is a patented technology, meaning that it is illegal to use without paying license fees to the MPEG-LA. Without such a license, it is not legal to use or distribute software that produces or consumes H.264-encoded content.
In short, the very idea that something is patented and requires licensing is prima facie proof that it is intolerable. Going on:
These license fees affect not only browser developers and distributors, but also represent a toll booth on anyone who wishes to produce video content. And if H.264 becomes an accepted part of the standardized web, those fees are a barrier to entry for developers of new browsers, those bringing the web to new devices or platforms, and those who would build tools to help content and application development.
Yeah, can you imagine if any other technology were encumbered by patents? They’d have to pass on the costs to customers too! Imagine if television were patented, or if automobiles involved 100,000 patents… surely those products could never exist and never be affordable.
There is a case to be made against patents and intellectual property as a whole, but this blog doesn’t make it. Instead, it blithely refuses to acknowledge that we do live in a world of IP, decrying its costs as if they are out of the ordinary or unjust. Ultimately, it flees back to the intellectual dead-end of “everything should be in Ogg”, a stance so untenable that even ESR conceded it was a non-starter, four years ago.
A final irony: by refusing to support H.264, Mozilla bolsters the primary alternative for video on the web: the Flash plug-in, which is not just patent-encumbered but proprietary, available only from Adobe and only in those environments where it serves the company’s strategic ends. Shaver, to be fair, admits that proprietary plug-ins are bad too, but declines to say they’re far more bad than patent-encumbered standards. Instead, he holds out for a pollyannaish vision of completely free web video technology, naming Ogg as his moral standard-bearer, without acknowledging Ogg’s lamentable but obvious feet of clay.
In advance of Friday night's finale of Battlestar Galactica, here's a little game to play to see if they tie up all the loose ends:
| B | I | N | G | O |
| FREE | ||||
The board is written with JavaScript from 80-some possible items, so reload if you get a bad board. Tested on WebKit and Firefox. Thanks to Michael Ivey, Robert Cooper, and Michael Stemmle for trying out the prototype.
I’ve been using the WebKit nightly build as my preferred browser for about a year (ever since they put in the HTML5 <video> tag support, which of course is now ubiquitous in modern browsers).
On Sunday, I found a crashing bug, filed it, tracked the bug it turned out to be a duplicate of, and as of last night’s build, the bug is fixed.
Damn, that’s fast. Thanks guys, you rock.
I used to care about Java video (c.f., parts I, II, and III of an ONJava blog series about it). I don’t anymore.
It’s not just the shifting of my interest to the iPhone, it’s that Java’s powers that be have chosen their way, and it’s not something I think will work. No, change that: it won’t achieve any of the things that I care about. It may achieve the things they care about. Which, frankly, is demoing well at JavaOne.
At this year’s show, Sun announced they’d licensed some On2 codec, and showed off a “video cloud” demo of dozens of videos being played on panels being rotated about a 3D space.
People who don’t know better were impressed. Which is the point. What Sun needed to show was that it heard the concerns of Java developers about how badly the platform has lagged in multimedia support, and by showing a gee-whiz video tech demo, they mollified that crowd somewhat. Of course, this crowd of developers assumes it understands media because they all have iPods and have watched YouTube. it’s sort of like me presuming to be a security expert because I remember passwords and use login screens.
Problem is, as the problem always is with Java, Sun is talking to the customers they already have. Their solution is targeted to the developers already using Java. Were they talking to media application developers? Probably not, because their solution lacks credibility to anyone who actually works in the field.
Here, in a nutshell, is the problem. By choosing a non-standard video codec, they opt out of the economies and ecosystems that have developed around mainstream codecs like H.264 and VC-1. With the MPEG standards, for example, there is competition among encoders, as only decoding is standardized. Any product or process that produces a compliant bitstream is fair game. As a result, the bitrates needed to produce broadcast quality MPEG-2 dropped from 8 Mbps in the late 90′s to about 2-3 Mbps in the last few years. H.264 is expected to see the same evolution, driven by competition between encoders. With a single On2 encoder, Java Media Components won’t enjoy this kind of improvement. The no-brand codec also opts out of compressionist expertise.
And there’s also the question of whether the encoder is appropriate for multiple purposes, as H.264 has proven to be (albeit with so many optional features that the different profiles might well have been different codecs in another era). To my way of thinking, you generally need three functional styles of codecs, which often requires three totally different codecs:
- One-to-many playback – cases where the content is encoded once, and played back many times. The encoding may be thought of as “asynchronous”, in that the encoding may require much more time, CPU power, compressionist expertise, etc., than the playback. Example: DVD.
- One-to-one telephony – cases where encoder and player are more or less equals, needing a codec that can be encoded and decoded with the same resources (CPU power, time, etc.). Example: video conferencing
- Production – cases where the content will be heavily edited and processed. Codec here needs to be scrubbable, be easy to effect and edit, not be subject to generation loss, etc.
In general, I don’t think you find one codec that does all of these things well, and the fact that you see H.264 used for both playback and videoconferencing may speak to its high level of configurability. Presumably, you’re not using B-frames for video conferencing, due to the compression-time expense and the introduction of latency.
So having said all this, the idea that one codec is going to take care of all Java Media needs is highly implausible, especially since it’s not H.264.
The other thing that’s a big gotcha with Java Media is that the HTML5 <video> tag is poised to utterly commoditize media playback anyways. Why endure the hassle of trying to get Java runtimes on end-user systems, or have your developers and designers master JavaFX, when legions of JavaScript-savvy Ajax developers are getting video support provided to them with an industry-standard tag? Even if <video> doesn’t end up with a de jure official codec, it’s highly likely that H.264 will fill that role, as browser makers can rely on QuickTime and Flash, among other libraries, to render it. And if not, the prospect of browser-sniffing and sending different <video> tags to different browsers may still be more appealing than trying to goad users into installing Java.
To the Java crowd, this looks like a perfectly reasonable way to bring modern media support for the platform. It won’t matter, and they probably won’t think to ask why.
Surfin’ Safari notes initial support for the HTML5 <video> and <audio> tags in their latest nightly builds.
Indeed, if you have a browser that supports the video tag, then you (hopefully) can see an autoplaying video here:
There’s been a little bit of controversy over the fact that calls for inclusion of the Ogg formats have been removed in more recent versions of the spec. Section 3.14.7.1, “Video and audio codecs for video elements”, currently reads:
It would be helpful for interoperability if all browsers could support the same codecs. However, there are no known codecs that satisfy all the current players: we need a codec that is known to not require per-unit or per-distributor licensing, that is compatible with the open source development model, that is of sufficient quality as to be usable, and that is not an additional submarine patent risk for large companies. This is an ongoing issue and this section will be updated once more information is available.
That more or less matches my take on Ogg, which is that it poses an unknown patent liability risk: the /. mob insists it’s patent-free, but how the hell do they know? They don’t; they just want it to be so, because it suits their worldview. And Ogg may indeed be patent-free, but I don’t think anybody knows for sure, and even so, proving it would be expensive. To top it all off, Ogg just isn’t that popular or useful outside the warm bubble of Linux zealotry.
Still, there’s a huge need for at least one video and audio codec to be available more less everywhere, or at least for one class of devices: i.e., one codec you can expect all desktops to have, one for all phones, etc. To just dump out to “whatever QuickTime supports on the Mac, whatever Windows Media supports on Windows, etc.” ends up moving the problem, either to the web author (who has to sniff the OS from the user-agent and write the tag on the fly… to say nothing of hosting multiple encodings of every clip) or to the end-user.
It’s funny, because while the HTML5 <video> tag should displace Flash as the only practical option for web video — something that’s become screamingly obvious in the two years or so since YouTube launched — it might not, if it gets tangled up in codec hassles. The remarkable thing about Flash Video isn’t that it’s good (it’s not), but that it’s consistent and available on all Flash-enabled desktops.
That’s turned out to be a much bigger deal than the quality of competitors like H.264 and WMV, or the fact that other approaches could support many more codecs. Flash doesn’t try to support every codec under the sun, or even offer extension points for third parties to do so, but it doesn’t matter — with a known-viable video codec, content providers can just push their content with the package-deal of FLV and the Flash plug-in. Sure, the QuickTime plugin, a QuickTime for Java applet (or even a JMF applet, fercryinoutloud) could support more formats and codecs than Flash, but the typical use is not a general-purpose “play arbitrary content” application; the web-embedded player is usually meant to play the content from a single content provider, who’s perfectly happy to use a single, sub-optimal format, if the alternative is having to encode everything a dozen ways from Sunday to support the various OSes and devices.
Which makes me think that Flash’s ubiquity as a web-embedded video player won’t be threatened by HTML5, so long as there is neither a de jure nor de facto ubiquitous video codec for HTML5. Ironically, while H.264 might be the best candidate for that, Flash is already supporting it too.
Which leads me to an idea: if you were writing a browser on a platform without H.264 support from the native multimedia library, but you had Flash available, could you just pull the Flash player into service on the fly and have it play the H.264?
