Discussing Apple’s Java deprecation, Java creator James Gosling blogged about the background of Java on the Mac, saying “the biggest obstacle was their use of secret APIs. Yes, OS X has piles of secret APIs. Just like the ones that Microsoft had that contributed to their antitrust problems.”
In a recent Q&A at Google, available on YouTube, he elaborates further, around 43 minutes in (embedded YouTube clip will take you right there, otherwise read the blockquote):
At Sun, we had worked with them to try to take it over. But there were all kinds of issues, and it was mostly things like, you know, to integrate properly into the Mac OS, there were a bunch of secret APIs. And in their integration, there were all these secret APIs, and they wouldn’t tell us what they were, we just knew they were there. And then, you know, it’s sort of like half their brain wanted to give us the code, half their brain is like “no no no no no, we can’t”. So, nyah, that was all kind of spastic.
The fact that Dr. Gosling brings up “secret APIs” repeatedly when talking about the subject makes me think that he really wants to make this point that Apple’s use of secret APIs and its intransigence has been a major problem for Java on the Mac.
But… is it true? How big a deal are secret APIs in OSX and iOS anyways?
Nobody denies that there are undocumented and otherwise secret APIs throughout both OSX and iOS. They are easily found through techniques such as reverse-engineering and method swizzling. On OSX, they can be called, provided you can figure out their proper usage without documentation. Technically, this is also possible on iOS, although use of non-public APIs will get your app rejected by the App Store, so it’s largely pointless.
The benign explanation for secret APIs is that they’re used internally but haven’t been fully vetted for use by third-parties. We’ve all written code we’re not proud of and wouldn’t want others calling, or at least written utility functions and methods that were only thought through for certain uses and aren’t known to be appropriate for general use. An interesting example is iOS’
UIGetScreenImage function. As a devforums thread indicates, Apple started allowing use of this private API in 2009 because there wasn’t a good public alternative, with the proviso that its use would be disallowed once a suitable public API was released. This occurred with the arrival of AV Foundation in iOS 4.0, and direct calls to
UIGetScreenImage are again grounds for App Store rejection.
Aside from technical grounds, another reason for secret APIs is legal entanglements. There was an example of this in one of my earliest blogs: Apple licensed AAC encoding for OS X and for its own apps on Windows (iTunes, QuickTime Player), but not for third-party apps on Windows. According to Technical Q&A QA1347, a developer who wanted to provide this functionality on Windows would need to license the AMR encoding separately from VoiceAge, then provide proof of that license to Apple in order to get an SDK that would allow their code to make the secret call into QuickTime’s encoder.
But what can we say about Dr. Gosling’s complaints about secret APIs and Java? Certainly it plays well to the passions and politics of the Java community, but I’m not yet convinced. We know that most of Java actually ports to the Mac pretty easily: Landon Fuller’s “Soy Latte” project ported JDK 6 to the Mac in just a few person-weekends, and was later incorporated into OpenJDK’s BSD Ports subproject. But that left out some hard parts with intense native entanglements: sound, and the UI (Soy Latte, like most vanilla Java ports, relies on X11). Gosling acknowledges this in his blog, saying of these secret APIs that “the big area (that I’m aware of) where these are used is in graphics rendering.”
However, does this seriously mean that porting the Java graphics layer — Java2D, AWT, and Swing — is impractical or impossible without access to these secret APIs? It can’t be. After all, SWT exists for Mac as well, as a third-party creation, and it does the same things as these missing pieces of OpenJDK. In fact, SWT is more tightly-coupled to native code, as its whole approach is to bind Java objects to native peers (originally in Carbon, later in Cocoa), while Swing is all about avoiding native entanglements and instead painting look-alike widgets. Furthermore, I think Java’s rendering pipeline was switched over to an OpenGL implementation a while back, and that’s a public API that exists on OSX. So this kind of begs the question: what does Java need that isn’t provided by a public API? It doesn’t seem like graphics can be the problem.
The conspiracy theorists could argue that Apple has its own APIs that are more performant than the public APIs. Maybe, but what would be the point? Microsoft was roundly criticized for this in the 90′s, but Microsoft had more cases where their own products competed directly with third parties, and therefore could have incentive for their OS team to give a secret hand to the applications team. With Apple, software is their second-smallest revenue segment, and there are fewer cases where the company competes directly with a third-party rival (though there are clearly cases of this, such as Final Cut versus Premiere). Often, Apple’s software serves a strategic role – iLife may be more useful for selling Macs than for selling itself on DVD to existing Mac owners. So sure, Apple could be using secret APIs to give itself a leg up on competitors, but it’s hard to see how that would really be in their self-interest.
Having said all this, I’m still thwarted by a private API I needed this Summer: the “suck into a point” animation isn’t exposed by a Cocoa API on OSX, and asking for help on cocoa-unbound didn’t turn up an answer. Apparently, it’s possible on iOS, but via an undocumented method. Why this isn’t public on OSX or iOS, I can’t imagine, particularly given that Apple’s apps have made it a fairly standard behavior, meaning users will expect it when you use the round close button on a free-floating view. Oversight? Not ready for public consumption? Apple just being dicks? Who knows!
Of course, that brings up the last point about secret APIs. At the end of the day, they’re almost always conveniences. If something is possible at all, you could probably just do it yourself. I don’t know exactly what transforms are involved in the suck-to-close animation, but it’s surely possible to create a reasonably close approximation with Core Animation. Similarly, instead of calling QuickTime’s secret AAC encoder on Windows, you could license some other library or framework, or write your own. It might not be easy or practical, but if Apple can move the bits in some specific way, it must at least be possible for a third-party to do the same.
Outrage du jour is the tale of a Danish app maker whose Android magazine app was rejected by Apple.
App rejection stories are fun because they get everyone’s hair up in a dander. There’s something for everyone to hate in Apple’s App Approval Guidelines, and the restrictions on content are especially pernicious and ridiculous. One that bugs me, for example: you can buy the heartbreaking anime series Rumbling Hearts on iTunes, complete with its handful of sex scenes, but the original visual novel it was based on would not be permitted on the App Store, because of those same scenes.
The thing is, though, we’ve sort of already had this argument about the App Store policies, and we got to do so in detail when Apple put up an official set of policies a few months ago. One that got a lot of abuse was guideline 3.1, which says that apps that mention other mobile platforms in their metadata will get rejected. That’s petty, childish, and of dubious self-interest to Apple (are they really that threatened that, say, a game description might mention that it’s also available on WebOS and Android?).
But you know what? When the rule is that clear, and you violate it that obviously, what do you think is going to happen?
Sure, one could appeal to a sense of fairness and ask for Apple to bend the rules on this one… a single app isn’t going to make that big of a difference, right? But of course, this leads to the snowball argument: if these guys get a pass, then the next app to brazenly break the rules is going to expect a waiver too.
It’s a stupid rule, and these guys broke it in a very presumptuous and obvious way. They can be disappointed, but they can’t be surprised.
Still, I can’t help but wonder what the point of all this really is? Are there really a lot of iPad owners who want to read about Android development? Oh, I can only imagine the kinds of articles that would go in an Android magazine.
- Stealing Angry Birds IP For Fun And Profit
- Our 10 Favorite Screen Dimensions
- How Railing Against Apple Proves You Are An Independent Thinker And In No Way A Corporate Tool (a special advertising section provided by Google)
- Five Games You Can Write Without Resorting To Native Code (Six If You Count Hangman)
- Healthy Snacks You Can Make While Your App Starts Up
- How To Write Your First Unremovable Carrier Crapware App
- Eye Scream, You Scream, We All Scream For Ads That Involve Getting Metal In Your Eye
Totally looking forward to the inevitable Flash version of this. Totally.
I’m late to the party blogging about Apple deprecating its Java for Mac OS X, with particularly good posts up from Matt Drance and Lachlan O’Dea. And I’ve already posted a lot of counterpoints on the Java Posse’s Google group (see here, here, and here). So let me lead with the latest: there’s now a petition calling on Apple to contribute its Mac Java sources to OpenJDK. This won’t work, for at least four reasons I can think of:
- Petitions never work.
- Apple doesn’t listen.
- Apple is a commercial licensee of Java. The terms in their contract with Sun/Oracle almost certainly prohibit open-sourcing their Sun-derived code.
- Even if it could be open-sourced, Apple’s code was developed for years with the assumption it was proprietary code. The company would want nothing less than an absolutely forensic code analysis to ensure there are no loopholes, stray imports or links, or anything else that a creative FSF lawyer could use to claim that Mac OS X links against the GPL’ed OpenJDK and must therefore itself be GPL’ed. Apple has nothing to earn and everything to lose by open-sourcing its JDK, so don’t hold your breath.
It’s also frustratingly typical that many of the signatories of this petition have spent the last week blogging, tweeting, podcasting, and posting that they will never buy another Apple product (and they’re gonna go tell all their friends and relations that Apple sucks now, just for good measure). Here’s the thing: companies generally try to do right by their customers. When you assert that you will never be their customer again, you remove any reason the company would have to listen to your opinion. Really, this much should be common sense, right?
Buried in all the denunciations of “control freak Steve Jobs” and his nefarious skullduggery is a wake-up call that Oracle and the Java community need to hear: one of your biggest commercial licensees, the second biggest US corporation by market cap, doesn’t think licensing Java will help them sell computers anymore. Why does nobody take this screamingly obvious hint?
I imagine part of the reason that the activists want Apple to contribute its code instead of letting it go to waste is that they’ve realized what a tall order it would be for the community to attempt a port on its own. Java is huge, and its native dependencies are terribly intractable: even the vaunted Linux community couldn’t get Blackdown Java up to snuff, and Sun came to the rescue with an official version for Linux. How likely is it that we’ll find enough people who know Java and Mac programming well enough — and who care to contribute their time — to do all this work for free? It may well be a non-starter. Landon Fuller got the headless bits of JDK 6 ported to OS X ported in a few weeks in the form of the Soy Latte project, but had to settle for an X11-based UI, and his announcement asked for help with Core Audio sound support and OS X integration in general… and I don’t believe any help ever materialized.
Aside: when volunteers aren’t enough, the next step is to get out the checkbook and call in mercenaries. I looked at javax.sound yesterday and estimated it would take me 4-6 weeks, full-time, to do a production-quailty port using Core Audio. I’ll bid the project out at $20,000. Sign a contract and I’ll contribute the sources to any project you like (OpenJDK, Harmony, whatever). You know where to reach me. And I’m not holding my breath.
The real problem with porting Java is that Java’s desktop packages – AWT, Swing, javax.sound, etc. – are very much a white elephant, one which perfectly fits Wikipedia’s definition:
A white elephant is an idiom for a valuable possession of which its owner cannot dispose and whose cost (particularly cost of upkeep) is out of proportion to its usefulness or worth.
As I’ve established, Java’s desktop packages are egregiously expensive. In fact, with Apple’s exit, it’s not clear that there’s anybody other than Oracle delivering a non-X11 AWT/Swing implementation for any platform: it’s just too much cost and not enough value. End-user Desktop Java applications are rare and get rarer every day, displaced largely by browser-based webapps, but also by Flash and native apps.
We know the big use for AWT and Swing, and it’s a terrible irony: measured by app launches or time spent in an app, the top AWT/Swing apps are surely NetBeans and IntelliJ, IDEs used for creating… other Java applications! The same can be said of the SWT toolkit, which powers the Eclipse IDE and not much else. This is what makes this white elephant so difficult to dispose of: all the value of modern-day Java is writing for the server, but nearly all the Java developers are using the desktop stuff to do so, making them the target market (and really the only market) for Desktop Java. If there were other viable Java applications on the desktop, used by everyday end-users, Apple couldn’t afford to risk going without Java. There aren’t, and it can.
There’s lots of blame to go around, not the least of which should be directed at Sun for abandoning desktop Java right after Swing 1.0 came out. It’s embarrassing that my five-year-old Swing book is more up-to-date with its technology than my year-old-iPhone book is, but it illustrates the fact that Apple has continued to evolve iOS, while Sun punted on client-side Java for years, and then compounded the problem by pushing aside the few remaining Swing developers in favor of the hare-brained JavaFX fiasco. Small wonder that nearly all the prominent desktop Java people I know are now at Google, working on Android.
Other languages are easier to port and maintain because Ruby, Python, and the like don’t try to port their own entire desktop API with them from platform to platform. Again, the irony is that this stuff is so expensive in Java, and little related to the server-side work where Java provides nearly all of its value. I’m not the first to say it, but Oracle would do itself a huge favor by finding a way to decouple all this stuff from the parts of Java that actually get used, likely as a result of Project Jigsaw. Imagine something like this:
In this hypothetical arrangement, the desktop packages are migrated out of Java SE, which now contains the baseline contents of Java that everybody uses: collections, I/O, language utilities, etc. These are the parts that EE actually uses, and that dependency stays in place. I’ve colored those boxes green to indicate that they don’t require native widgets to be coded. What does require that is the new Java “Desktop Edition” (“Java DE”), which is SE plus AWT, Swing, Java2D, javax.sound, etc. For the rare case where you need UI stuff on the server, like image manipulation as a web service, combine EE and DE to create “Java OE”, the “Omnibus edition”. Also hanging off SE are ME (which is SE minus some features, and with a new micro UI), as well as non-Sun/Oracle products that derive from SE today, such as Eclipse’s quasi-platform, and Android. Of course, it’s easy to play armchair architect: the devil is in the details, and Mark Reinhold mentioned on Java Posse 325 that there are some grievously difficult dependencies created by the unfortunate
java.beans package. Still, when the cost of the desktop packages is so high, and their value so low, they almost certainly need to be moved off the critical path of Java’s evolution somehow. The most important thing Oracle needs to provide in Java 7 or 8 is an exit strategy from desktop Java.
Instead, we’ll continue to hear about what an utter rat-bastard Steve Jobs is, and how the deprecation of Apple’s Java is part of a scheme to “force developers to use Objective-C”, as if there were a flood of useful and popular Java applications being shoved off the Mac platform. Many of the ranters insist on dredging up Jobs’ vow to “make the Mac the best Java platform”, in determined ignorance of the fact that this statement was made over ten years ago, at JavaOne 2000. For context: when Jobs was on the JavaOne stage, the US President was Bill Clinton, and Sun was #150 on the Fortune 500, ahead of Oracle and Apple (and Google, which didn’t even enter the list until 2005). If TV’s Teen Titans taught us anything, it’s that Things Change.
And for a while, maybe the Mac was briefly the best Java platform: unlike Windows (legally enjoined from shipping Java by their attempts to subvert it) and Linux (where many distros turned up their noses at un-free Java for years), Apple shipped Java as a core part of the OS for years. Not an option: it was installed as part of the system and could not practically be removed. Apple’s Mac look-and-feel for Swing was also widely praised. Do you know who said the following?
I use the MAC because it’s a great platform. One of the nice things about developing in Java on the Mac is that you get to develop on a lovely machine, but you don’t cut yourself off from deploying on other platforms. It’s a fast and easy platform to develop on. Rock solid. I never reboot my machine… Really! Opening and closing the lid on a Powerbook actually works. The machine is up and running instantly when you open it up. No viruses. Great UI. All the Java tools work here: NetBeans and JEdit are the ones I use most. I tend to think of OSX and Linux with QA and Taste.
A lot of us who’ve developed Java on the Mac have expected this for years, and this week’s reactions are all a lot of sound and fury, signifying nothing. The Java crowd will keep doing what it does — making web sites — but they’ll have to make a choice: either use Windows or Linux for development, or get away from the IDEs and write their code with plain text editors and the command line on the Mac. If I were still doing Java, I probably would barely notice, as I wrote nearly all my Java code with emacs after about 2000 or so, and never got hooked on NetBeans or Eclipse. But with the most appealing desktops and the most popular mobile devices ditching Java, it’s time for the Java community to face up to the fact that Desktop Java is a ruinously expensive legacy that they need to do something about.
All the angry screeds against Steve Jobs won’t change the fact that this is the “ball and chain” that’s pulling Java below the waves.
Much renewed hope and delight last week, after Apple pulled back from its most audacious and appalling land-grabs in its iOS developer agreement, notably the revised section 3.3.1 that prohibited any languages but C, Objective-C, and C++ for iOS development (Daring Fireball quotes the important changes in full). Whether this is the result of magnanimity or regulatory pressures in the U.S. and Europe is unknowable and therefore unhelpful. What’s interesting is thinking about what we can do with our slightly-loosened shackles.
For example, it would be interesting to see if someone, perhaps a young man or lady with a mind for mischief or irony, could bring Google’s Go programming language to iOS development. Since the Go SDK runs on Mac and can compile for ARM, it might well be possible to have a build script call the Go compiler as needed. And of all the hot young languages, Go might be the most immediately applicable, as it is compiled, rather than interpreted.
And that brings up the other major change, the use of interpreters. Nobody seems to be noting that the change in section 3.3.2 is not just a loosening of this Spring’s anti-Flash campaign, but is in fact far more lenient than this policy has ever been. Since the public SDK came out in 2008, all forms of interpreted code have been forbidden. This is what dashed early plans to bring Java to the iPhone as an application runner, despite its absence as an applet runner in Safari. As Matt Drance has pointed out, the new policy reflects the reality on the ground that interpreters (especially Lua) have been tolerated for some time in games. The new phrasing forbids downloading of executable content, but allows for cases where the interpreter and all scripts are included in the app bundle. This has never been allowed before, and is a big deal.
Now let me stretch the definition of “interpreter” a bit, to the point where it includes virtual machines. After all, the line between the two is hard to define: a “virtual machine” is a design philosophy, not a technical trait. A VM uses an interpreter (often a byte code interpreter rather than source, but not necessarily), and presumably has more state and exposes more library APIs. But languages and their interpreters are getting bigger – Ruby I/O is in the language rather than in a library (like C or Java), but that doesn’t make Ruby a VM, does it?
You might have surmised where I’m going with this: I don’t think the revised section 3.3.2 bans a hypothetical port of the Flash or Java VMs to iOS anymore, if they’re in a bundle with the .swf or .jar files that they will execute.
I could be wrong, particularly given Steve Jobs’ stated contempt for these sorts of intermediary platforms. But if a .swf-and-Flash-VM bundle were rejected today, it would be by fiat, and not by the letter of section 3.3.2.
Whether any of this matters depends on whether anyone has stand-alone Flash applications (such as AIR apps) or Java applications that have value outside of a browser context, and are worth bringing to a mobile platform.
SFX: CRICKETS CHIRPING
I can’t say why AIR never seemed to live up to its billing, but the failings of Desktop Java can in part be blamed on massive neglect by Sun, exacerbated by internecine developer skirmishes. Swing, the over-arching Java UI toolkit, was plagued by problems of complexity and performance when it was introduced in the late 90′s, problems that were never addressed. It’s nigh impossible to identify any meaningful changes to the API following its inclusion in Java 1.2 in 1998. Meanwhile, the IBM-funded Eclipse foundation tied their SWT more tightly to native widgets, but it was no more successful than Swing, at least in terms of producing meaningful apps. Each standard powers one IDE, one music-stealing client, and precious little else.
So, aside from the debatability of section 3.3.2, and wounded egos in the Flash and Java camps, the biggest impediment to using a “code plus VM” porting approach may be the fact that there just isn’t much worth porting in the first place.
Speaking of Desktop Java, the Java Posse’s Joe Nuxoll comes incredibly close to saying something that everybody in that camp needs to hear. In the latest episode, at 18:10, he says “…gaining some control over the future of mobile Java which, it’s over, it’s Android, it’s done.” He later repeats this assertion that Android is already the only form of mobile Java that matters, and gets agreement from the rest of the group (though Tor Norbye, an Oracle employee, can’t comment on this discussion of the Oracle/Google lawsuit, and may disagree). And this does seem obvious: Android, coupled with the rise of the smartphone, has rendered Java ME irrelevant (to say nothing of JavaFX Mobile, which seems stillborn at this point).
But then at 20:40, Joe makes the big claim that gets missed: “Think of it [Android] as Desktop Java for the new desktop.” Implicit in this is the idea that tablets are going to eat the lunch of traditional desktops and laptops, and those tablets that aren’t iPads will likely be Android-based. That makes Android the desktop Java API of the future, because not only have Swing, SWT, and JavaFX failed, but the entire desktop model is likely threatened by mobile devices. There are already more important, quality, well-known Android apps after two years than the desktop Java APIs produced in over a decade. Joe implies, but does not say, what should be obvious: all the Java mobile and desktop APIs are dead, and any non-server Java work of any relevance in the future will be done in Android.
No wonder Oracle is suing for a piece of it.
Disclaimer: there are aspects of the App Store approval process I find utterly appalling, particularly the stealth “no competing with Apple” unwritten rule.
Nevertheless, here’s a mental exercise for you. Consider the user-facing download-to-own software stores offered by the game consoles: Wii Ware, XBox Live Arcade, and PlayStation Network.
In what way is any Apple App Store policy more onerous than these stores’ developer policies?
Here are some links to help you research:
- To become an Authorized Developer for Wii, WiiWare and/or Nintendo DS/DSi
- WELCOME TO MICROSOFT CASUAL GAMES!
- SCE DevNet (almost utterly opaque… see also Developing for PS3 PlayStation Network (PSN))
Once we get past the much more expensive licensing, the pre-vetting of both product ideas (Microsoft: “Email firstname.lastname@example.org with a description of your game and your contact information. We’ll send you a content submission form.”) and developers (Nintendo: “the authorization for Wii/WiiWare or Nintendo DS will be based upon your relevant game industry experience.”), and the fact that these platforms generally reject entire classes of applications (anything that isn’t a game)… I think it’s interesting to compare the anger and fury vented over the App Store, and consider that almost nobody is railing against these stores, even though they’re much more closed than Apple’s platform, and may collectively reach more users.
We might also do well to note how closed mobile development was before the iPhone. I know I’ve told this story before, but in a JavaOne conversation with O’Reilly people about how to get Java ME books moving, I said that everyone with an interest in ME (myself included) had figured out that getting your apps to end users was effectively impossible, and that with the network API often disabled for third-party apps, there wasn’t much point in writing ME apps anyways. My suggestion for an ME book that would move copies would be one which provided “the names, e-mails, and phone numbers of all the carrier and handset executives you’d have to go down on in order to get your apps on their phones.”