[Time code];

iPhone SDK 3.0 provided limited access to the iPod Music Library on the device, allowing third party apps to search for songs (and podcasts and audiobooks, but not video), inspect the metadata, and play items, either independently or in concert with the built-in media player application. But it didn’t provide any form of write-access — you couldn’t add items or playlists, or alter metadata, from a third-party app. And it didn’t allow for third-party apps to do anything with the songs except play them… you couldn’t access the files, convert them to another format, run any kind of analysis on the samples, and so on.

So a lot of us were surprised by the WWDC keynote when iMovie for iPhone 4 was shown importing a song from the iPod library for use in a user-made video. We were even more surprised by the subsequent claim that everything in iMovie for iPhone 4 was possible with public APIs. Frankly, I was ready to call bullshit on it because of the iPod Library issue, but was intrigued by the possibility that maybe you could get at the iPod songs in iOS 4. A tweet from @ibeatmaker confirmed that it was possible, and after some clarification, I found what I needed.

About this time, a thread started on coreaudio-api about whether Core Audio could access iPod songs, so that’s what I set out to prove one way or another. So, my goal was to determine whether or not you could get raw PCM samples from songs in the device’s music library.

The quick answer is: yes. The interesting answer is: it’s a bitch, using three different frameworks, coding idioms that are all over the map, a lot of file-copying and possibly some expensive conversions.

It’s Just One Property; It Can’t Be That Hard

The big secret of how to get to the Music Library isn’t much of a secret. As you might expect, it’s in the MediaLibrary.framework that you use to interact with the library. Each song/podcast/audiobook is a MPMediaItem, and has a number of interesting properties, most of which are user-managed metadata. In iOS 4, there’s a sparkling new addition to the the list of “General Media Item Property Keys”: MPMediaItemPropertyAssetURL. Here’s the docs:

A URL pointing to the media item, from which an AVAsset object (or other URL-based AV Foundation object) can be created, with any options as desired. Value is an NSURL object.

The URL has the custom scheme of ipod-library. For example, a URL might look like this:

ipod-library://item/item.m4a?id=12345

OK, so we’re off and running. All we need to do is to pick an MPMediaItem, get this property as an NSURL, and we win.

Or not. There’s an important caveat:

Usage of the URL outside of the AV Foundation framework is not supported.

OK, so that’s probably going to suck. But let’s get started anyways. I wrote a throwaway app to experiment with all this stuff, adding to it piece by piece as stuff started working. I’m posting it here for anyone who wants to reuse my code… all my classes are marked as public domain, so copy-and-paste as you see fit.

MediaLibraryExportThrowaway1.zip

Note that this code must be run on an iOS 4 device and cannot be run in the Simulator, which doesn’t support the Media Library APIs.

The app just starts with a “Choose Song” button. When you tap it, it brings up an MPMediaPickerController as a modal view to make you choose a song. When you do so, the -mediaPicker:didPickMediaItems: delegate method gets called. At this point, you could get the first MPMediaItem and get its MPMediaItemPropertyAssetURL media item property. I’d hoped that I could just call this directly from Core Audio, so I wrote a function to test if a URL can be opened by CA:

BOOL coreAudioCanOpenURL (NSURL* url) {
	OSStatus openErr = noErr;
	AudioFileID audioFile = NULL;
	openErr = AudioFileOpenURL((CFURLRef) url,
		 kAudioFileReadPermission ,
		 0,
		 &audioFile);
	if (audioFile) {
		AudioFileClose (audioFile);
	}
	return openErr ? NO : YES;
}

Getting a NO back from this function more or less confirms the caveat from the docs: the URL is only for use with the AV Foundation framework.

AV for Vendetta

OK, so plan B: we open it with AV Foundation and see what that gives us.

AV Foundation — setting aside the simple player and recorder classes from 3.0 — is a strange and ferocious beast of a framework. It borrows from QuickTime and QTKit (the capture classes have an almost one-to-one correspondence with their QTKit equivalents), but builds on some new metaphors and concepts that will take the community a while to digest. For editing, it has a concept of a composition, which is made up of tracks, which you can create from assets. This is somewhat analogous to QuickTime’s model that “movies have tracks, which have media”, except that AVFoundation’s compositions are themselves assets. Actually, reading too much QuickTime into AV Foundation is a good way to get in trouble and get disappointed; QuickTime’s most useful functions, like AddMediaSample() and GetMediaNextInterestingTime() are antithetical to AV Foundation’s restrictive design (more on that in a later blog) and therefore don’t exist.

Back to the task at hand. The only thing we can do with the media library URL is to open it in AVFoundation and hope we can do something interesting with it. The way to do this is with an AVURLAsset.

NSURL *assetURL = [song valueForProperty:MPMediaItemPropertyAssetURL];
AVURLAsset *songAsset = [AVURLAsset URLAssetWithURL:assetURL options:nil];

If this were QuickTime, we’d have an object that we could inspect the samples of. But in AV Foundation, the only sample-level access afforded is a capture-time opportunity to get called back with video frames. There’s apparently no way to get to video frames in a file-based asset (except for a thumbnail-generating method that operates on one-second granularity), and no means of directly accessing audio samples at all.

What we can do is to export this URL to a file in our app’s documents directory, hopefully in a format that Core Audio can open. AV Foundation’s AVAssetExportSession has a class method exportPresetsCompatibleWithAsset: that reveals what kinds of formats we can export to. Since we’re going to burn the time and CPU of doing an export, it would be nice to be able to convert the compressed song into PCM in some kind of useful container like a .caf, or at least an .aif. But here’s what we actually get as options:

compatible presets for songAsset: (
 AVAssetExportPresetLowQuality,
 AVAssetExportPresetHighestQuality,
 AVAssetExportPreset640x480,
 AVAssetExportPresetMediumQuality,
 AVAssetExportPresetAppleM4A
 )

So, no… there’s no “output to CAF”. In fact, we can’t even use AVAssetExportPresetPassthrough to preserve the encoding from the music library: we either have to convert to an AAC (in an .m4a container), or to a QuickTime movie (represented by all the presets ending in “Quality”, as well as the “640×480″).

This Deal is Getting Worse All the Time!

So, we have to export to AAC. That’s not entirely bad, since Core Audio should be able to read AAC in an .m4a container just fine. But it sucks in that it will be a lossy conversion from the source, which could be MP3, Apple Lossless, or some other encoding.

In my GUI, an “export” button appears when you pick a song, and the export is kicked off in the event-handler handleExportTapped. Here’s the UI in mid-export:

To do the export, we create an AVExportSession and provide it with an outputFileType and outputIURL.

AVAssetExportSession *exporter = [[AVAssetExportSession alloc]
		initWithAsset: songAsset
		presetName: AVAssetExportPresetAppleM4A];
NSLog (@"created exporter. supportedFileTypes: %@", exporter.supportedFileTypes);
exporter.outputFileType = @"com.apple.m4a-audio";
NSString *exportFile = [myDocumentsDirectory()
		stringByAppendingPathComponent: @"exported.m4a"];
myDeleteFile(exportFile);
[exportURL release];
exportURL = [[NSURL fileURLWithPath:exportFile] retain];
exporter.outputURL = exportURL;

A few notes here. The docs say that if you set the outputURL without setting outputFileType that the exporter will make a guess based on the file extension. In my experience, the exporter prefers to just throw an exception and die, so set the damn type already. You can get a list of possible values from the class method exporter.supportedFileTypes. The only supported value for the AAC export is com.apple.m4a-audio. Also note the call to a myDeleteFile() function; the export will fail if the target file already exists.

Aside: I did experiment with exporting as a QuickTime movie rather than an .m4a; the code is in the download, commented out. Practical upshot is that it sucks: if your song isn’t AAC, then it gets converted to mono AAC at 44.1 KHz. It’s also worth noting that AV Foundation doesn’t give you any means of setting export parameters (bit depths, sample rates, etc.) other than using the presets. If you’re used to the power of frameworks like Core Audio or the old QuickTime, this is a bitter, bitter pill to swallow.

Block Head

The code gets really interesting when you kick off the export. You would probably expect the export, a long-lasting operation, to be nice and asynchronous. And it is. You might also expect to register a delegate to get asynchronous callbacks as the export progresses. Not so fast, Bucky. As a new framework, AV Foundation adopts Apple’s latest technologies, and that includes blocks. When you export, you provide a completion handler, a block whose no-arg function is called when necessary by the exporter.

Here’s what mine looks like.

// do the export
[exporter exportAsynchronouslyWithCompletionHandler:^{
	int exportStatus = exporter.status;
	switch (exportStatus) {
		case AVAssetExportSessionStatusFailed: {
			// log error to text view
			NSError *exportError = exporter.error;
			NSLog (@"AVAssetExportSessionStatusFailed: %@",
				exportError);
			errorView.text = exportError ?
				[exportError description] : @"Unknown failure";
			errorView.hidden = NO;
			break;
		}
		case AVAssetExportSessionStatusCompleted: {
			NSLog (@"AVAssetExportSessionStatusCompleted");
			fileNameLabel.text =
				[exporter.outputURL lastPathComponent];
			// set up AVPlayer
			[self setUpAVPlayerForURL: exporter.outputURL];
			[self enablePCMConversionIfCoreAudioCanOpenURL:
				exporter.outputURL];
			break;
		}
		case AVAssetExportSessionStatusUnknown: {
			NSLog (@"AVAssetExportSessionStatusUnknown"); break;}
		case AVAssetExportSessionStatusExporting: {
			NSLog (@"AVAssetExportSessionStatusExporting"); break;}
		case AVAssetExportSessionStatusCancelled: {
			NSLog (@"AVAssetExportSessionStatusCancelled"); break;}
		case AVAssetExportSessionStatusWaiting: {
			NSLog (@"AVAssetExportSessionStatusWaiting"); break;}
		default: { NSLog (@"didn't get export status"); break;}
	}
}];

This kicks off the export, passing in a block with code to handle all the possible callbacks. The completion handler function doesn’t have to take any arguments (nor do we have to set up a “user info” object for the exporter to pass to the function), since the block allows anything in the local scope to be called from the block. That means the exporter and its state don’t need to be passed in as parameters, because the exporter is a local variable that can be accessed from the block and its state inspected via method calls.

The two messages I handle in my block are AVAssetExportSessionStatusFailed, which dumps the error to a previously-invisible text view, and AVAssetExportSessionStatusCompleted, which sets up an AVPlayer to play the exported audio, which we’ll get to later.

After starting the export, my code runs an NSTimer to fill a UIProgressView. Since the exporter has a progress property that returns a float, it’s pretty straightforward… check the code if you haven’t already done this a bunch of times. Files that were already AAC export almost immediately, while MP3s and Apple Lossless (ALAC) took a minute or more to export. Files in the old .m4p format, from back when the iTunes Store put DRM on all the songs, fail with an error, as seen below.

The Invasion of Time

Kind of as a lark, I added a little GUI to let you play the exported file. AVPlayer was the obvious choice for this, since it should be able to play whatever kind of file you export (.m4a, .mov, whatever).

This brings up the whole issue of how to deal with the representation of time in AV Foundation, which turns out to be great for everyone who ever used the old C QuickTime API (or possibly QuickTime for Java), and all kinds of hell for everyone else.

AV Foundation uses Core Media’s CMTime struct for representing time. In turn, CMTime uses QuickTime’s brilliant but tricky concept of time scales. The idea, in a nutshell, is that your units of measurement for any particular piece of media are variable: pick one that suits the media’s own timing needs. For example, CD audio is 44.1 KHz, so it makes sense to measure time in 1/44100 second intervals. In a CMTime, you’d set the timescale to 44100, and then a given value would represent some number of these units: a single sample would have a value of 1 and would represent 1/44100 of a second, exactly as desired.

I find it’s easier to think of Core Media (and QuickTime) timescales as representing “nths of a second”. One of the clever things you can do is to choose a timescale that suits a lot of different kinds of media. In QuickTime, the default timescale is 600, as this is a common multiple of many important frame-rates: 24 fps for film, 25 fps for PAL (European) TV, 30 fps for NTSC (North America and Japan) TV, etc. Any number of frames in these systems can be evenly and exactly represented with a combination of value and timescale.

Where it gets tricky is when you need to work with values measured in different timescales. This comes up in AV Foundation, as your player may use a different timescale than the items it’s playing. It’s pretty easy to write out the current time label:

CMTime currentTime = player.currentTime;
UInt64 currentTimeSec = currentTime.value / currentTime.timescale;
UInt32 minutes = currentTimeSec / 60;
UInt32 seconds = currentTimeSec % 60;
playbackTimeLabel.text = [NSString stringWithFormat:
		@"%02d:%02d", minutes, seconds];

But it’s hard to update the slider position, since the AVPlayer and the AVPlayerItem it’s playing can (and do) use different time scales. Enjoy the math.

if (player && !userIsScrubbing) {
	CMTime endTime = CMTimeConvertScale (player.currentItem.asset.duration,
		currentTime.timescale,
		kCMTimeRoundingMethod_RoundHalfAwayFromZero);
	if (endTime.value != 0) {
		double slideTime = (double) currentTime.value /
				(double) endTime.value;
		playbackSlider.value = slideTime;
	}
}

Basically, the key here is that I need to get the duration of the item being played, but to express that in the time scale of the player, so I can do math on them. That gets done with the CMTimeConvertScale() call. Looks simple here, but if you don’t know that you might need to do a timescale-conversion, your math will be screwy for all sorts of reasons that do not make sense.

Oh, you can drag the slider too, which means doing the same math in reverse.

-(IBAction) handleSliderValueChanged {
	CMTime seekTime = player.currentItem.asset.duration;
	seekTime.value = seekTime.value * playbackSlider.value;
	seekTime = CMTimeConvertScale (seekTime, player.currentTime.timescale,
			kCMTimeRoundingMethod_RoundHalfAwayFromZero);
	[player seekToTime:seekTime];
}

One other fun thing about all this that I just remembered from looking through my code. The time label and slider updates are called from an NSTimer. I set up the AVPlayer in the completion handler block that’s called by the exporter. This call seems not to be on the main thread, as my update timer didn’t work until I forced its creation over to the main thread with performSelectorOnMainThread:withObject:waitUntilDone:. Good times.

Final Steps

Granted, all this AVPlayer stuff is a distraction. The original goal was to get from iPod Music Library to decompressed PCM samples. We used an AVAssetExportSession to produce an .m4a file in our app’s Documents directory, something that Core Audio should be able to open. The remaining conversion is a straightforward use of CA’s Extended Audio File Services: we open an ExtAudioFileRef on the input .m4a, set a “client format” property representing the PCM format we want it to convert to, read data into a buffer, and write that data back out to a plain AudioFileID. It’s C, so the code is long, but hopefully not too hard on the eyes:

-(IBAction) handleConvertToPCMTapped {
	NSLog (@"handleConvertToPCMTapped");

	// open an ExtAudioFile
	NSLog (@"opening %@", exportURL);
	ExtAudioFileRef inputFile;
	CheckResult (ExtAudioFileOpenURL((CFURLRef)exportURL, &inputFile),
				 "ExtAudioFileOpenURL failed");

	// prepare to convert to a plain ol' PCM format
	AudioStreamBasicDescription myPCMFormat;
	myPCMFormat.mSampleRate = 44100; // todo: or use source rate?
	myPCMFormat.mFormatID = kAudioFormatLinearPCM ;
	myPCMFormat.mFormatFlags =  kAudioFormatFlagsCanonical;
	myPCMFormat.mChannelsPerFrame = 2;
	myPCMFormat.mFramesPerPacket = 1;
	myPCMFormat.mBitsPerChannel = 16;
	myPCMFormat.mBytesPerPacket = 4;
	myPCMFormat.mBytesPerFrame = 4;

	CheckResult (ExtAudioFileSetProperty(inputFile,
			kExtAudioFileProperty_ClientDataFormat,
			sizeof (myPCMFormat), &myPCMFormat),
		  "ExtAudioFileSetProperty failed");

	// allocate a big buffer. size can be arbitrary for ExtAudioFile.
	// you have 64 KB to spare, right?
	UInt32 outputBufferSize = 0x10000;
	void* ioBuf = malloc (outputBufferSize);
	UInt32 sizePerPacket = myPCMFormat.mBytesPerPacket;
	UInt32 packetsPerBuffer = outputBufferSize / sizePerPacket;

	// set up output file
	NSString *outputPath = [myDocumentsDirectory()
			stringByAppendingPathComponent:@"export-pcm.caf"];
	NSURL *outputURL = [NSURL fileURLWithPath:outputPath];
	NSLog (@"creating output file %@", outputURL);
	AudioFileID outputFile;
	CheckResult(AudioFileCreateWithURL((CFURLRef)outputURL,
		   kAudioFileCAFType,
		   &myPCMFormat,
		   kAudioFileFlags_EraseFile,
		   &outputFile),
		  "AudioFileCreateWithURL failed");

	// start convertin'
	UInt32 outputFilePacketPosition = 0; //in bytes

	while (true) {
		// wrap the destination buffer in an AudioBufferList
		AudioBufferList convertedData;
		convertedData.mNumberBuffers = 1;
		convertedData.mBuffers[0].mNumberChannels = myPCMFormat.mChannelsPerFrame;
		convertedData.mBuffers[0].mDataByteSize = outputBufferSize;
		convertedData.mBuffers[0].mData = ioBuf;

		UInt32 frameCount = packetsPerBuffer;

		// read from the extaudiofile
		CheckResult (ExtAudioFileRead(inputFile,
			  &frameCount,
			  &convertedData),
			 "Couldn't read from input file");

		if (frameCount == 0) {
			printf ("done reading from file");
			break;
		}

		// write the converted data to the output file
		CheckResult (AudioFileWritePackets(outputFile,
			   false,
			   frameCount,
			   NULL,
			   outputFilePacketPosition / myPCMFormat.mBytesPerPacket,
			   &frameCount,
			   convertedData.mBuffers[0].mData),
			 "Couldn't write packets to file");

		NSLog (@"Converted %ld bytes", outputFilePacketPosition);

		// advance the output file write location
		outputFilePacketPosition +=
			(frameCount * myPCMFormat.mBytesPerPacket);
	}

	// clean up
	ExtAudioFileDispose(inputFile);
	AudioFileClose(outputFile);

	// GUI update omitted
}

Note that this uses a CheckResult() convenience function that Kevin Avila wrote for our upcoming Core Audio book… it just looks to see if the return value is noErr and tries to convert it to a readable four-char-code if it seems amenable. It’s in the example file too.

Is It Soup Yet?

Does all this work? Rather than inspecting the AudioStreamBasicDescription of the resulting file, let’s do something more concrete. With Xcode’s “Organizer”, you can access your app’s sandbox on the device. So we can just drag the Application Data to the Desktop.

In the resulting folder, open the Documents folder to find export-pcm.caf. Drag it to QuickTime Player to verify that you do, indeed, have PCM data:

So there you have it. In several hundred lines of code, we’re able to get a song from the iPod Music Library, export it into our app’s Documents directory, and convert it to PCM. With the raw samples, you could now draw an audio waveform view (something you’d think would be essential for video editors who want to match video to beats in the music, but Apple seems dead-set against letting us do do with AV Foundation or QTKit), you could perform analysis or effects on the audio, you could bring it into a Core Audio AUGraph and mix it with other sources… all sorts of possibilities open up.

Clearly, it could be a lot easier. It’s a ton of code, and two file exports (library to .m4a, and .m4a to .caf), when some apps might be perfectly happy to read from the source URL itself and never write to the filesystem… if only they could. Having spent the morning writing this blog, I may well spend the afternoon filing feature requests on bugreport.apple.com. I’ll update this blog with OpenRadar numbers for the following requests:

• Allow Core Audio to open URLs provided by MediaLibrary’s MPMediaItemPropertyAssetURL
• AV Foundation should allow passthrough export of Media Library items
• AV Foundation export needs finer-grained control than just presets
• Provide sample-level access for AVAsset

Still, while I’m bitching and whining, it is remarkable that iOS 4 opens up non-DRM’ed items in the iPod library for export. I never thought that would happen. Furthermore, the breadth and depth of the iOS media APIs remain astonishing. Sometimes terrifying, perhaps, but compared to the facile and trite media APIs that the other guys and girls get, we’re light-years ahead on iOS.

Have fun with this stuff!

Disclaimer: there are aspects of the App Store approval process I find utterly appalling, particularly the stealth “no competing with Apple” unwritten rule.

Nevertheless, here’s a mental exercise for you. Consider the user-facing download-to-own software stores offered by the game consoles: Wii Ware, XBox Live Arcade, and PlayStation Network.

In what way is any Apple App Store policy more onerous than these stores’ developer policies?

Here are some links to help you research:

• To become an Authorized Developer for Wii, WiiWare and/or Nintendo DS/DSi
• WELCOME TO MICROSOFT CASUAL GAMES!
• SCE DevNet (almost utterly opaque… see also Developing for PS3 PlayStation Network (PSN))

Once we get past the much more expensive licensing, the pre-vetting of both product ideas (Microsoft: “Email mcg@microsoft.com with a description of your game and your contact information. We’ll send you a content submission form.”) and developers (Nintendo: “the authorization for Wii/WiiWare or Nintendo DS will be based upon your relevant game industry experience.”), and the fact that these platforms generally reject entire classes of applications (anything that isn’t a game)… I think it’s interesting to compare the anger and fury vented over the App Store, and consider that almost nobody is railing against these stores, even though they’re much more closed than Apple’s platform, and may collectively reach more users.

We might also do well to note how closed mobile development was before the iPhone. I know I’ve told this story before, but in a JavaOne conversation with O’Reilly people about how to get Java ME books moving, I said that everyone with an interest in ME (myself included) had figured out that getting your apps to end users was effectively impossible, and that with the network API often disabled for third-party apps, there wasn’t much point in writing ME apps anyways. My suggestion for an ME book that would move copies would be one which provided “the names, e-mails, and phone numbers of all the carrier and handset executives you’d have to go down on in order to get your apps on their phones.”

So, I spent first five months of this year on a grueling, panic-driven-development project on Mac OS X. As my longest single Mac engagement, I’m afraid this can’t help but wear down my enthusiasm for the platform.

It doesn’t help that what I was working on was well into the edge-case realm: our stuff needed to silently update itself in the background, which gets into the management of daemons, starting and stopping them at will. This isn’t bad in theory, but getting it to work consistently across 10.4 through 10.6 is grueling. An always-on daemon sets a RunAtLoad property in its /Library/LaunchDaemons plist in order to come up immediately and stay up. Uninstalling and/or replacing such a daemon is tricky, as launchd keeps track of what daemons it has loaded and thinks are still running. The worst thing to do is to just kill the process… instead, you need to use launchctl to load or unload the daemon as needed. And, as a root-owned process, you need to perform this launchctl as root. Oh, and you’d better not delete the plist before you unload the daemon, since launchctl takes the path to the plist, so doing things in the wrong order can leave you with rogue daemons you can’t unload. And if you don’t unload the old daemon, a new launchctl load does nothing, as launchd thinks the original daemon is still running.

Now throw in some user agents. These are like daemons, except that they run for each user at login, and are owned by the user. So to uninstall or update, you need to launchctl unload as each user. Which is possible with sudo -u username in a shell script, unless you’re on 10.4 and you get the user list from /usr/bin/users, as the 10.4 version truncates user names to 8 characters, breaking the sudo.

Oh, and who’s doing these unloads? A script in an .mpkg installer. Which is a whole ‘nother bundle of fun, given how fabulously broken is Package Maker, Apple’s utility for creating .pkg and .mpkg installers. Package Maker crashes frequently, doesn’t consistently persist (especially when source-controlled) settings for file ownership, and creates broken installers when invoked by the command-line utility /usr/bin/packagemaker, making it utterly unsuitable for use in Makefiles or other automated build processes. IMHO, Package Maker is as big an ass-muffin as I’ve ever seen come out of Cupertino, at least since that quickly-pulled iTunes 2 release that reformatted the host drive.

Working with all these taped-together technologies — desperately trying shell script voodoo in an .mpkg post-install step to make things right — eventually wore me out. And granted, this is an edge case: most Mac apps can be distributed as app bundles without installers, and most installers can require the user to restart if it’s making radical changes like installing or updating daemons (not an option for me because the installer runs in the background, called by another daemon). Still, it’s enough to make you long for the “curated”, locked-down walled garden of the App Store, whose distribution and update system really does work remarkably well.

With the Mac all but expelled from this week’s WWDC, some pundits are happy to declare the death of the Mac. That’s a silly overstatement, but it is sensible to accept that after a decade of improvement and marketing, the Mac is a mature platform, likely secure at its 10%-or-so market share. What else would we want Apple to do with the Mac? I’d like to see things fixed and cleaned up, dubious legacies cleared out and things set right. And Apple has done that. Problem is, the result is not Mac OS X 10.7, it’s iOS 4.

But then again, if the iPad replaces the laptop for some number of users, this is a brilliant way to grow Apple’s share — nibbling away at traditional computers cannibalizes the Mac somewhat, but chows away mostly at Windows.

Suffice to say I’m very happy to be re-orienting myself to iPhone/iPad/iPod-touch development. It’s got the feel of a somewhat clean start, based on proven technology, but not burdened by old breakage. Now I’ve got some catching up to do to adapt to all the changes in iOS 4.

Fanaticism consists in redoubling your effort when you have forgotten your aim.

George Santayana, Life of Reason (1905) vol. 1, Introduction

A few weeks back, I asked the Twitterverse “When did user-facing Linux become completely and officially irrelevant? Seems like a fait accompli, doesn’t it?”. With the OSS community’s reaction to the iPad, I’m now sure of it.

And by reaction, I’m of course thinking of the response by the FSF’s John Sullivan to Steve Jobs’ much-discussed Thoughts on Flash. Sullivan’s piece is a straightforward “pox on both your houses” take, damning Apple and Adobe alike for not adopting the One True Faith of Open Source Software.

I had allowed myself to hope that the OSS community would take the hugely successful launch of the iPad and the flight of developers to the iPhone OS platform, even with Apple’s cravenly self-serving App Store policies, as an opportunity to take a good, long look in the mirror. With iPad sales now already estimated at over a million, you have to wonder how soon use of this nascent platform will exceed that of desktop Linux, if it hasn’t already. What should the open source community think of this profound preference for the unfree over the free? I would think they’d see it as a very public, and very obvious repudiation of the party line, suggestive of an urgent need for some soul-searching and time to re-examine how the OSS community is engaging the world at large.

Sullivan’s piece, sadly, shows no such self-awareness. It’s the usual total detach from real people that we’ve come to expect from True Believers. Indeed, the last third of it is the all-too-familiar zealous denunciation of all proprietary software, calling on readers to adopt Linux and Ogg Theora.

Seriously, Ogg? They’re still clinging to the “everything should be in Ogg” kick? OK, let’s ask the people who should know on this one, content professionals, to see if they buy in at all. DV magazine’s website logs 0 hits in searches for Theora and for Ogg, StreamingMedia.com’s 11 hits (versus 248 for H.264) are largely related to codec politics and not mentions in training articles or product reviews, and legendary retailer B & H assumes the search term theora is a misspelling of “theory”. No matter how many essays and insults the zealots spew forth, they seem unable to generate any inkling to know or care about Theora among video professionals.

In effect, the FSF’s position is to tell the world “screw you for not wanting the stuff we produce”, oblivious to the fact that the marketplace of ideas is very clearly telling them “screw you for not producing the stuff we want.”

The quote at the top of this article was often cited by Chuck Jones in describing the character of Wile E. Coyote, who was never afforded a moment of clarity to reconsider his hunting regimen, the inherent dangers therein, and the prospects for a way of life that didn’t involve falling off cliffs every 30 seconds. I fear the OSS community is so wrapped up in their own self-righteousness, they are similarly immune to equally-needed moments of humility, clarity, and perspective.

At the beginning of the month, I spoke at 360iDev in San Jose. I’d wanted to do a go-for-broke Core Audio talk for a long time, sent in an ambitious proposal, and got accepted, so this was destined to be it. Now I just had to come up with something that didn’t suck.

Luckily, there was another Core Audio talk, from Robert Strojan, that did a high-level tour of audio on the iPhone, with two deep dives into OpenAL and Audio Units. So that got everyone ready.

I called my talk Core Audio: Don’t Be Afraid To Play It LOUD (from a caption in the liner notes to Matthew Sweet’s Girlfriend), and went with the no fear angle by focusing almost entirely on audio units. The idea being: you’ve seen these low-latency audio apps that do crazy stuff, you know it must be possible, so how? The answer is: you involve yourself in the processing of samples down at the audio unit level.

Click through the 170+ slides — it’s not that bad, I just expanded all the Keynote “builds” into their own slides — and you’ll get the basic grounding in the stuff you always have to do with audio units, and the Remote I/O unit in particular. Stuff like finding the component, getting an instance, setting properties on the input and output scopes to enable I/O and set an AudioStreamBasicDescription, things you will get as used to as implementing init and dealloc in an Obj-C class.

The big win is the examples, which came together in a hideous mess of two spaghetti code files that I’m embarrassed to say are now in the hands of all the attendees. One reason for the blog entry you’re reading is to present cleaned up versions of the five sample applications from the session.

In the past, I’ve done an audio unit example that produces a sine wave by cranking out samples in the callbacks. It’s sort of like the hello world of audio in that it gets you down to the nitty-gritty of samples without too much pain, but it doesn’t play to a crowd all that well. FWIW, the second chapter of the Core Audio book writes a sine wave to a file, again to get you thinking about samples as soon as possible.

But for this talk, I decided to do a set of examples that work with audio input. That way, we got to play with the mic and the speaker — bus 1 and bus 0 for you folks who already know this stuff — and get some halfway interesting audio.

The first example goes through the drudgery of creating and initializing the Remote I/O unit, and connects bus 1 output to bus 0 input to do a pass through: anything that comes in on the mic goes out the speakers. I used to do this with an AU Graph and AUGraphConnectNodeInput(), not realizing it’s easily done without a graph, by just setting a unit’s kAudioUnitProperty_MakeConnection property. With that, I could speak into the simulator and get audio out over the PA (or into my device’s mic, but I used the simulator because it shows better).

Well, yay, I’ve turned the iPhone simulator into a mediocre PA system. What’s next. The key to the good stuff is to be able to involve yourself in the processing of samples, so the next example replaces the direct connection with a render callback. This means we write a function to supply samples to a caller (the Remote IO unit, which needs something to play). In the basic version, we call AudioUnitRender() on the IO unit’s bus 1, which represents input into the device, to provide samples off the mic.

Still boring, but we’re getting there. Instead of just copying samples, example 3 performs a trivial effect by adding a gain slider, and applying that gain to every sample as it passes through.

Now we can apply any DSP that suits us as samples go through the render callback function. In example 4, we apply a ring modulator to the samples, which combined a 23 Hz sine wave with the input signal from the mic to create a reasonably plausible Dalek voice.

Dalek voice demo:

I was pretty much over time at this point, but the last example is too much fun to miss. To show off other units, I brought a multichannel mixer unit into play. On bus 0, it got a render callback to our existing Dalek code. For bus 1, I read an entire LPCM song into RAM (which is totally bad and would blow the RAM of earlier iPhones, but I couldn’t get the damned CARingBuffer working), and provided a render callback to supply its samples in order. The result, infamously, is “Dalek Sing-A-Long”:

Dalek-sing-a-long demo:

Anyways, great conference, great speakers, great attendees. Thanks for reading, and here’s the code:

• Slides (on slideshare.com)
• 1-AUPassThroughWithConnection.zip
• 2-AUPassThroughWithRenderCallback.zip
• 3-AUPassThroughWithGainEffect.zip
• 4-AUPassThroughWithDalekEffect.zip
• 5-DalekSingAlong.zip

Last weekend was the Voices That Matter: iPhone Developers Conference in Seattle, put on by Pearson, who’s publishing our Core Audio book. However, co-author Kevin Avila handled the Core Audio talk for this conference, so I took on two topics that I had fairly deep knowledge of, thanks to my work on Road Tip.

Core Location and Map Kit: Bringing Your Own Maps

This talk starts with a basic tour of Core Location’s more or less straightforward means of getting information about your current location, getting updates as the data gets better (i.e., GPS kicks in) or you move. Then we got into Map Kit and how to get map images of any given location. The sample app for this takes a list of Apple Store locations, which I converted from CSV to a plist and stuck in the app bundle, and shows you the nearest store to a given start location. Each time you hit a + button in the nav bar, it drops the next closest store as a pin on the map and re-scales the map so that all pins are visible.

This is where it gets good. Starting from a canned location at my house, the closest Apple Store is here in Grand Rapids. The next two that come up are outside Detroit (Ann Arbor and Novi). The fourth closest store is in Milwaukee. The comedy is when you see this on the map — Milwaukee is close only if you ignore the fact that going there would involve driving 100 miles straight across Lake Michigan. Since ferries across the lake are slow and expensive, and run only in Summer, you would probably drive through or around Chicago to get to Milwaukee… and go right past 7 Apple Stores in the process.

This is a common mistake to make — I forgot to show in my slides that the apple.com retail store finder does the same thing. Still, as-the-crow-flies distance calculations, paired with map images, can be a problem. Map Kit doesn’t know anything about roads, bodies of water, geographic features, political borders, etc… all it does is serve up images, and provide a touch UI to interact with them.

The last third of the talk is about “bringing your own maps”, meaning integration with third party map data providers to get some navigational intelligence into your app. The final code sample uses the MapQuest directions web service to get actual driving distances to the first few hits, and to keep the list of locations sorted in order by driving distance. This not only keeps me from going to Milwaukee, it even smartens up the earlier results: being right on I-96, the Novi store is now my second closest result, and as it turns out, even Indianapolis is a shorter drive than going around the lake to Milwaukee.

• Slides (on slideshare.com)
• SimpleCoreLocationDemo.zip
• AppleStorePOIMapper.zip
• MQAppleStorePOIMapper.zip

In-App Purchase: Best/Worst Thing Ever

My second talk was on In-App Purchase. In many ways, it covered the same hard-earned experience that I covered in An In-App Purchase Brain Dump. I spent a little time on both the iPhone Provisioning Portal (to set up an AppID, authorize it for I-AP, and provide enough of an app submission to create purchase objects, without actually going into the review queue), and on iTunes Connect (creating purchase products). The sample app simulated an online social game, minus the game, in which users might be able to purchase digital goods like virtual clothes for their avatar. The example offered a tabbed UI with a blank view for the game, a table of purchased objects, and a table for the store. When you tap “store”, the app collects the available products with a call to Store Kit, and tapping one of them kicks off the purchase. If the purchase goes through, the item is added to the inventory page.

Of course, this is easier and demos better because it uses non-consumable products, which have always been the best-understood and best-supported class of I-AP products. I did weigh in against the still-broken subscriptions, and how they have to be restored to a user’s many devices, even though restoreCompletedTransactions doesn’t support subscriptions, and nothing in Store Kit gives you a key you can associate with the user to save the purchase on your own server.

Erica Sadun talked with me before this session and mentioned an interesting workaround. Get your user to purchase a non-consumable item, and persist its transactionId. When they purchase subscriptions, log the purchase and this other transactionId on your server. Then, when they restore on another device, they’ll get this non-consumable back and its transactionId, which you can then use to query your server for subscription purchases. Depending on the nature of your app, and how well you hide the clunkiness from the user, this could be a very viable workaround.

That said, I still think I-AP subscriptions suck and hope that Apple deprecates them soon.

The last part of the talk covers of doing commerce without I-AP, which is more viable than you might think. If you’re already selling digital goods from a web store, you can continue to do so, and make your iPhone app a rich client to your web-vended content. In this case, you’re already doing everything that I-AP offers, so there’s no reason to give Apple a 30% cut. For example, the various e-bookstores within Stanza don’t use I-AP — they go out to websites for O’Reilly, All Romance Books, etc., to complete the purchase. This might not be allowed for an iPhone-only app, but where the goods are available in multiple forms, it only makes sense for Apple to allow an iPhone app to be a rich client to such a store’s goods.

My final example is streaming anime: Crunchyroll sells premium content subscriptions on their website, and their free iPhone app lets you use the same login to get the same content. By contrast, The Anime Network has premium subscriptions on their website and via I-AP in their paid iPhone app, and credentials aren’t shared between the two. Worse, the iPhone app offers fewer videos for the same price. Their use of I-AP is bad for them (they’re paying Apple to process payments they’re already capable of handling), and bad for their users. WTF?

• Slides (on slideshare.com)
• VTMPurchaseDemo.zip

Coming next: my 360iDev slides, and the much-anticipated cleaned-up Core Audio sample code