Real life intervenes again (parsing PDF, whee!) and I have to cut short a planned epic iDevBlogADay entry. But I do want to bang out a few quick notes on various topics of interest.
The first is Core Audio in iOS 5, which we can now talk about publicly. If we go through the iOS 4.3 to iOS 5.0 API Differences document, we see that Audio Units accounts for a large number of diffs. This comes from the addition of a suite of units that finally make programming at the unit level a lot more interesting. Whereas we used to get a single effects unit (AUiPodEQ), we now get high and low pass filters, the varispeed unit, a distortion box, and a parametric EQ that lets us play with sliders instead of the “canned” EQ settings like “Bass Booster” and “Spoken Word”. Even more useful, we get the AUFilePlayer, meaning you can now put audio from a file at the front of an AUGraph, instead of the sheer pain of having to decode your own samples and pass them to the AUGraph through a CARingBuffer.
iOS also gets the AUSampler unit introduced in Lion, which provides a MIDI-controlled virtual instrument whose output is pitch-shifted from a source sample. This was shown off at WWDC, although providing the source to the unit by means of an .aupreset is still a dark (undocumented) art. This is the first actual MIDI audio unit in iOS, which makes the presence of Core MIDI more useful on the platform.
Core Audio giveth, but Core Audio also taketh away: iOS 5 removes (not deprecates) VoiceIOFarEndVersionInfo. This struct, and its related constants (specifically kVoiceIOFarEndAUVersion_ThirdParty), were documented as interoperating with a hypothetical “3rd-party device following open FaceTime standards”, something I took note of last May as possibly meaning that FaceTime was still ostensibly being opened up. With the removal of these APIs, I think that closes the book on Apple having any intention to live up to its vow to publish FaceTime as an open standard.
There’s lots more to talk about, but I’m already over my allotted blogging time, and work beckons. Perhaps you’d like to hear me speaking about this stuff and demo’ing it? I’m doing an all-new-for-iOS-5 Core Audio talk at two upcoming conferences:
- Voices That Matter: iOS Developers Conference, Nov. 12-13 in Boston, and $150 off with discount code
BSSPKR5. - CocoaConf, Dec. 2-3 in Raleigh, NC, which has early-bird pricing available through Oct. 31.
I’ll also be doing a talk about AV Foundation capture at these conferences. And back on audio, I just heard from my editor that the last three chapters of Core Audio should be in the Rough Cut on Safari Online Books in the next week or so, although I still have some work to do to clean up bits that are broken on Lion (read the sidebar on Audio Components if you’re having a problem creating audio units with the old Component Manager calls) and to clear out forward references to stuff that didn’t end up making the final cut for the book.
I’m a half-day late with my iDevBlogADay post… sorry.
So I was thinking about conference panels recently, something I don’t often attend or participate in. Panels to me seem like something that should work better than they usually do. You have smart, interesting people, but unless they know to “play ball”, to go out of their way to find ways to dig deeper or draw out conflicts and differences between each other, you tend to end up with a lot of head-nodding and personal pet theories that the rest of the panel doesn’t really have a stake in.
It’s not clear that the audience gets a lot out of it either. At Cocoaconf, I was on an iOS developer panel and the first question we got was the hopelessly played out “how do I get my app noticed” one. Ugh. You don’t need a panel for that, we’ve all been griping about that for three damn years now, and if we don’t have good answers yet, we’re never going to. Moreover, I’m not sure that attendees have a good sense of the potential of panels and how they can draw that out.
So here’s a solution. It comes to us by way of the fine folks at Harmonix, makers of Rock Band, Dance Central, the new iOS novelty VidRhythm, the rare iPod nano/Classic game Phase, etc. At their last two panels at PAX, they did a “Reverse Q&A”, which works like this: the panelists either ask big poll-type questions of the room, ask followup questions and get shouted-out responses from the crowd, or they ask “man on the street” style questions to whoever is at the front of the line for the mic. Either way, the topic is then followed up by the panelists and whoever from the crowd happens to be at the front of the line for the mics.
It still seems like a work-in-progress on the Harmonix podcasts, but there is a gem of a great idea here. Anyone who’s working in iOS and attending conferences has something interesting to say, and probably some unique real-world perspectives that wouldn’t necessarily be obvious to the kind of people that get picked for panels. We’re all self-employed hipster indies and authors, so we likely have little if any idea how iOS is playing out in big enterprises, how well or poorly it rubs shoulders with other technologies, etc. So in a Reverse Q&A Panel, I could ask these kinds of questions of whoever is first at the mic: “what do you use iOS for… how’s that working out… what’s missing that you think should be there…”
The responses we would get from the attendees would drive panel discussion, and in a sense, the person at the front of the line for the mic becomes a temporary member of the panel. In this, it’s a lot like the “open chair panel” that I’ve seen pulled off only once (at the Java Mobility conference in Jamuary 2008, where I saw the last gasp of the old world prior to the iPhone SDK announcement a few weeks later).
And I still like the format of both the Reverse Q&A and the Open Chair Panel more than I like straight-up open spaces, which at the end of the day are just chats, and chatting is best done over food and drink, like at the end of Cocoaconf where Bill Dudney, Scott Ruth and I grabbed two guys from Ohio U. that Bill had met and headed down to Ted’s for some bison burgers. That’s chattting. If you’re going to schedule a time and a room, it’s already more formal, and a structure helps set expectations.
I’m inclined to talk up Reverse Q&A as a format to the Cocoaconf and CodeMash organizers… would like to give this a try in the next few months.
And speaking of which, let’s practice. Here are some questions I’d like to ask of Reverse Q&A attendees. Feel free to answer any of them in the comments. I’d like to know what you guys and girls are thinking:
- Do you learn new platforms, languages, and frameworks from books, blogs, official docs, or what? (I want to know so I can figure out whether I should bother writing books anymore… signs point to no)
- What do other platforms do better than iOS?
- What’s the one App Store policy that pisses you off the most?
- Do you sell your own apps, write apps for someone else (employer, contract clients, etc.) or something else? Which of these do you think makes the most sense for you?
- Do you want more or less webapps in your life?
OK, you guys and girls talk for a while…
I hopped in on the MIDI chapter of the nearly-finished Core Audio book because what we’ve got now is a little obscure, and really needs to address the most obvious questions, like “how do I hook up my MIDI hardware and work with it in code?” I haven’t taken MIDI really seriously in the past, so this was a good chance to catch up.
To keep our focus on iOS for this blog, let’s talk about MIDI support in iOS. iOS 4.2 added CoreMIDI, which is responsible for connecting to MIDI devices via physical cables (through the dock connector) or wifi (on OSX… don’t know if it works on iOS).
Actually getting the connection to work can be touchy. Start with the Camera Connection Kit‘s USB connector. While Apple reps are typically quick to tell you that this is not a general-purpose USB adapter, it’s well-known to support USB-to-MIDI adapters, something officially blessed (with slides!) in Session 411 (“Music in iOS and Lion”) at WWDC 2011.
The catch is that the iPad supplies a tiny amount of power out the dock connector, not necessarily enough to power a given adapter. iOS MIDI keeps an updated list of known-good and known-bad adapters. Price is not a good guide here: a $60 cable from Best Buy didn’t work for me, but the $5 HDE cable works like a charm. The key really is power draw: powered USB devices shouldn’t need to draw from the iPad and will tend to work, while stand-alone cables will work if and only if they eschew pretty lights and other fancy power-draws. The other factor to consider is drivers: iOS doesn’t have them, so compatible devices need to be “USB MIDI Class”, meaning they need to follow the USB spec for driver-less MIDI devices. Again, the iOS MIDI Devices List linked above is going to help you out.
For keys, I used the Rock Band 3 keyboard, half off at Best Buy as they clear out their music game inventory (man, I need to get Wii drums cheap before they become collector’s items). This is
Once you’ve got device, cable, and camera connection kit, try playing your keys in GarageBand to make sure everything works.
If things are cool, let’s turn our attention to the Core MIDI API. There’s not a ton of sample code for it, but if you’ve installed Xcode enough times, you likely have Examples/CoreAudio/MIDI/SampleTools/Echo.cpp, which has a simple example of discovering connected MIDI devices. That’s where I started for my example (zip at the bottom of this blog).
You set up a MIDI session with MIDIClientCreate(), and make your app an input device with MIDIInputPortCreate(). Both of these offer callback functions that you set up with a function pointer and a user-info / context that is passed back to your function in the callbacks. You can, of course, provide an Obj-C object for this, though those of you in NDA-land working with iOS 5 and ARC will have extra work to do (the term __bridge void* should not be unfamiliar to you at this point). The first callback will let you know when devices connect, disconnect, or change, while the second delivers the MIDI packets themselves.
You can then discover the number of MIDI sources with MIDIGetNumberOfSources(), get them as MIDIEndpointRef‘s with MIDIGetSource(), and connect to them with MIDIPortConnectSource(). This connects your input port (from the previous graf) to the MIDI endpoint, meaning the callback function specified for the input port will get called with packets from the device.
MIDIPackets are tiny things. The struct only includes a time-stamp, length, and byte array of data. The semantics fall outside of CoreMIDI’s responsibilities; they’re summarized in the MIDI Messages spec. For basic channel voice messages, data is 2 or 3 bytes long. The first byte, “status”, has a high nybble with the command, and a low nybble indicating which MIDI channel (0-16) sent the event. The remaining bytes depend on the status and the length. For my example, I’m interested in the NOTE-ON message (status 0x9n, where n is the channel). For this message, the next two bytes are called “data 1″ and “data 2″ and represent the rest of the message. The bottom 7 bits of data 1 identify the note as a number (the high bit is always 0), while the bottom 7 bits of data 2 represent velocity, i.e., how hard the key was hit.
So, a suitable callback that only cares about NOTE-ON might look like this:
static void MyMIDIReadProc (const MIDIPacketList *pktlist,
void *refCon,
void *connRefCon)
{
MIDIPacket *packet = (MIDIPacket *)pktlist->packet;
Byte midiCommand = packet->data[0] >> 4;
// is it a note-on
if (midiCommand == 0x09) {
Byte note = packet->data[1] & 0x7F;
Byte veolocity = packet->data[2] & 0x7F;
// do stuff now...
So what do we do with the data we parse from MIDI packets? There’s nothing in Core MIDI that actually generates sounds. On OSX, we can use instrument units (kAudioUnitType_MusicDevice), which are audio units that generate synthesized sounds in response to MIDI commands. You put the units in an AUGraph and customize them as you see fit (maybe pairing them with effect units downstream), then send commands to the instrument units via the Music Device API, which provides functions like MusicDeviceMIDICommand, and takes the unit, and the status, data1 and data2 bytes from the MIDI packet, along with a timing parameter. Music Device isn’t actually in Xcode’s documentation, but there are adequate documentation comments in MusicDevice.h. On OSX, the PlaySoftMIDI example shows how to play notes in code, so it’d be straight-forward to combine this with CoreMIDI and play through from MIDI device to MIDI instrument: get the NOTE-ON events and send them to the instrument unit of your choice.
On iOS, we don’t currently have instrument units, so we need to do something else with the incoming MIDI events. What I decided to do for my example was to just call System Sounds with various iLife sound effects (which should be at the same location on everyone’s Macs, so the paths in the project are absolute). The example uses 4 of these, starting at middle C (MIDI note 60) and going up by half-steps.
To run the example, you’ll actually have to run it twice: first to put the app on your iPad, then stop, plug in your keyboard, and run again. It might just be easier to watch this demo:
Anyways, that’s a brief intro to CoreMIDI on iOS. The book will probably skew a little more OSX, simply because there’s more stuff to play with, but we’ll make sure both are covered. I’m also going to be getting into this stuff at CocoaHeads Ann Arbor on Thursday night.
- Sample code:CoreMidiToSystemSoundExample.zip
Trying something different for my two 90-minute AV Foundation presentations at CocoaConf today, I decided to do my presentation entirely from the iPad with the VGA adapter… no laptop.
In some ways, it was an easy choice to make: I’d already done the slides in Keynote, so running the same presentation off the iPad only necessitated changing a few fonts (I usually code in Incosolata, which required a change to Courier). If anything, the app demos worked better, since running apps in the simulator precludes showing off anything particular to the device hardware, such as accelerometers, location, or (in my case) video capture. In fact, as soon as you touch the AV Foundation capture APIs in a project, you lose the ability to build for the Simulator.
Downsides include the fact that I couldn’t hop into source on Xcode, so any important code needed to be in slides (I continue to hope for Xcode for iPad, though its painful performance on my 4GB MacBook has dashed those hopes somewhat). Still, I remain impressed that I can get so much done with just the iPad… probably my biggest disappointment today was having to use the official WordPress app to upload this entry’s picture, as WordPress always destroys user data, and even if I do need it only for uploading pictures from the iPad, I don’t need the pictures more than I need the blog itself (how can it be that the WordPress app is as bad as it is?)
One thing to keep in mind for iPad-only presentations is that you cannot charge while the VGA cable is plugged in, so you need to start with enough battery power to get through your presentation. That said, it’s not hard: between two 90-minute talks, I drained my battery from 90% to 55%. The battery certainly seemed likely to outlast both my voice and my legs, so that’s not a problem.
I’m happy to leave the laptop behind whenever possible, and will probably do so at my next conference. Speaking of which, the Voices That Matter: iOS Developers’ Conference is coming to Boston on Nov. 12-13, and you can get $150 off with my speaker code: BSSPKR5.
Tomorrow is the second and final day for CocoaConf, which has a shockingly deep and thorough collection of talks for a first time conference. Nice to have another good Mac/iOS conference in this part of the country.
Update 8/23/11 – Here, after much delay, are links to the slides and sample code from my CocoaConf presentation.
- Introduction to AV Foundation (CocoaConf, Aug ’11) (slides from slideshare.net)
- Advanced AV Foundation (CocoaConf, Aug ’11) (slides from slideshare.net)
- VTM_Player.zip, VTM_AVRecPlay.zip, VTM_AVEditor.zip – see Slides and stuff from Voices That Matter talk
- VTM_iPodReader.zip – see From iPod Library to PCM Samples in Far Fewer Steps Than Were Previously Necessary
- VTM_AVEffects.zip – see Wrap up from Voices That Matter iPhone, Spring 2011
- CCFScreenRecorder.zip
- Zxing – see “Barcodes” app in the zxing project on Google Code
Quick reminder: Early Bird pricing ($350) ends Friday (July 22) for CocoaConf in Columbus, Ohio. It’s a new conference, but it has put together a very impressive set of speakers, most recently adding Bill Dudney, former Apple evangelist and co-author on my revised iOS SDK Development book.
I’m doing two talks on AV Foundation. These may draw on the three talks I’ve done at the Voices That Matter conferences, but if I have time, I’m considering rebooting them from the “tour the APIs” format to a “one big project” format. Maybe something along the lines of “Let’s Write QuickTime Player for iPhone” and “Let’s Write Final Cut Pro for iPad”. We’ll see… don’t want to over-promise when I’m already buried with programming work and two books.
Oh, and Columbus has my favorite chain restaurant, a holdover from my time in Atlanta, and my former employer at One CNN Center. Just sayin’, in case plans are needed for Saturday night. Ahem.
Speaking announcement: I’ll be doing two talks at a new OSX/iOS conference, CocoaConf, being held in Columbus, OH on August 12 and 13.
I’m going to do two talks, introductory and advanced AV Foundation. Obviously, they’ll probably be a lot like what I’ve already done at Voices That Matter, except that the advanced talk I did in Seattle was about 1/3 recap, whereas this is all going to be in the same track, so it really can be a monster immersion in two parts.
It’s not settled whether sessions are going to be 60 or 90 minutes… I keep going long at VTM’s 75, so 90 would be fine. If that’s the case, then my two sessions are going to amount to a three-hour intro to AV Foundation. So we should be able to cover a lot of ground and get really deep into the tough stuff.
