[Time code];

Well, that came together a lot faster than I thought it would.

Recap: the old QuickTime SequenceGrabber limits you to capturing from one video device at a time. Leopard’s QTKit capture classes free you from this restriction, allowing you to capture, simultaneously, from however many devices you happen to have available.

Only thing is, the tutorial only shows how to work with a single device at one time. So it took a little experimenting getting this working.

As I mentioned in earlier installments, I decided to do an app where each window would have its own preview and would allow you to pick from the available devices. The QTCaptureView is associated with a single QTCaptureSession object, so I decided to do a document-based Cocoa application, with the MyDocument class responsible for delegating to the session, and handling UI events (i.e., acting as a controller).

I should still do a heck of a clean-up on the code, as there are a lot of commented-out false starts, and no effort whatsoever to manage memory properly. Actually, if QTKit capture is Leopard only, I could punt and turn on garbage collection. Anyways, while I’ll release the whole code once I add more features (capturing audio and recording to disk), here are the important parts, for the benefit of anyone who finds this via Google.

• awakeFromNib

- (void)awakeFromNib {
    NSLog(@"awakeFromNib!");

    // create session and find default device
    captureSession = [[QTCaptureSession alloc] init];
    NSLog(@"got QTCaptureSession %@", captureSession);
    defaultDevice = [self getDefaultCaptureDeviceForMediaType: QTMediaTypeVideo];

    // build video source popup and select default
    [self buildNamesToDevicesDictionaryForMediaType:QTMediaTypeVideo
                                      includeMuxDevices:YES];
    [videoDevicePopUp removeAllItems];
    [videoDevicePopUp addItemsWithTitles: [namesToDevicesDictionary allKeys]];
    [videoDevicePopUp selectItemWithTitle: defaultDeviceMenuTitle];

    // interesting: programmatic selection of popup doesn't send a selection event.
    // ok then...
    [self chooseVideoDevice: self];
}

When a new window is created, I create a new QTCaptureSession and look up all the available devices (see next section). I use this to populate the NSPopupMenu and select the default device. Actually, setting the current item in the menu programatically doesn’t call the IBAction like a user action would, so I have to call that method manually here.

• buildNamesToDevicesDictionaryForMediaType

- (void)buildNamesToDevicesDictionaryForMediaType: (NSString *)mediaType
                             includeMuxDevices:(BOOL)includeMux {
    if (namesToDevicesDictionary == nil) {
        namesToDevicesDictionary = [[NSMutableDictionary alloc] init];
    }
    [namesToDevicesDictionary removeAllObjects];

    // add default device first
    QTCaptureDevice *defaultDevice =
              [self getDefaultCaptureDeviceForMediaType: mediaType];
    // create an item called "Default (device_name)"
    defaultDeviceMenuTitle =
        [NSString stringWithFormat: @"Default (%@)",
                  [defaultDevice localizedDisplayName]];
    [namesToDevicesDictionary setObject: defaultDevice
                     forKey: defaultDeviceMenuTitle];

    // then find the rest
    NSArray* devicesWithMediaType =
              [QTCaptureDevice inputDevicesWithMediaType:mediaType];
    NSArray* devicesWithMuxType =
               [QTCaptureDevice inputDevicesWithMediaType:QTMediaTypeMuxed];
    NSMutableSet* devicesSet =
               [NSMutableSet setWithArray: devicesWithMediaType];
    [devicesSet addObjectsFromArray: devicesWithMuxType];

    // add all devices from set to dictionary
    NSEnumerator *enumerator = [devicesSet objectEnumerator];
    id value;
    while ((value = [enumerator nextObject])) {
        QTCaptureDevice *device = (QTCaptureDevice*) value;
        [namesToDevicesDictionary setObject: device
                     forKey: [device localizedDisplayName]];
    }

}

This method builds an NSDictionary mapping device names to QTCaptureDevices for every discovered device for a given media type (such as QTMediaTypeVideo). It optionally grabs “muxed” devices, those that send audio and video over the same stream. It adds the default device as an extra item in this menu. I should probably put the default device at the top of the list, and sort the rest of the devices alphabetically.

• getDefaultCaptureDeviceForMediaType

- (QTCaptureDevice*)getDefaultCaptureDeviceForMediaType: (NSString *)mediaType {
    NSLog (@"getDefaultCaptureDevice");
    // set up the default device
    QTCaptureDevice *foundDevice =
        [QTCaptureDevice defaultInputDeviceWithMediaType: mediaType];
    NSLog (@"got default device %@", foundDevice);
    // try for a muxed device (eg, a dv camcorder) if that was nil
    if (foundDevice == nil)
        foundDevice = [QTCaptureDevice defaultInputDeviceWithMediaType:
                                      QTMediaTypeMuxed];
    return foundDevice;
}

This is a convenience method to get the default audio or video device, called by buildNamesToDevicesDictionaryForMediaType, above. Looking at it now, I suppose I should make the fall-back-to-muxed-devices block optional.

• chooseVideoDevice

- (IBAction)chooseVideoDevice:(id)sender {
    NSLog (@"chooseVideoDevice");

    // stop the session while we mess with it?
    [captureSession stopRunning];

    NSString* chosenDeviceName = [videoDevicePopUp titleOfSelectedItem];

    NSLog (@"choose device %@", chosenDeviceName);
    QTCaptureDevice *chosenDevice =
        (QTCaptureDevice*) [namesToDevicesDictionary objectForKey:
                   chosenDeviceName];
    if (chosenDevice == nil) {
        NSLog (@"couldn't find %@ in dictonary", chosenDeviceName);
        return;
    }
    NSLog (@"looked up device %@", chosenDevice);

    // remove any existing inputs
    NSEnumerator* inputEnum = [[captureSession inputs] objectEnumerator];
    NSLog (@"removing %d existing inputs", [[captureSession inputs] count]);
    id value;
    while ((value = [inputEnum nextObject])) {
        QTCaptureInput *input = (QTCaptureInput*) value;
        [captureSession removeInput: input];
    }

    // add an input for this device
    NSError** openError = nil;
    [chosenDevice open: openError];
    if (openError != nil) {
        NSLog (@"Can't open %@: %@", chosenDevice, openError);
        return;
    }

    QTCaptureDeviceInput* deviceInput =
           [QTCaptureDeviceInput deviceInputWithDevice: chosenDevice];
    NSLog (@"Got an input");

    NSError** addInputError = nil;
    [captureSession addInput: deviceInput error: addInputError];
    NSLog (@"added input to session");

    // TODO: move to awakeFromNib?
    [capturePreviewView setCaptureSession: captureSession];
    NSLog (@"set session on view");

    // (re-)start session
    [captureSession startRunning];

Magic time. This is called initially from awakeFromNib with the default device, then again whenever the user makes a selection from the video device popup menu. It stops the QTCaptureSession temporarily (actually I need to take that out and see if it’s really necessary; I did it here because you had to stop the old SequenceGrabber when you dicked with it), removes any existing QTCaptureInputs, gets a new input from the selected device, connects that input to the session, and restarts the session.

And that’s about it

That’s pretty much all the code that matters in MyDocument.m This is surely buggy and needs review and testing before using it for anything more serious than screwing around: I’m completely careless about opening and closing devices, haven’t checked whether one session’s use of a device will break another’s, etc. And I’ve got debugging NSLog statements all over the place. I also noticed a couple of these in my log after device switches:

CoreImage: detected malformed affine matrix:
AFFINE [nan nan nan inf nan nan] ARGB_8

But it’s an experiment, not an ADC article. And it’s proven to me that QTKit’s multiple-camera support works, and really well.

Camera tour

As for the cameras used in this test, here’s a shot of my desk:

1. A MacAlly IceCam – A USB 1.1 video capture device, one of the few that ships with drivers for Mac and Windows. Since it’s a low-speed device, the image quality and framerate are dreadful.
2. An original Apple iSight, the external FireWire variety that you can’t buy anymore.
3. A Logitech QuickCam for Notebooks Pro, a USB Video Class device that I blogged about on O’Reilly a while back.