Occasionally, my iSight stopped working, even in the middle of a Skype video session, with the camera in full control of Skype. Any effort to re-enable the camera failed with a "Camera in use by another application" message. I could not wrap my head around the issue, until I investigated a bit and I think I managed to understand what's going on.

Occasionally, during video chat, Skype detaches from the camera in order to switch resolution, probably to scale it accordingly to current bandwidth conditions. During this split second, it may happen that the google talk browser plugin takes over the camera, and never relinquishes it. The camera is now stuck with google talk, Skype can't grab it anymore, and so for all other camera applications. The only solution appears to be a reboot.

I was able to fix the problem by issuing

ps uaxw |grep Firefox

which may produce the following output

sbo     21173   0.0  0.1   414040   2492   ??  S     5May11   0:07.47
        Contents/MacOS/plugin-container /Library/Internet Plug-Ins/
        googletalkbrowserplugin.plugin -omnijar /Applications/
        Contents/MacOS/omni.jar 21168 gecko-crash-server-pipe.21168
        org.mozilla.machname.855559451 plugin

The solution is to kill this process with the appropriate pid (in this case, 21173).

kill 21173

this frees the camera without a reboot.