Android Live Streaming courtesy of JavaCV and FFMPEG

For the last little or should I say, long while I have been working on wrangling a solution for live streaming from Android that is both decent quality and extensible. For those of you interested, the litter in my GitHub account documents various previous attempts.

As far as I can tell, most folks that are streaming live video from Android are relying upon the video captured by the MediaRecorder and underlying classes and doing a bit of trickery with the file either while it sent to the server or on the server for distribution. This is fine but it doesn’t give hooks into the actual video frames before they are encoded and sent over the network.

Recently, I came across Samuel Audet’s amazing open source JavaCV project. In that project Samuel is wrapping FFMPEG’s underlying libraries (avcodec, avformat, and so on) using his equally amazing JavaCPP project to expose their functionality to any Java application.

Finally, after a few weeks of experimentation and little (actually a LOT) of help from Samuel himself, I have something working!

Running the live streaming app on a Galaxy Camera
Running the live streaming app on a Galaxy Camera

App and resulting stream on desktop via Wowza Media Server and Flash

There is a quick example of writing a file up on the JavaCV site which provides the foundation: https://code.google.com/p/javacv/source/browse/samples/RecordActivity.java

I have the beginnings of a full blown project (which needs some updating based on the above example) up on GitHub: https://github.com/vanevery/JavaCV_0.3_stream_test

Bringing in a generic H.264 stream to Wirecast (via Wowza and Wirecast’s Generic IP Camera support)

Wirecast is truly a studio in a box. It has a great support for multiple cameras, mixing live and recorded sources, graphic overlays and so on. Recent versions even allow you to bring in live feeds from IP cameras including support for specific Axis cameras.

Since I am a big fan of IP cameras and Axis in particular this is great news. Unfortunately Wirecast doesn’t have direct support for most models and I had to dig quite a bit to get things to work using their “Generic” IP camera support.

First test was to get a straight H.264 encoded into Wowza and then out to Wirecast. To do this, I used the Flash Media Live Encoder and set it publish to “rtmp://localhost:1935/img” (I have Wowza running on my local machine and an application called “img” which is a copy of Wowza’s “live” application). I set the Stream name in FMLE to “media.sav” which is what Wirecast is looking for by default.

In Wirecast’s Source Settings, I added a new IP Camera and set it’s IP address to: “127.0.0.1:1935” and choose “Generic” as the type.

Viola! It works, the video is being captured and encoded by FMLE, sent to Wowza and pulled into Wirecast as a Generic IP camera. In this manner, I can have live cameras via FMLE from anywhere in the world brought live into my final stream.

(Big thanks to Steve McFarlin the developer of the LiveU iPhone broadcasting app for his post on Wirecast’s Forum detailing how he got his software working.)

Local Report 2012 | Creative Time Reports

The piece that I helped Robert Whitman create is up on Creative Time.  

We had 90 or so callers send in video and make phone calls via a custom iPhone app, Android app and regular phone number over the course of an hour.

Check it out: http://creativetimereports.org/2012/10/18/local-report/

Open for Business

Walking Productions provides software development and consulting services. Appealing projects are those that deal with online and mobile media (audio/video). Get in touch: vanevery@walking-productions.com

Android Application Development
Flash Video Players, including P2P (Adobe Stratus)
Wowza Media Server Plugin/Module Development
Flash Media Server Development
Development related to Axis IP Cameras
QuickTime/Darwin Streaming Solutions
Audio and Video Encoding/Transcoding Pipelines
iPhone Application Development
JME/J2ME Application Development
Asterisk and VoIP Application Development
Phone call to streaming applications
Voicemail to Blog/CMS
Podcasting Systems
Mobile and Microblogging Solutions
SMS Campaign Management Software
2 Screen Interactive Television Applications (Enhanced TV)
EBIF iTV Application Development
HTML 5 Video Player Development
Media Asset Management Systems
AJAX/JavaScript/DHTML Development
LAMP Application Development (Linux, MySQL, PHP)
Java Desktop Application Development
Mobile Video Capture, Sharing and Playback Applications
Live Mobile Video Streaming
Computer Vision Applications in Java and Flash
Flash Video Capture
Location Aware Mobile Applications
Video Indexing, Searching, Recommendation Engine and Presentation Systems
Network Controlled Devices
WordPress and Drupal Plugin/Module Development
Flash Lite Application Development
AIR/ActionScript 3 Application Development
WebService Integration and Development (XML-RPC, SOAP, REST)
Podcasting (Audio/Video) Solutions
MP3 Streaming Servers
MMS Gateway Solutions
Java and AJAX Chat Application Development
Interactive Whiteboard Applications

Flash Media Server Sending Images

One of my students in my live web class is developing an interesting application that sends screenshots to other people. I put together some sample code to help him along and thought this would be of general interest.

Using the Flash Media Server with Remote Shared Objects this can be built. Here is a walk through of the code:

First of all, this uses the JPGEncoder class from the AS3CoreLib so you probably want to grab that and import it.

	import com.adobe.images.JPGEncoder;

This example uses a SharedObject, a NetConnection and NetStreams for sending the video as well as the screen shots. Once the NetConnection is established, the SharedObject can be setup:

        // Listener for connection
        private function connectionHandler (e:NetStatusEvent):void 
        { 
        	// If we are connected
             if (e.info.code == "NetConnection.Connect.Success") 
             {
				// Set up the shared object
				// We'll call it SimpleSO, pass in the app url and not make it persistent
				sharedObject = SharedObject.getRemote("SimpleSO",nc.uri,false);
				
				// Tell the shared object to call methods in this class if requested
				sharedObject.client = this;				
				
				// Add a listener for when shared object is changed
			   	sharedObject.addEventListener (SyncEvent.SYNC,syncEventCallBack); 

				// Connect the shared object to our netConnection
				sharedObject.connect(nc);
				
				// All of the video streaming setup
				doEverything();
				
				// Register mouseclicks, how we'll determine when to send a frame
				stage.addEventListener(MouseEvent.MOUSE_DOWN, saveFrame);
             } 
        }

Here is the method that is called when the mouse is clicked. It creates a bitmapdata object, encodes as a JPEG and sends that as a bytearray through the shared object:

        private function saveFrame(e:MouseEvent):void
        {        
        	// Save the frame to the bitmapdata object
        	var bmpd:BitmapData = new BitmapData(320,240);
        	bmpd.draw(videoIn1);
                	
        	// First encode as JPEG so it is smaller
			var jpgEncoder:JPGEncoder = new JPGEncoder(100);
			var jpgByteArray:ByteArray = jpgEncoder.encode(bmpd);
			
			// Send it via the shared object
			sharedObject.send("newBitmap",jpgByteArray);
        }

The SharedObject.send method calls the function “newBitmap” on everyone who is connected and passes in the jpgByteArray. The newBitmap function uses the Loader to uncompress the JPG and when it is done calls “gotBitmapData”:

		public function newBitmap(jpgByteArray:ByteArray):void
		{
			var loader:Loader = new Loader();
			loader.contentLoaderInfo.addEventListener(Event.COMPLETE, gotBitmapData)
			loader.loadBytes(jpgByteArray);
		}

gotBitmap data just creates a regular Bitmap and displays it:

	private function gotBitmapData(e:Event):void
	{
		var decodedBitmapData:BitmapData = Bitmap(e.target.content).bitmapData

        	// Create the bitmap image
        	var bmp:Bitmap = new Bitmap(decodedBitmapData);
        	
        	// Add it to the stage
        	bmp.x = 0;
        	bmp.y = 240;
        	addChild(bmp);			
	}

Here is the full AS3 class (it could be improved but it works):

package
{
	// Import JPEGEncoder Class from: http://code.google.com/p/as3corelib/
	import com.adobe.images.JPGEncoder;

	import flash.display.Sprite; 
	import flash.display.MovieClip; 
	import flash.events.NetStatusEvent; 
	import flash.net.NetConnection; 
	import flash.net.NetStream; 
	import flash.media.Camera; 
	import flash.media.Microphone; 
	import flash.media.Video; 
	import flash.display.Bitmap;
	import flash.display.BitmapData;
	import flash.events.MouseEvent;
	import flash.utils.ByteArray;
	import flash.net.SharedObject; 
  	import flash.events.SyncEvent; 
  	import flash.events.Event;
	import flash.display.Loader;

	public class VideoCapture extends Sprite
	{
		// Shared Object for communication
   		private var sharedObject:SharedObject;

 		// Overall NetConnection for communicating with FMS
        private var nc:NetConnection; 
        
        // RTMP URL, same as directory on FMS
        private var rtmpURL:String = "rtmp://xxxx/webcam";
         
        // NetStreams for each stream 
        private var netStreamOut:NetStream; 
        private var netStreamIn1:NetStream; 
        
        // Camera
        private var camera:Camera; 
        // Microphone
        private var microphone:Microphone; 
                
        // My Video
        private var videoOut:Video; 
        
        // Video Components
        private var videoIn1:Video; 
        
        // Stream Names
        private var outStream:String; 
        private var inStream1:String; 

        public function VideoCapture() 
        { 
        	// Construct NetConnection and connect to server
             nc = new NetConnection(); 
             nc.connect(rtmpURL); 
             
             // Add a listener for connection
             nc.addEventListener (NetStatusEvent.NET_STATUS,connectionHandler); 
        } 
        
        
        // Listener for connection
        private function connectionHandler (e:NetStatusEvent):void 
        { 
        	// If we are connected
             if (e.info.code == "NetConnection.Connect.Success") 
             {

				// Set up the shared object
				// We'll call it SimpleSO, pass in the app url and not make it persistent
				sharedObject = SharedObject.getRemote("SimpleSO",nc.uri,false);
				
				// Tell the shared object to call methods in this class if requested
				sharedObject.client = this;				
				
				// Connect the shared object to our netConnection
				sharedObject.connect(nc);
				
				// All of the video streaming
				doEverything();
				
				// Register mouseclicks, how we'll determine when to send a frame
				stage.addEventListener(MouseEvent.MOUSE_DOWN, saveFrame);
             } 
        }
        
        // Gets the results from the server
        private function doEverything():void 
        { 
        	// Name of streams
			outStream="one"; 
			inStream1="one"; 
        
			// Setup the camera        
			camera = Camera.getCamera(); 

			// setup the microphone             
			microphone = Microphone.getMicrophone(); 

			// Video components
			videoOut = new Video(); 
    		videoIn1 = new Video();

			// Set positions 
			videoOut.x = 0;
			videoOut.y = 0;
			videoIn1.x = 320;
			videoIn1.y = 0;

			// Add them to the screen
			addChild(videoOut);
			addChild(videoIn1);

            // Publish our stream
            netStreamOut = new NetStream(nc); 
            netStreamOut.attachAudio(microphone); 
            netStreamOut.attachCamera(camera); 
            netStreamOut.publish(outStream, "live"); 
             
            // Attach camera to our video 
            videoOut.attachCamera(camera); 
             
            //Play incoming streamed video 
            netStreamIn1 = new NetStream(nc); 

			videoIn1.attachNetStream(netStreamIn1);

			netStreamIn1.play(inStream1);
        }
        
        private function saveFrame(e:MouseEvent):void
        {        
        	// Save the frame to the bitmapdata object
        	var bmpd:BitmapData = new BitmapData(320,240);
        	bmpd.draw(videoIn1);
                	
        	// First encode as JPEG so it is smaller
			var jpgEncoder:JPGEncoder = new JPGEncoder(100);
			var jpgByteArray:ByteArray = jpgEncoder.encode(bmpd);
			
			// Send it via the shared object
			sharedObject.send("newBitmap",jpgByteArray);
        }
        
		
		public function newBitmap(jpgByteArray:ByteArray):void
		{
			var loader:Loader = new Loader();
			loader.contentLoaderInfo.addEventListener(Event.COMPLETE, gotBitmapData)
			loader.loadBytes(jpgByteArray);
		}
		
		private function gotBitmapData(e:Event):void
		{
			var decodedBitmapData:BitmapData = Bitmap(e.target.content).bitmapData

        	// Create the bitmap image
        	var bmp:Bitmap = new Bitmap(decodedBitmapData);
        	
        	// Add it to the stage
        	bmp.x = 0;
        	bmp.y = 240;
        	addChild(bmp);			
		}
	} 
}