
Lately I’ve been goofing around a bit with some interesting code.
Basically I started thinking wouldn’t it be nice if you could render some stuff in processing and send it to Flash. But how? SOCKET SERVERS!!!
Basically almost like having a “streaming server” that renders stuff on the fly. However if it’s just streaming out 3d stuff why not just pre-render it and stream it through an actual streaming server.
Well socket servers can have two way communication. So I’ve written some code for a basic server in Processing that receives a boolean and rotation X and rotation Y renders out a cube with the proper rotation and sends it back to Flash. Flash receives the data as a ByteArray which I turn into a Bitmap and display it.
As seen here:

I’m still not so sure about the feasibility of this small project but it’s an interesting one. At least for me… The following code really needs some work still. In fact when I switch the server to render out a Sphere instead of a cube Flash does not know what file type it’s casting the ByteArray to. My guess is that it doesn’t receive all the bytes before I try to cast it.
Download the source from here. Or check it out below.
Here is the code for Flash:
package { import flash.display.Bitmap; import flash.display.Loader; import flash.display.Sprite; import flash.events.Event; import flash.events.IOErrorEvent; import flash.events.ProgressEvent; import flash.net.Socket; import flash.utils.*; [SWF(width="980", height="570", framerate="30", backgroundColor="#000000")] public class JavaSocketListener extends Sprite { //Boolean for if we are connected to the socket server private var isConnected:Boolean=false; //Socket to the processing socket server private var socket:Socket; //Bitmap that will display the 'PNG' sent back from Processing private var bitmap:Bitmap; //This loader is used to cast the ByteArray sent from Processing to a Bitmap private var loader:Loader; //Rotations which will be sent to processing private var rotX:Number=0; private var rotY:Number=0; public function JavaSocketListener() { //Connect to the server socket=new Socket("localhost", 3333); //Event for when we are connected socket.addEventListener(Event.CONNECT, connected); //The server has closed the connection socket.addEventListener(Event.CLOSE, closed); //Picks up errors from the socket IE. Server is not up socket.addEventListener(IOErrorEvent.IO_ERROR, error); //Data received from the server. In our case sending the rendered image socket.addEventListener(ProgressEvent.SOCKET_DATA, receivedData); } private function connected(ev:Event):void { isConnected=true; //send the rotation data to the server and wait to receive a file back send(); } private function closed(ev:Event):void { trace("HAS CLOSED"); isConnected=false; } private function error(ev:Event):void { trace("THERE WAS AN ERROR: "+ev); } private function send():void { //Check if we are already connected to the server if(isConnected) { //set the rotation values for the server rotX=mouseY/stage.stageHeight*(Math.PI*2); rotY=mouseX/stage.stageWidth*(Math.PI*2); //Boolean for the server to check if it should render socket.writeBoolean(true); //Send the rotations to the server socket.writeFloat(rotX); socket.writeFloat(rotY); //This clears whats being sent to the server and lets the server //know we are done sending data (you must have this here) socket.flush(); } } private function receivedData(ev:ProgressEvent):void { var bArray:ByteArray=new ByteArray(); //read the bytes for the server into the bytearray socket.readBytes(bArray); loader=new Loader(); //add the event listener for when the bytes have "finished" loading //(basically when we can get something out of loader) loader.contentLoaderInfo.addEventListener(Event.COMPLETE, bytesLoaded); //load in the bytearray so we can cast it to a Bitmap loader.loadBytes(bArray); } private function bytesLoaded(ev:Event):void { //check if the bitmap is initiliazed //if so just swap the bitmapdata it's displaying if(bitmap) { bitmap.bitmapData=(loader.content as Bitmap).bitmapData; } else { bitmap=new Bitmap((loader.content as Bitmap).bitmapData); bitmap.x=stage.stageWidth/2-bitmap.width/2; bitmap.y=stage.stageHeight/2-bitmap.height/2; addChild(bitmap); } send(); } } }
And here is the code for Processing:
import processing.opengl.*; import processing.net.*; import java.io.ByteArrayOutputStream; import javax.imageio.ImageIO; import java.awt.image.BufferedImage; import javax.swing.ImageIcon; import java.io.ByteArrayInputStream; import java.io.DataInputStream; Server s; //server that sends image to flash //rotations for the cube float rotX = 0; float rotY = 0; //if an image should be sent to a client boolean sendImage=false; void setup() { size(450, 255, OPENGL); //creates the server thats going to send //images to flash s = new Server(this, 3333); } void draw() { //get a client if they are sending something Client client=s.available(); if(client!=null) { try { //get the bytes from the stream ByteArrayInputStream byte_in = new ByteArrayInputStream (client.readBytes()); //send it to a data input stream so we can get some variables out of it DataInputStream data_in = new DataInputStream (byte_in); //get the data sendImage=data_in.readBoolean(); rotX=data_in.readFloat(); rotY=data_in.readFloat(); } catch(IOException e) { //should write a better catch here println("ERROR: "+e); } } //do some drawing background(0); directionalLight(255, 255, 255, 0, 0, -1); noStroke(); fill(255, 0, 255); translate(width/2.0, height/2.0, -100); rotateX(rotX); rotateY(rotY); smooth(); box(height/2); //check if we need to send an image if(sendImage) { //get the "stage" to be sent out PImage img=get(); //The following is to convert a PImage to a byte array //using some functions i found through google Image _image=img.getImage(); byte[] bArray=imageToByteArray(toBufferedImage(_image)); //send out the image as a byte array to the clients s.write(bArray); } } /****************************************************/ /***THE FOLLOWING FUNCTIONS I FOUND THROUGH GOOGLE***/ /*****BASICALLY THEY'RE USED TO CONVERT A PIMAGE*****/ /******************TO A BYTE ARRAY*******************/ /****************************************************/ public byte[] imageToByteArray(BufferedImage _image) { ByteArrayOutputStream baOut = new ByteArrayOutputStream(); try { ImageIO.write(_image, "png", baOut); } catch(IOException e) { } return baOut.toByteArray(); } public static BufferedImage toBufferedImage(Image image) { if (image instanceof BufferedImage) {return (BufferedImage)image;} // This code ensures that all the pixels in the image are loaded image = new ImageIcon(image).getImage(); // Determine if the image has transparent pixels boolean hasAlpha = hasAlpha(image); // Create a buffered image with a format that's compatible with the screen BufferedImage bimage = null; GraphicsEnvironment ge = GraphicsEnvironment.getLocalGraphicsEnvironment(); try { // Determine the type of transparency of the new buffered image int transparency = Transparency.OPAQUE; if (hasAlpha == true) {transparency = Transparency.BITMASK;} // Create the buffered image GraphicsDevice gs = ge.getDefaultScreenDevice(); GraphicsConfiguration gc = gs.getDefaultConfiguration(); bimage = gc.createCompatibleImage(image.getWidth(null), image.getHeight(null), transparency); } catch (HeadlessException e) {} //No screen if (bimage == null) { // Create a buffered image using the default color model int type = BufferedImage.TYPE_INT_RGB; if (hasAlpha == true) {type = BufferedImage.TYPE_INT_ARGB;} bimage = new BufferedImage(image.getWidth(null), image.getHeight(null), type); } // Copy image to buffered image Graphics g = bimage.createGraphics(); // Paint the image onto the buffered image g.drawImage(image, 0, 0, null); g.dispose(); return bimage; } public static boolean hasAlpha(Image image) { // If buffered image, the color model is readily available if (image instanceof BufferedImage) { BufferedImage bimage = (BufferedImage)image; return bimage.getColorModel().hasAlpha(); } // Use a pixel grabber to retrieve the image's color model; // grabbing a single pixel is usually sufficient PixelGrabber pg = new PixelGrabber(image, 0, 0, 1, 1, false); try { pg.grabPixels(); } catch (InterruptedException e) {} // Get the image's color model ColorModel cm = pg.getColorModel(); return cm.hasAlpha(); }
8 Comments, Comment or Ping
Interesting idea. I’m actually interested in the idea of using Processing as a workhorse in general for creating data to send off to Flash. In some ways this seems similar to what PixelBender does.
October 27th, 2008
Yep… Totally possible this is what I was thinking doing too. Some stuff with sound maybe. We’ll see.
I would dig to see what you come up with.
October 27th, 2008
Just in case anyone else has an issue, it appears that PImage.getImage() appears in Processing 0149. Anything earlier will give you a “Cannot find a function “getImage()” in class PImage.
November 21st, 2008
Sorry Brent. I probably should have noted what version of Processing I was using.
I’ve had problems playing around with other people’s code and versioning in Processing too.
I guess that’s what you get with something thats super beta still.
BTW… Looked at your site Brent. Looks like you’re playing quite heavily with stuff I wish I had time to look at. Maybe I can get an Arduino for Christmas.
November 22nd, 2008
I can’t recommend the Arduino enough.
Your code was particularly interesting to me since I’ve been looking for a way to set up a socket connection from Processing (running a terminal client interfacing with an XBee) to Flash, which would perform some visualization to the web. Your code was very helpful, thanks!
November 22nd, 2008
No prob man… Glad it helped you out.
November 23rd, 2008
Hi there! I’m trying to send midi messages from a midi controller/keyboard to flash, do you think this could be made by sending those signals with processing? I mean, if you were able to send visual data, this should be possible.. the thing is that i dont know how can i start this processing/flash comunication… thanks!
November 17th, 2009
Reply to “Processing and Flash”