Processing and Flash


Processing And Flash

Lately I’ve been goofing around a bit with some interesting code.

Basically I started thinking wouldn’t it be nice if you could render some stuff in processing and send it to Flash. But how? SOCKET SERVERS!!!

Basically almost like having a “streaming server” that renders stuff on the fly. However if it’s just streaming out 3d stuff why not just pre-render it and stream it through an actual streaming server.

Well socket servers can have two way communication. So I’ve written some code for a basic server in Processing that receives a boolean and rotation X and rotation Y renders out a cube with the proper rotation and sends it back to Flash. Flash receives the data as a ByteArray which I turn into a Bitmap and display it.

As seen here:
Processing sending images to Flash through a socket server

I’m still not so sure about the feasibility of this small project but it’s an interesting one. At least for me… The following code really needs some work still. In fact when I switch the server to render out a Sphere instead of a cube Flash does not know what file type it’s casting the ByteArray to. My guess is that it doesn’t receive all the bytes before I try to cast it.

Download the source from here. Or check it out below.

Here is the code for Flash:

package {
  import flash.display.Bitmap;
  import flash.display.Loader;
  import flash.display.Sprite;
  import flash.events.Event;
  import flash.events.IOErrorEvent;
  import flash.events.ProgressEvent;
  import flash.net.Socket;
  import flash.utils.*;
 
  [SWF(width="980", height="570", framerate="30", backgroundColor="#000000")]
  public class JavaSocketListener extends Sprite
  {
    //Boolean for if we are connected to the socket server
    private var isConnected:Boolean=false; 
 
    //Socket to the processing socket server
    private var socket:Socket;
 
    //Bitmap that will display the 'PNG' sent back from Processing
    private var bitmap:Bitmap;
 
    //This loader is used to cast the ByteArray sent from Processing to a Bitmap
    private var loader:Loader; 
 
    //Rotations which will be sent to processing
    private var rotX:Number=0;
    private var rotY:Number=0;
 
    public function JavaSocketListener()
    {
      //Connect to the server
      socket=new Socket("localhost", 3333);
 
      //Event for when we are connected
      socket.addEventListener(Event.CONNECT, connected); 
 
      //The server has closed the connection
      socket.addEventListener(Event.CLOSE, closed); 
 
      //Picks up errors from the socket IE. Server is not up
      socket.addEventListener(IOErrorEvent.IO_ERROR, error);
 
      //Data received from the server. In our case sending the rendered image
      socket.addEventListener(ProgressEvent.SOCKET_DATA, receivedData);
    }
 
    private function connected(ev:Event):void
    {
      isConnected=true;
 
      //send the rotation data to the server and wait to receive a file back
      send();
    }
 
    private function closed(ev:Event):void
    {
      trace("HAS CLOSED");
      isConnected=false;
    }
 
    private function error(ev:Event):void
    {
      trace("THERE WAS AN ERROR: "+ev);
    }
 
    private function send():void
    {
      //Check if we are already connected to the server
      if(isConnected)
      {
        //set the rotation values for the server
        rotX=mouseY/stage.stageHeight*(Math.PI*2);
        rotY=mouseX/stage.stageWidth*(Math.PI*2);
 
        //Boolean for the server to check if it should render
        socket.writeBoolean(true);
 
        //Send the rotations to the server
        socket.writeFloat(rotX);
        socket.writeFloat(rotY);
 
        //This clears whats being sent to the server and lets the server
        //know we are done sending data (you must have this here)
        socket.flush();
      }
    }
 
    private function receivedData(ev:ProgressEvent):void
    {
      var bArray:ByteArray=new ByteArray();
 
      //read the bytes for the server into the bytearray
      socket.readBytes(bArray); 
 
      loader=new Loader();
 
      //add the event listener for when the bytes have "finished" loading
      //(basically when we can get something out of loader)
      loader.contentLoaderInfo.addEventListener(Event.COMPLETE, bytesLoaded); 
 
      //load in the bytearray so we can cast it to a Bitmap
      loader.loadBytes(bArray);
    }
 
    private function bytesLoaded(ev:Event):void
    {
      //check if the bitmap is initiliazed
      //if so just swap the bitmapdata it's displaying
      if(bitmap)
      {
        bitmap.bitmapData=(loader.content as Bitmap).bitmapData;
      }
      else
      {
        bitmap=new Bitmap((loader.content as Bitmap).bitmapData);
 
        bitmap.x=stage.stageWidth/2-bitmap.width/2;
        bitmap.y=stage.stageHeight/2-bitmap.height/2;
 
        addChild(bitmap);
      }
 
      send();
    }
  }
}

And here is the code for Processing:

import processing.opengl.*;
import processing.net.*;
 
import java.io.ByteArrayOutputStream;
import javax.imageio.ImageIO;
import java.awt.image.BufferedImage;
import javax.swing.ImageIcon;
import java.io.ByteArrayInputStream;
import java.io.DataInputStream;
 
Server s; //server that sends image to flash
 
//rotations for the cube
float rotX = 0;
float rotY = 0;
 
//if an image should be sent to a client
boolean sendImage=false;
 
void setup()
{
  size(450, 255, OPENGL);
 
  //creates the server thats going to send
  //images to flash
  s = new Server(this, 3333);
}
 
void draw()
{
  //get a client if they are sending something
  Client client=s.available();
 
  if(client!=null)
  {
    try
    {
      //get the bytes from the stream
      ByteArrayInputStream byte_in = new ByteArrayInputStream (client.readBytes());
      //send it to a data input stream so we can get some variables out of it
      DataInputStream data_in = new DataInputStream (byte_in);
 
      //get the data
      sendImage=data_in.readBoolean();
      rotX=data_in.readFloat();
      rotY=data_in.readFloat();
    }
    catch(IOException e)
    {
      //should write a better catch here
      println("ERROR: "+e);
    }
  }
 
  //do some drawing
  background(0);
  directionalLight(255, 255, 255, 0, 0, -1);
  noStroke();
  fill(255, 0, 255);
  translate(width/2.0, height/2.0, -100);
  rotateX(rotX);
  rotateY(rotY);
  smooth();
  box(height/2);
 
  //check if we need to send an image
  if(sendImage)
  {
    //get the "stage" to be sent out
    PImage img=get();
 
    //The following is to convert a PImage to a byte array
    //using some functions i found through google
    Image _image=img.getImage();
    byte[] bArray=imageToByteArray(toBufferedImage(_image));
 
    //send out the image as a byte array to the clients
    s.write(bArray);
  }
}
 
/****************************************************/
/***THE FOLLOWING FUNCTIONS I FOUND THROUGH GOOGLE***/
/*****BASICALLY THEY'RE USED TO CONVERT A PIMAGE*****/
/******************TO A BYTE ARRAY*******************/
/****************************************************/
public byte[] imageToByteArray(BufferedImage _image)
{
  ByteArrayOutputStream baOut = new ByteArrayOutputStream();
 
  try
  {
    ImageIO.write(_image, "png", baOut);
  }
  catch(IOException e)
  {
  }
 
  return baOut.toByteArray();
}
 
public static BufferedImage toBufferedImage(Image image)
{
  if (image instanceof BufferedImage) {return (BufferedImage)image;}
 
  // This code ensures that all the pixels in the image are loaded
  image = new ImageIcon(image).getImage();
 
  // Determine if the image has transparent pixels
  boolean hasAlpha = hasAlpha(image);
 
  // Create a buffered image with a format that's compatible with the screen
  BufferedImage bimage = null;
 
  GraphicsEnvironment ge = GraphicsEnvironment.getLocalGraphicsEnvironment();
 
  try
  {
    // Determine the type of transparency of the new buffered image
    int transparency = Transparency.OPAQUE;
 
    if (hasAlpha == true) {transparency = Transparency.BITMASK;}
 
    // Create the buffered image
    GraphicsDevice gs = ge.getDefaultScreenDevice();
 
    GraphicsConfiguration gc = gs.getDefaultConfiguration();
 
    bimage = gc.createCompatibleImage(image.getWidth(null), image.getHeight(null), transparency);
  }
 
  catch (HeadlessException e) {} //No screen
 
  if (bimage == null)
  {
    // Create a buffered image using the default color model
 
    int type = BufferedImage.TYPE_INT_RGB;
 
    if (hasAlpha == true) {type = BufferedImage.TYPE_INT_ARGB;}
 
    bimage = new BufferedImage(image.getWidth(null), image.getHeight(null), type);
  }
 
  // Copy image to buffered image
  Graphics g = bimage.createGraphics();
 
  // Paint the image onto the buffered image
  g.drawImage(image, 0, 0, null);
  g.dispose();
 
  return bimage;
}
 
public static boolean hasAlpha(Image image)
{
  // If buffered image, the color model is readily available
  if (image instanceof BufferedImage)
  {
    BufferedImage bimage = (BufferedImage)image;
 
    return bimage.getColorModel().hasAlpha();
  }
 
  // Use a pixel grabber to retrieve the image's color model;
  // grabbing a single pixel is usually sufficient
  PixelGrabber pg = new PixelGrabber(image, 0, 0, 1, 1, false);
  try
  {
    pg.grabPixels();
  }
  catch (InterruptedException e) {}
 
  // Get the image's color model
  ColorModel cm = pg.getColorModel();
  return cm.hasAlpha();
}

8 Comments, Comment or Ping

  1. Interesting idea. I’m actually interested in the idea of using Processing as a workhorse in general for creating data to send off to Flash. In some ways this seems similar to what PixelBender does.

    October 27th, 2008

  2. Mikko Haapoja

    Yep… Totally possible this is what I was thinking doing too. Some stuff with sound maybe. We’ll see.
    I would dig to see what you come up with.

    October 27th, 2008

  3. Just in case anyone else has an issue, it appears that PImage.getImage() appears in Processing 0149. Anything earlier will give you a “Cannot find a function “getImage()” in class PImage.

    November 21st, 2008

  4. Sorry Brent. I probably should have noted what version of Processing I was using.

    I’ve had problems playing around with other people’s code and versioning in Processing too. :(

    I guess that’s what you get with something thats super beta still.

    BTW… Looked at your site Brent. Looks like you’re playing quite heavily with stuff I wish I had time to look at. Maybe I can get an Arduino for Christmas. ;)

    November 22nd, 2008

  5. I can’t recommend the Arduino enough.

    Your code was particularly interesting to me since I’ve been looking for a way to set up a socket connection from Processing (running a terminal client interfacing with an XBee) to Flash, which would perform some visualization to the web. Your code was very helpful, thanks!

    November 22nd, 2008

  6. No prob man… Glad it helped you out.

    November 23rd, 2008

  7. jon

    Hi there! I’m trying to send midi messages from a midi controller/keyboard to flash, do you think this could be made by sending those signals with processing? I mean, if you were able to send visual data, this should be possible.. the thing is that i dont know how can i start this processing/flash comunication… thanks!

    November 17th, 2009

Reply to “Processing and Flash”