Playing with Events in the Cloud

One more video of a talk I gave at OW2Con 2012 about a research project I am working on since two years now called Play (and not the play framework…).

« The PLAY project develops an elastic and reliable architecture for dynamic and complex, event-driven interaction in large highly distributed and heterogeneous service systems.
Such an architecture will enable ubiquitous exchange of information between heterogeneous services, providing the possibilities to adapt and personalize their execution, resulting in the so-called situational-driven process adaptivity… »

Playing in Brussels

Last week was the first annual review meeting of the Play project I work on since one year. I am involved at several levels in this project: from the architecture point of view, to the software integration and quality ones. On my side, my goal is to provide the efficient software infrastructure for events actors, or how to build an Event Driven Architecture based on Petals Distributed Service Bus. There are others points which have to be developed, especially all the platform governance stuff and Service Level Agreement for events, what we call Event Level Agreement.

We showed several things to the European Commission reviewers and we were also able to show an early prototype (this one was originally planned to be show at mid project ie in 6 months…). I made a video capture of the demo, which really needs to be explained…

  • There is the idea of a market place for events : The event marketplace. From there users are able to subscribe to things they are interested in. For now it is just a simple Web application which subscribe on behalf of the user to events through topics. This subscription is sent to the Play paltform using Web standards. Here we use OASIS Web service Notification to create this communication link.
  • Events are collected from several sources by events adapters. In the video above, we can see the user setting this FB status which is collected by the Play system and transformed into a notification which is published to the platform. There are also some Twitter adapters and Pachube ones which collects data in real time and publish them to the platform. This time again, we use Web standards for adapters communication.
  • Now, what happens in the video? The user logs in to event marketplace and subscribes to FB events. The user then publishes something to its FB wall (note that it is not mandatory to have the same user in the event marketplace and in FB. A user can subscribe to FB events and receive events from all the FB statuses collected from the FB application users). After the event propagation delay, the event marketplace display the event to the user.

So do we need such machinery to do things like that? No, if you want to do some other simple mashup portal. There are several components which are under active development: Storage and processing. Yes we store all events in the Play platform. This storage will be huge, but it is one of the project goal: Providing efficient and elastic storage in some P2P way. The need for this storage comes with the other important aspect of the project: Complex Event Processing. We will soon be able to create complex rules on top of events and be able to generate notifications based on past and real time notifications because we have efficient storage and real time stuff inside the platform. I am not an expert of this domain, so I can not give more details about that point but capabilities are huge! For example, we can express something like « Hey Play, can you send me a SMS me when there is my favorite punk rock band playing just around and I am not on a business trip and X and !Y or Z ». All of this intelligence coming from the processing of various sources I push since months in the platform coming from Twitter, FB, and other data providers.

Now let’s take some time to work on my OW2Con talk. The session name is pretty cool : Open Cloud Summit Session.

Service Bus Live Monitoring

I explained in the last articles how I tested the Play Framework, Web sockets and how I integrated all this nice stuff with a real example based on a Service Bus, Web services Notifications, etc…

This time, let’s go one step further. We have a Service Bus which is Web service notification enabled like last time. We can bind services to the bus, expose service endpoints as Web services, blahblahblah… But, this time, I am interested on having some real time monitoring of service invocations. It means that each time a message goes through the service bus (a service invocation in fact), I want to know (almmost) immediatly the service response time.
Hopefuly, the PetalsLink Distributed Service Bus I develop and use provides many extension points. One is the capability to add modules to the routing engine ie the software module each message must be able to go through on service request and response. So adding some router module which catch all the messages, timestamp them and then send this monitoring data to someone is quite easy. At the implementation level, this monitoring router module publishes monitoring reports to the service bus notification engine topic dedicated to monitoring stuff.

So, a client interested in monitoring data just has to register itself as subscriber to the monitoring notification topic. Every time a message is published in the topic, it will be delivered to all the subscribers. Up to the subscriber to display data as soon as it can. This is where Play, Web sockets and some cool javascript library came in. Since I never developed javascript stuff, I tried to find an easy to integrate solution to create some moving plots, asking twitter. I finally found the Smoothie Chart library which is really easy to use and updates graph in real time.

The high level architecture of the system can be defined as

The following video shows the result of the complete stack: Each time a message a service is invoked with SOAPUI, a Web service notification is sent to a Play application which subscribed to the monitoring topic, the Play application then pushes the data to the client by using a Web socket. Finally, the javascript code on the client side feeds the Smoothie chart which updates automatically. At the end, it is quite simple and efficient.

Oh, I forgot to say something: This took me 2 or 3 hours to create all this stuff… The code has been published on github in the dsbmanager-webapp project.

Play Project Kick Off

C’est l’époque à laquelle les feuilles tombent, mais c’est aussi celle où les nouveaux projets commencent… La semaine se tenait la réunion de démarrage du projet de recherche Play à Karlsruhe – Allemagne. Suite naturelle pour moi du projet SOA4All qui se termine dans les prochains mois, le projet Play (qui n’a aucun lien avec le Framework Web Play, très bon soit dit en passant) va permettre de mettre continuer mes travaux sur la fédération des architectures SOA, du Cloud et de plancher sérieusement sur les architectures événementielles.

Un scénario de présentation de la plate forme du projet est le suivant :

Paul is a businessman who has been flying from Paris to New York. He used the entertainment service on board, but hasn’t finished watching the movie before the landing. Two hours later he is entering his room in the downtown hotel he booked earlier and wow: the room entertainment service is ready to PLAY the movie Paul was watching in the plane – of course only the unfinished part.

Vu comme ça, rien (ou presque) de révolutionnaire en soit. Quoi que, en y réfléchissant bien et en pensant à la plate forme qui pourrait supporter de telles choses, on peut se demander comment on peut effectivement savoir que Paul va pouvoir regarder son film depuis sa chambre d’hôtel alors qu’il était en train de le regarder dans l’avion 2 heures plus tôt? Est ce que l’on peut alors parler de SOA, de cloud, de fédération, de gouvernance, d’adaptation, d’EDA, de CEP, …? Oui! Bref, l’architecture àvenir sera probablement certainement détaillée ici dans quelques temps, le temps de mettre tout dans le bon ordre… Il sera tout de même déjà question de mettre en place et d’étendre les produits que nous développons, que ce soit Petals ESB ou Petals Master pour ne citer qu’eux.

The PLAY project will develop and validate an elastic and reliable architecture for dynamic and complex, event-driven interaction in large highly distributed and heterogeneous service systems. Such an architecture will enable ubiquitous exchange of information between heterogeneous services, providing the possibilities to adapt and personalize their execution, resulting in the so-called situational-driven process adaptivity.

Bref, un beau projet qui commence pour moi, avec des partenaires experts dans leur domaine et beaucoup de choses à apprendre… Pour le moment je me demande si la prochaine réunion se tiendra encore dans un pays Bierophone (après la Belgique et l’Allemagne ces deux dernières semaines, question légitime) et par contre je ne doute pas que les discussions seront toujours aussi intéressantes, que ce soit chacun derrière son ordinateur ou autour d’une bonne table.