BPEL support in Petals DSB

As promised in the last article about my talk at OW2Con 2011 last week, here is a video on something I was not able to show due to some low resolution problems. The video is a bit long but shows several things (in the right order):

1. The DSB Manager Web application is used to manage the Distributed Service Bus. It uses the DSB Web service API to interact with node instances running somewhere…
2. The DSB Manager is used to bind business services to the DSB (let’s forget JBI, the user does not care about it…). DSB services are also exposed. Every DSB node provides the same business API with the help of the distributed endpoint registry it uses.
3. The DSB Manager uses the DSB BPEL API to deploy BPEL processes to the DSB. Up to the DSB to use the right internal endpoint when the process is executed. Services can be hosted on any node, it is the role of the DSB to route messages to the right endpoint on the right node. The BPEL process is exposed as Web service and can be invoked by any Web service client. Here I just use SOAPUI client.
4. We can monitor what happens when invoking a service! For now the DSB Manager uses Web service notification to subscribe to some monitoring topic hosted on the DSB node. When a message is exchanged between the client and the services involved in the process execution, notification are automatically published to the DSB Manager which has just subscribed. The monitoring uses Web sockets for live display in the browser…
5. Last thing is just a test to show more monitoring data when many calls are exchanged between consumers and providers.

Let’s go one step further… The BPEL engine we use in the DSB is our own (PetalsLink) BPEL engine we developed from scratch. This allows us to have a complete control on it and to be able to extend it and embed it as we want without any constraint. In the current case, the BPEL Engine is hosted on a dedicated DSB component. It means that we do not have an external thing which talk with services through some exposed services. This is really important to notice that by doing such thing we can really base process execution on a Service Oriented Architecture. When developing the BPEL process with the Petals Studio, or when creating a BPM process (more details in a future post), you do not have to care about service endpoints. You just have to say to the process that you want to call operation X of service Y or interface Z. It is up to the DSB hosting the BPEL engine to resolve endpoints at runtime. By using this approach we can really do interesting things, just because the DSB is Distributed: services can be hosted on any nodes, can be replicated, can move, can be updated without any impact on the process itself: Oh wait this is SOA!

Related articles

• Back From OW2Con 2011 (chamerling.org)
• Some Play Framework, Service Bus, WS Notification and Web Sockets… (chamerling.org)
• Let’s talk at OW2Con 2011 (chamerling.org)

Playing in Brussels

Last week was the first annual review meeting of the Play project I work on since one year. I am involved at several levels in this project: from the architecture point of view, to the software integration and quality ones. On my side, my goal is to provide the efficient software infrastructure for events actors, or how to build an Event Driven Architecture based on Petals Distributed Service Bus. There are others points which have to be developed, especially all the platform governance stuff and Service Level Agreement for events, what we call Event Level Agreement.

We showed several things to the European Commission reviewers and we were also able to show an early prototype (this one was originally planned to be show at mid project ie in 6 months…). I made a video capture of the demo, which really needs to be explained…

• There is the idea of a market place for events : The event marketplace. From there users are able to subscribe to things they are interested in. For now it is just a simple Web application which subscribe on behalf of the user to events through topics. This subscription is sent to the Play paltform using Web standards. Here we use OASIS Web service Notification to create this communication link.
• Events are collected from several sources by events adapters. In the video above, we can see the user setting this FB status which is collected by the Play system and transformed into a notification which is published to the platform. There are also some Twitter adapters and Pachube ones which collects data in real time and publish them to the platform. This time again, we use Web standards for adapters communication.
• Now, what happens in the video? The user logs in to event marketplace and subscribes to FB events. The user then publishes something to its FB wall (note that it is not mandatory to have the same user in the event marketplace and in FB. A user can subscribe to FB events and receive events from all the FB statuses collected from the FB application users). After the event propagation delay, the event marketplace display the event to the user.

So do we need such machinery to do things like that? No, if you want to do some other simple mashup portal. There are several components which are under active development: Storage and processing. Yes we store all events in the Play platform. This storage will be huge, but it is one of the project goal: Providing efficient and elastic storage in some P2P way. The need for this storage comes with the other important aspect of the project: Complex Event Processing. We will soon be able to create complex rules on top of events and be able to generate notifications based on past and real time notifications because we have efficient storage and real time stuff inside the platform. I am not an expert of this domain, so I can not give more details about that point but capabilities are huge! For example, we can express something like « Hey Play, can you send me a SMS me when there is my favorite punk rock band playing just around and I am not on a business trip and X and !Y or Z ». All of this intelligence coming from the processing of various sources I push since months in the platform coming from Twitter, FB, last.fm and other data providers.

Now let’s take some time to work on my OW2Con talk. The session name is pretty cool : Open Cloud Summit Session.

Petals DSB is not Petals ESB

Almost true… In fact Petals DSB uses and extends Petals ESB in several ways. When I started to think about extending the Enterprise Service Bus, it was just to avoid all the JBI stuff at the management level i.e. use a real, simple and efficient API. So I added many management stuff exposed as Web services : Bind external services, expose internal services (oh yes bind + expose = proxy), get endpoints, activate things, etc…
One other goal was to avoid to use closed protocols for inter node communication: The ESB uses at least three ports to create inter node communications for JMX, NIO and SOAP. So why not, just using an open protocol like SOAP and just one port? This is what I did, I changed some implementations to use the same port and the same protocol for all services.

All this stuff has been developed focusing on extensibility and easier development. This is mainly because the ESB can be hard to extend for newbies, there are so many things inside… Today the DSB is not only a Distributed Service Bus, it is also a framework so that developers can easily extend the DSB without the need to know how it works inside. Some examples? Want to expose a kernel service : Add the JAXWS @WebService annotation. Want to subscribe to Web service notifications : Annotate you Java method with @Notify. Want to be notified about new endpoints : Add the @RegistryListener annotation. That’s all, you do not have to search for the Web service server to expose your service, nor read all the WSN documentation to receive notifications, … Simple, efficient.

There are other things you can do, mostly all is detailed in the DSB page.

Let’s talk at OW2Con 2011

This year again, I submitted a talk proposal to the OW2 annual conference and it has just been approved by the OW2 management office.

While last year I spoke about some conceptual things around the Distributed Service Bus and the Cloud, this year I will go one step further with some live demonstrations not only dealing with service bus stuff, but also with some BPM tools and the Cloud stack we actively develop in the research team at PetalsLink. Here is my talk proposal:

All the services are moving to the Cloud, so are business processes. In this talk, we will show how to create collaborative business processes using an open source SaaS BPMN Editor. But designing business processes is not enough, why not running them in the Cloud? We will see that we can rely on a completely Cloud-aware SOA software infrastructure combining several open sources solutions such as a Service Bus and IaaS framework. The resulting ‘Cloud Service Bus’ allows the integration of in-house services in order to benefit from Cloud-based features such as elasticity, load balancing, service clustering and migration. This Cloud Service Bus will serve as the runtime basis of the business processes producing a Petals Cloud Stack solution. All in the Cloud, all open source!

I really hope to have time to work on some cool things to add more foggy-cloudy stuff and have things running on a real cloud infrastructure. I have many ideas in my mind these days and it is really really really cool.

Oh and I will also talk a bit about what we are currently building in the Play FP7 project. We have to show things in two weeks at the European Commission and these things are really interesting to share with OW2 attendees and staff.

BTW, I think that there is some free beer social event this year at OW2Con, see you there of course!

Service Bus Live Monitoring

I explained in the last articles how I tested the Play Framework, Web sockets and how I integrated all this nice stuff with a real example based on a Service Bus, Web services Notifications, etc…

This time, let’s go one step further. We have a Service Bus which is Web service notification enabled like last time. We can bind services to the bus, expose service endpoints as Web services, blahblahblah… But, this time, I am interested on having some real time monitoring of service invocations. It means that each time a message goes through the service bus (a service invocation in fact), I want to know (almmost) immediatly the service response time.
Hopefuly, the PetalsLink Distributed Service Bus I develop and use provides many extension points. One is the capability to add modules to the routing engine ie the software module each message must be able to go through on service request and response. So adding some router module which catch all the messages, timestamp them and then send this monitoring data to someone is quite easy. At the implementation level, this monitoring router module publishes monitoring reports to the service bus notification engine topic dedicated to monitoring stuff.

So, a client interested in monitoring data just has to register itself as subscriber to the monitoring notification topic. Every time a message is published in the topic, it will be delivered to all the subscribers. Up to the subscriber to display data as soon as it can. This is where Play, Web sockets and some cool javascript library came in. Since I never developed javascript stuff, I tried to find an easy to integrate solution to create some moving plots, asking twitter. I finally found the Smoothie Chart library which is really easy to use and updates graph in real time.

The high level architecture of the system can be defined as

The following video shows the result of the complete stack: Each time a message a service is invoked with SOAPUI, a Web service notification is sent to a Play application which subscribed to the monitoring topic, the Play application then pushes the data to the client by using a Web socket. Finally, the javascript code on the client side feeds the Smoothie chart which updates automatically. At the end, it is quite simple and efficient.

Oh, I forgot to say something: This took me 2 or 3 hours to create all this stuff… The code has been published on github in the dsbmanager-webapp project.