Kids are sick? Take coffee and code

I had some time since several nights (thanks kids to be sick…) to start watching how to implement a REST API using node.js (my new friend).

I plan to create an OW2 API this year; but before I started to wrap some existing services as proof of concept. The first feature focuses on how to get projects data from the gitorious instance we run at OW2. There is no clear API for gitorious but after some googling and source code search, it looks that there is a read-only XML API. Since I am targeting node.js, let’s translate this API into a JSON one: xml2js will do the job.

This results in a first library (gitoriou.js) which just call the gitorious instance and translates the XML data into JSON without any other data mapping. The following gist is showing how to get information from a project:

	/**
	* Use the OW2.org gitorious instance for samples.
	*
	* Copyright(c) 2013 Christophe Hamerling <christophe.hamerling@gmail.com>
	* MIT Licensed
	*/
	var Gitorious = require('../index').Gitorious
	, _ = require('underscore');
	var client = new Gitorious({url:'http://gitorious.ow2.org&#39;});
	console.log('# Getting OW2 Shelbie information : ')
	client.getProject('ow2-shelbie', function(err, result) {
	if (err) {
	console.log('Error:', err);
	return;
	}
	// console.log(JSON.stringify(result, null, 4));
	console.log('## Title:', result.project.title);
	console.log('### Repositories');
	_.each(result.project.repositories.mainlines.repository, function(item) {
	console.log(' – ', item.clone_url);
	})
	});

view raw

gitorious.js

hosted with ❤ by GitHub

I talked about having a REST API, let’s use express.js for that and let’s run it on Heroku (this platform rules, just took 2 minutes to create, deploy and run the service and it is up at http://ow2apisample.herokuapp.com/).

	/**
	* OW2 API Sample
	*
	* Copyright(c) 2013 Christophe Hamerling <christophe.hamerling@gmail.com>
	* MIT Licensed
	*/
	var express = require('express')
	, http = require('http')
	, path = require('path')
	, app = express();
	var Gitorious = require('gitoriou.js').Gitorious;
	var config = {
	url : 'http://gitorious.ow2.org&#39;,
	port : 3000
	}
	var client = new Gitorious(config);
	app.configure(function() {
	app.set('port', process.env.PORT || config.port);
	app.use(express.logger('dev'));
	app.use(express.bodyParser());
	app.use(express.static(path.join(__dirname, '../public')));
	});
	app.configure('development', function() {
	app.use(express.errorHandler({ dumpExceptions: true, showStack: true }));
	});
	app.configure('production', function() {
	app.use(express.errorHandler());
	});
	app.get('/projects', function(req, res) {
	client.getProjects(function(err, result) {
	if (err) {
	res.json(500, err);
	} else {
	res.json(200, result);
	}
	});
	});
	app.get('/project/:name', function(req, res) {
	client.getProject(req.param('name'), function(err, result) {
	if (err) {
	res.json(500, err);
	} else {
	res.json(200, result);
	}
	});
	});
	var server = http.createServer(app);
	server.listen(app.get('port'), function () {
	console.log('OW2 Server is started and listening on', app.get('port'));
	});

view raw

server.js

hosted with ❤ by GitHub

To get information (as JSON) about the Jasmine project, just HTTP GET http://ow2apisample.herokuapp.com/project/ow2-jasmine

If you master any scripting language, you will be able to get all the repositories URLs from the JSON response, if not you can use the ow2git project to clone them all. Assuming node.js and npm are available on your system (you already have java, ruby and every other stuff installed, why not adding node?), you can install the binary:

npm install -g ow2git

and then clone all the repositories from any gitorious project

ow2git –clone ow2-jasmine

Will clone all the ow2-jasmine repositories into your current folder.

I am really impressed by node.js runtime, tools and community. Even if all of this is just a proof of concept, not really well designed, you can have something running quickly without many effort. Time to get something real. Soon…

Notify me when someone forks me

All is cool in Github and the API really rocks! I created several tools around it and especially QuickHub, an OS X app to quickly access to your Github space and be notified when something changes.

Last night, I was thinking about creating something which can notify you when someone forks or start watching your repositories even if you are offline. So I had a look to the Repository Hook API and just found that almost all the hooks are targeting push ie you can use tens of connector to be notified on the system of your choice only when a push occurs on your repository. Let’s have a look to the event API but this is not a really good idea, I already use it in QuickHub to notify about activities and it’s just about polling for changes.
So what? Hopefully, there is one alternative: The pubsubhubbub API. According to the GitHub documentation, one can subscribe to any event type using this API. You just have to tell which one on which repository and give it a callback. Giving it a callback is just what we want: Having something which runs online and which can receive events and notify me even if I am offline… Here we go for some Web development with the PlayFramework and Heroku for hosting, here is mygithubnotifier.

mygithubnotifier uses the GitHub Java API from Eclipse. Even if JSON is easy to parse, having it already done is faster. The application retrieves all your personal repositories and provides the user the choice to subscribe/unsubscribe using the pubsubhubbub protocol. This protocol is really simple to use: One endpoint and three parameters are enough. If I want to subscribe to forks on my repository, I just need an authenticated HTTP call like to https://api.github.com/hub with some parameters for defining the topic (repository), the mode (subscribe/unsubscribe), the event type (fork, watch, …) and the callback URL which will be some REST endpoint exposed by the Play application. The application will send an email every time someone forks one of your repository if you subscribed to this event before.

The code is really simple and will not be detailed here. You can get it from GitHub at chamerling/mygithubnotifier. I assume that deploying the Play application to Heroku is as easy as described in their documentation. You just have to configure the application by specifying the right email and GitHub properties as described in the README.md.

OAuth and Desktop Apps

I started to develop QuickHub some weeks ago by focusing on features and without taking into account security issues such as this critical information which are login and password credentials… As mentioned in the GitHub developer pages, I started to use Basic Auth for all the requests QuickHub does to get retrieve data from GitHub. I started to think about moving to OAuth when a user said me that he bought QuickHub but that he did not use it because of Basic Auth. He was afraid that I can rob its credentials and so have a look to its repositories and more. I totally understand this argument and so I started to think that it limits QuickHub adoption by developers and that I should do something.

So, I discussed with GitHub guys through their support channel (Here I want to say that I am really impressed on how fast and professional is the GitHub support team!). Of course, they also told me that they will never use QuickHub if OAuth is not provided. After discussions with the guys, I started to implement a workaround to provide OAuth support in QuickHub by using the OAuth Web flow and adding some stuff to QuickHub.

Let’s see how we can do that in a generic way so that you can use it in your app if needed since every serious Internet service also provide OAuth support… Note that I will not give many details about OAuth itself but I will use some terms, so have a look to OAuth documentation for more details.

Register a Web application

The first step is to register your application to the developer portal. Here you need to provide some information such as the app name and its callback URL. Even if we are developing a desktop app and not a Web one, we need to be able to provide this HTTP based callback URL. We will see in the next section what is inside this application. By registering our application, the service will generate and give us some keys to be used in the next steps, mostly to recognize us when calling the service.

Create a Web application

We need a Web application to handle callback calls from the service we want to use OAuth. This application will only be used at authorization time and just have to be able to receive the service call, get the code sent by the service, then call the service again with the code and our application keys (there is a secret one to be used here). As a result, the service will send you back your OAuth token to be used in all the future service calls. This token is used to authenticate your service calls, no more password stuff inside! Here is a quick sequence diagram showing all the exchanges between actors…

On the QuickHub side, I chose to create this Web application by using the Play Framework I already mentioned several times in this blog. I used the Heroku paas to provide the Web application publicly.

As showed in this Gist (https://gist.github.com/1466592), the callback method is really simple (as usual with Play!): just get the code and call GitHub with it and your application keys. When all lis done, just display a result page with a specific URL. Here it starts with ‘quickhubapp://oauth?’. Let’s see what it means in the next section…

Add custom URL handlers to your desktop app

Once the desktop application is authorized, our Web application redirects the user to a Web page which embeds a button targeting an URL starting by ‘quickhubapp://oauth?’. Here you already understand that by clicking on such a link must drive you to something special. The cocoa framework allows developers to register specific URL handlers for their applications. So we need to register QuickHub handlers so that OS X knows that every link  with the quickhubapp scheme must me routed to QuickHub. This is as easy as showed in this Gist https://gist.github.com/1466628.

This code snippet just tells QuickHub to invoke the getUrl method when it receives a quickhubapp event. Up to the getUrl method to handle things. In QuickHub, I just extract the OAuth token in order to persist it locally for future use.

Done!

So now we are able to use OAuth Web flows for desktop applications. In the best of the worlds, every service provider may provide a real desktop flow to be used without the need to create this additional web application. In the real world, it is not the case but as you see it can be bypassed quite easily.

I will provide more technical details on the second section ‘Create a Web application’ with real sample and code in the next weeks.

BPEL support in Petals DSB

As promised in the last article about my talk at OW2Con 2011 last week, here is a video on something I was not able to show due to some low resolution problems. The video is a bit long but shows several things (in the right order):

1. The DSB Manager Web application is used to manage the Distributed Service Bus. It uses the DSB Web service API to interact with node instances running somewhere…
2. The DSB Manager is used to bind business services to the DSB (let’s forget JBI, the user does not care about it…). DSB services are also exposed. Every DSB node provides the same business API with the help of the distributed endpoint registry it uses.
3. The DSB Manager uses the DSB BPEL API to deploy BPEL processes to the DSB. Up to the DSB to use the right internal endpoint when the process is executed. Services can be hosted on any node, it is the role of the DSB to route messages to the right endpoint on the right node. The BPEL process is exposed as Web service and can be invoked by any Web service client. Here I just use SOAPUI client.
4. We can monitor what happens when invoking a service! For now the DSB Manager uses Web service notification to subscribe to some monitoring topic hosted on the DSB node. When a message is exchanged between the client and the services involved in the process execution, notification are automatically published to the DSB Manager which has just subscribed. The monitoring uses Web sockets for live display in the browser…
5. Last thing is just a test to show more monitoring data when many calls are exchanged between consumers and providers.

Let’s go one step further… The BPEL engine we use in the DSB is our own (PetalsLink) BPEL engine we developed from scratch. This allows us to have a complete control on it and to be able to extend it and embed it as we want without any constraint. In the current case, the BPEL Engine is hosted on a dedicated DSB component. It means that we do not have an external thing which talk with services through some exposed services. This is really important to notice that by doing such thing we can really base process execution on a Service Oriented Architecture. When developing the BPEL process with the Petals Studio, or when creating a BPM process (more details in a future post), you do not have to care about service endpoints. You just have to say to the process that you want to call operation X of service Y or interface Z. It is up to the DSB hosting the BPEL engine to resolve endpoints at runtime. By using this approach we can really do interesting things, just because the DSB is Distributed: services can be hosted on any nodes, can be replicated, can move, can be updated without any impact on the process itself: Oh wait this is SOA!

Related articles

• Back From OW2Con 2011 (chamerling.org)
• Some Play Framework, Service Bus, WS Notification and Web Sockets… (chamerling.org)
• Let’s talk at OW2Con 2011 (chamerling.org)

Create a management application in 3 hours with Play

Story, code, compare

Yet another ‘nightly project’ (thanks to current house build project and the lack of sleep it brings). This time I needed to be able to manage the so-famous Service Bus from some Web enabled tooling. I already developed such tool in a research project but the fact is that the licence of some libraries are not compatible with the petalslink open source approach. The second thing is that it is GWT based (which bores me, has almost 100 libraries dependencies and takes 10 minutes to compile). So this application is a good candidate to compare development productivity between GWT and the Play Framework. Here is a summary of the application creation:

• I bootstraped the application yesterday during lunch between 1PM and 2PM. I was already able to invoke most of the interesting service bus actions with the help of service bus SOAP API.
• I added some pages and actions last night, let say that since it was between 11PM and 1AM I was not very productive…
• I fixed some bugs this morning

As a result, I think I worked around 3 hours on this application. I think I spent one hour to resolve a dependency conflict between Play and a CXF dependency but as a result I have a good result which is almost equivalent in functionality to the GWT based application. I still miss some operations but I do not need them for now… I feel ashamed to say how long it tooks me to create the GWT version…

Deploy

Let’s talk about deployment… I did not have time to play with heroku to push this application in the cloud. Tis will be a future step but since I use Play and git, I am able to push the code to github and then pull it on an OVH server I rent. I am able to provide this instance for some project partners if they want to manage the service bus. The time it took? 5 minutes (wget play, git clone and play start…).

Code

The Play enabled application is available on github at chamerling/dsbmanager-webapp.