Introducing the CloudStack RIA + P2Web framework

While Rich Internet Application (RIA) technologies, like AJAX and Flex, have enabled a new breed of interactive web applications, the desktop still offers far more richness than can be afforded by a remote web application. This is partly because the desktop has a lot more "local" resources to offer (storage, memory, and processing power), and partly because certain applications and content are simply best suited for the desktop.

The biggest shortcoming of the desktop though, is that unlike the web, sharing from the desktop is hard. Approaches, like P2P, require special software to be running on both sides, which limit their long-term usability and mainstream uptake. The use of proprietary protocols also typically results in closed networks or darknets.

A hybrid peer to web (p2web) model overcomes these shortcomings by bringing together the best of both worlds - combining the ubiquity of the web with the richness of the desktop. A peer to web model lets you leave all your content on your desktop while allowing you to selectively share it with others using just a web browser. In this article, the author introduces an open source framework, CloudStack, for building these types of p2web applications.

CloudStack Background

The CloudStack framework was extracted from the popular file-sharing application, BoxCloud, developed by WiredReach, Inc. BoxCloud is a service that lets people share large files right from their desktop - without any emailing, uploading or ftping.

When we started building BoxCloud, we realized that sharing files today was much harder than it needed to be. We also realized that people needed a solution that adapted to the way they work:

1. they work on a desktop and not on a server and
2. they want to avoid downloading and installing yet another application, if possible.

So we employed a p2web model that allows sharing from the desktop but viewing over a web browser. When a file is shared in BoxCloud, a link is emailed to the recipient that can be accessed over a regular web browser. Because the link points to the local file that resides on the sharer's hard drive, the sharer can continue working on the file without ever having to republish or upload the file.

BoxCloud effectively turns the sharer's desktop into a web addressable server without the need of a static or public IP address. Each application instance is addressable through a well-known URL like http://wiredreach.boxcloud.com, which serves as the "document root" for the application.

We felt that the p2web model could effectively be applied to domains other than file sharing, where there is a similar need to share "content" directly from a desktop or device. We knew that we couldn't build (or even think up) all these applications ourselves. So we decided to extract this framework and set it free for the world to use.

What types of applications can I build

• A lightweight CRM system (universal address book) that integrates with your favorite contact management system e.g. Outlook, ACT, etc. that can be accessed from anywhere and shared with anyone
• A web accessible shareable calendar that integrates with Outlook or iCal
• A photo or video sharing application that doesn't require uploading to an external service
• A whole slew of mobile applications that can access data from your PC or enterprise while you are away from the office
• A sophisticated monitoring and administration system for embedded devices and sensors

CloudStack Manifesto

• CloudStack brings together the best of the web and the desktop: We strongly believe in the convergence of the desktop and the web and will continually strive to expose more services that bring out the best from both.
• CloudStack enables rapid application development and deployment: Out of the box, CloudStack provides a fully brand able and deployable shell application that can be used as a starting point to jumpstart application development. CloudStack also provides a scalable deployment environment for hosting your applications.
• CloudStack leverages existing web technologies: We built the CloudStack P2WebServer container over the J2EE compliant Jetty web server. As a result, CloudStack applications are built using standard web technologies like AJAX, HTML, JavaScript, Flash, Flex, etc.
• CloudStack does not reinvent the wheel: We strive to reuse as much as possible from other open source projects and standards. By creatively stringing together seemingly disparate pieces, like P2P and HTTP, it?fs amazing to create something that's really much greater than the sum of the parts.
• CloudStack does aim to simplify existing technologies: We will abstract and simplify existing interfaces if needed. For example, we built simpler abstractions for JXTA (P2P) and Jena (RDF Store).
• CloudStack encourages HTML-based interfaces: We believe that the web browser is the most portable desktop application container with HTML being the lingua franca of the web. Rather than writing a native widget interface for the local desktop application and another web-based interface for the remote view, we encourage writing a single interface that can be reused across both local and remote views. HTML based interfaces are inherently cross-platform and provide good decoupling of design from code (versus having the UI as compiled code). Also, we feel that web based interfaces coupled with RIA technologies can provide as good and sometimes an even better user experience than a native application.
• CloudStack is extensible through plug-ins: We wanted to make everything in CloudStack a plug-in and decided to build it over the Eclipse plug-in model, which makes it infinitely extensible.

The CloudStack Network

The CloudStack network is where the p2web magic takes place. The network is made up of a private P2P network that is bridged to the traditional web through a cluster of gateway nodes. The gateway nodes are responsible for resolving http requests in real-time to one or more desktop endpoints.

We picked a P2P network to serve as the CloudStack backbone because P2P networks are inherently more robust, dynamic and self-organizing than a client/server configuration. P2P networks also support working from behind NATs and firewalls, which is a very common requirement today.

Also, because each node in a P2P network can address other nodes directly, in situations where both sides are running the same CloudStack application, a direct P2P connection can be used, that bypasses the CloudStack gateway nodes altogether. As a result, the gateway nodes serve more of a bootstrapping function in the network and are designed to be very lightweight and scalable.

Request Routing

Every CloudStack application registers with the P2P network using a global URL identifier, for example, http://ashmaurya.cloudstack.com. This is the URL that you enter in a web browser to access the application. HTTP requests made from a web browser are intercepted and dynamically resolved to the desktop endpoint by CloudStack. By default, the CloudStack network performs user authentication on all requests before forwarding them to the endpoint for processing.

Here is how a typical request is routed in CloudStack using the built-in MVC framework made up of a Dispatcher, Action Controllers and Action Viewers:

1. Remote requests made from a web browser are first routed to the CloudStack P2Web cluster where the user making the request is first authenticated using either an OpenID or CloudStack username/password.
2. The CloudStack P2Web cluster then resolves the request in real-time to one or more endpoints that can service the request.
3. The request is then forwarded to the Dispatcher, which is responsible for mapping the request URL to a registered Action Controller.
4. The Dispatcher routes the request to the appropriate Action Controller, which is responsible for assembling the model for the target page. An optional RDF data store may be used to provide the persistence layer.
5. Once the model is assembled, it is passed to a Velocity template engine for rendering.
6. Finally the resulting page is displayed in the requestor's browser.

Local requests can bypass the CloudStack P2Web cluster using a localhost based URL that directly accesses the local P2WebServer.

CloudStack Architecture

CloudStack functionality is organized into 7 modules shown below:

Fig. 3: CloudStack Architecture

Each of these modules is packaged as one or more Eclipse plug-ins providing the developer some flexibility when assembling applications. For example, a developer may choose not to bundle the Persistence module if a different persistence solution is desired.

Let’s briefly review each CloudStack module in turn:

Core - P2WebServer Container

The Core module defines the services extension point (org.cloudstack.core.services). A service in CloudStack is the public interface exposed by a module. Services are injected into the framework through the use of plug-ins. Everything in CloudStack is a plug-in and each CloudStack module exposes one or more services.

Here is an example of a service definition for the P2WebServer Container:

<extensionpoint="org.cloudstack.core.services">

  <service

    id="org.cloudstack.p2webserver"

    name="P2WebServer Service"

    class="org.cloudstack.p2webserver.P2WebServer">

  </service>

</extension>

You access this service using the ServiceRepository singleton class:

ServiceRepository sr = ServiceRepository.getInstance();

P2WebServer p2webserver = (P2WebServer) sr.getService(P2WebServer.SERVICEID);

The Core module also defines the P2WebServer Container, which is the building block for any CloudStack application. P2Web applications are also deployed as one or more plug-ins. The P2WebServer defines 2 extensions points: org.cloudstack.p2webserver.controllers and org.cloudstack.p2webserver.webapps, that are used to define p2web application plug-ins.

The controllers extension point is used to define applications using a built-in MVC model. The webapps extension point is used to define a J2EE web application.

Persistence

The Persistence module provides a robust RDF datastore built on the Jena RDF library and uses HSQLDB for the database. RDF (Resource Definition Framework) is a very simple, yet incredibly powerful, modeling language that can be used to represent resources (or objects) as a set of subject-predicate-object statements or triples. Unlike a static data model, like a database schema, RDF’s true power lies in its extensibility. 

The Persistence module also provides a simple ORM layer (modeled after Hibernate) that simplifies working with RDF. The ORM layer defines an RDFObject class, which behaves more like an object and abstracts away the details of RDF representation – i.e. dealing with resources and properties.

Here’s a snippet of how you’d persist a Person RDF object:

ServiceRepository sr = ServiceRepository.getInstance();

RDFStoreIF rdfStore = (RDFStoreIF) sr.getService(RDFStoreImpl.SERVICEID);

SessionIF sess = rdfStore.openSession();

Transaction tx = null;

try {

  tx = sess.beginTransaction();

  Person p = new Person(uid, formattedName);

  p.setProperty(VCARD.EMAIL, user.getUserEmail());

  sess.save(p);

  tx.commit();

} catch (Exception e) {

  if (tx != null)

    tx.rollback();

    log.error(e);

  } finally {

    sess.close();

}

P2P Services

When both sides have a CloudStack application installed, greater efficiency and interactivity can be achieved by using a direct P2P model for communication. The P2P module provides such an abstraction built on the open source JXTA P2P library.

Using this abstraction, a CloudStack application instance can address another application instance using its global URL identifier, which returns a JxtaSocket. A JxtaSocket is subclassed from JavaSocket and can be used to send arbitrary data (bytes) from one endpoint to another.

JXTASocket is a low-level point-to-point transport protocol, which can be cumbersome to work with. For this reason, CloudStack defines a channels extension point, which can be used to define higher-level protocols to provide more accessible, and reusable services for things like presence, instant messaging torrent transfers, etc.

File Transfer Channel

The File Transfer module is one such P2P channel that can be used to transfer large files using torrents. Torrents provide resiliency to failures by splitting up the file into chunks and can also support swarm downloading across multiple sources.

XMPP Channel

The XMPP module is another P2P Channel that provides a high-level XML streaming based application protocol. XMPP has been most popularly used for presence and instant messaging (Jabber) but through XMPP extensions (JEPs), it can be used for a lot more.

Sync Service

One of the obvious drawbacks of a P2Web model is that if the source computer is offline, it’s application and content is not available on the network. The Sync module defines a lightweight master-less synchronization protocol that allows you to replicate all or some content across two or more loosely coupled applications. This can be used to provide server side caching of critical or recent information, for example, or simply to mirror content across nodes. The Sync protocol is built on RSS with Simple Sharing Extensions (SSE).

UI – CloudStack MVC

The UI module provides a built –in MVC framework that let you assemble your application as a set of plug-ins.

• The persistence layer can be used to provide the Model or you can use your own.
• Velocity templates are used for the view technology.
• Controllers are defined using the org.cloudstack.p2webserver.controllers extension point

Here is an example controller extension definition:

<extensionpoint="org.cloudstack.p2webserver.controllers">

  <controllerid="com.wiredreach.boxcloud.default"

    name="default"

    path="/"

    class="com.wiredreach.boxcloud.filesharing.web.controllers.DefaultController">

      <actionid="default"

        name="default"

        path="/"

        class="com.wiredreach.boxcloud.filesharing.web.controllers.DefaultAction">

      </action>

   </controller>

</extension>

By default, the framework follows a REST approach to URL naming. The controller name is used to create a web application context and the action name is used to create the action servlet.

The Controller class must be derived from org.cloudstack.p2webserver.ApplicationController, which among other things sets the correct class loader for jetty. The controller class also adds the specified actions as servlets and is also responsible for setting up the default template directory.

The Action classes must be derived from org.cloudstack.p2webserver.ActionServlet, which reads the Velocity template from the default template directory. The action class is responsible for providing the Velocity Context that provides the variable definitions referenced in the template.

Since CloudStack is Eclipse based, you may choose instead to build your UI using SWT/Jface or Swing. Completely headless applications are also possible.

List of open source technologies and standard protocols used in CloudStack

Open source technologies used by CloudStack:

• Eclipse RCP
• Jena RDF
• Jetty web server
• JXTA P2P
• Smack XMPP
• Apache Commons
• Log4J

Open standards and protocols used in CloudStack:

• HTTP
• JXTA
• RDF
• RSS/SSE
• XMPP