14. April 2012
15:37
by: Alex Oberhauser

Development and Release Workflow

In the last few weeks I have tried to work out a suitable development and release workflow for the Sigimera project. For me it was important to have an easy overview about the deployed versions, without the need of intense ChangeLog writing or manual tracking of the versions.

This requirement is a must for bigger teams, but also useful for small teams and even for single developers.

SCM with GIT

1. Develop always in a separate branch, e.g. develop

This keeps the master branch clean and stable. You have always a working version that could be used for demonstration purposes, but also for manual/automatic deployment.

2. Never commit directly into the master branch.

All changes should be merge from the develop branch.

3. Merge always with –no-ff (no fast forwarding).

This allows the easy tracking of changes. You get a ChangeLog for free if you take all commits between two merge statements. As side effect you can automatically extract this ChangeLog. Look at the output at github or bitbucket, it looks really nice. You see there immediately the new changes.

4. If you plan to deploy (release) a version, merge the changes from the develop branch into the master branch and create a tag of the newly created merge commit.

You are able to see the changes with the help of the merge commit message, but the creation of an additional tag shows additional the version number of the deployed version.

 Workflow Enforcement

In my opinion the team working on a software project should accept the workflow. Otherwise you should consider to adapt it or you will fail. But we are all human make mistakes, specially in the beginning. For example if somebody commits to the master branch it breaks the ChangeLog mechanism and maybe the public version. That means we have to assure that only senior developers or a special deployment team have the needed rights to change the master branch and deploy the current version.

Gitolite

Gitolite is nice way to have a permission model on a branch base. Currently I use the following configuration file that allows only the deployment team to update the master branch and add/delete tags. The main developers work on the develop branch. Other needed branches, such as new_feature can not pushed to the remote repository. This keeps it clean, but allows the developers to work locally on as much branches they like.

As you can see in the gitolite configuration file only the admins group can rewind commits. The reason for this is that we want to be able to track all changes.

Continuous Integration

The continuous integration uses always the develop branch. This is enough, because the master branch takes all changes from the develop branch. This implies that if the develop branch succeeds, also the master branch must succeed.

Deployment

The deployment process uses always the master branch, doesn’t matter in which branch you are.

Currently the deployment works only for the Ruby on Rails part of the project. For this purpose Capistrano is used. Important for this step is that the deployment team shares the same user, but with different SSH-Keys. I have called this user deploy. On the server side the Ruby Version Manager (RVM) assure that the deploy user is able to install the needed dependencies.

The deployed version

If you develop and deploy an internal software, such as our Sigimera Admin Interface, you can integrate the ChangeLog directly into the application. The nice thing of that approach is that the commit messages can be linked directly to the source code repository, e.g. to github or bitbucket.

Outcome

The here presented approach is used currently in the Sigimera project, but was also adapted to other, independent projects on which I am currently working on. If you want more in depth information, please feel free to contact me.



Fork me on GitHub
Posted in Development, Sigimera

29. February 2012
21:57
by: Alex Oberhauser

Sigimera Frontend Development or why Ruby on Rails is the right choice

Sigimera Logo

As described in the previous post I have explained why we use MongoDB for the underlying storage of documents, logfiles and multimedia data. This post describes Sigimera (the Crisis Information Platform) from a higher level aka. the Frontends.

In the past the whole platform was developed in Java, Web Services for the backend logic and Servlets for the frontends. This approach should make the code portable (through the JVM, although that is not always true) and uses our knowledge as Java developers. The new version throws away the implemented REST API, the services for User- and Multimedia Management and keeps only the Crisis Crawler (that fetches the crisis from official sources) and the Crisis Channel Service (that distributes the services to social networks).

The new platform consists of an Admin Interface and a Sigimera Application. Both are developed in Ruby on Rails. The Sigimera Application is the main website with related REST API, it uses directly the underlying storage (MVC pattern style, as typical in Rails). The same is true for the Admin Interface, although this application is able to call the background services (currently Crisis Crawler and Crisis Channel) directly and allows us to monitor and admin the platform and acts additional as sandbox for new features.

As in the previous post about MongoDB I will list a few reasons for the change. I am quite sure that are a lot of more that I have temporary not in mind.

  • Fast Development: User Management and REST API development is basically for free. There exists a lot of good libraries that abstract this part from the developer. Previously I have started to implement the user management (with session handling) and the OAuth 2.0 workflow.  Additional you can start the development server once (rails s) and the code are loaded in real-time on-the-fly during changes.
  • Nice Structure: The application structure forces you to develop with the MVC (Model-View-Controller) Pattern. That is really nice and helps to keep clean structure.
  • Integration: It fits all together. You receive a REST API for free, the business logic can be called easily from the HTML code. The data is loaded automatically from the database if you instantiate your model. With MongoDB you have not to run database migration, that saves additional time and keeps all simple.
  • Testing: With the help of RSpec you are able to test from the model to the view all components. Additional Guard runs this tests automatically if changes in the file are detected.
  • Deploying: Once setup you are able to deploy the application with one command to the remote webserver (via SSH and GIT with Capistrano). With another command you can set the webpage to maintenance/production mode.
  • Fast Execution: With the help of memcache the files are stored in the memory and not served for request from the filesystem.
  • Extendable: You can use a lot of different rails libraries that simplifies the development, but you can use also web frameworks, such as twitter bootstrap and all javascript and stylesheet files that you can find. If that is not enough you can developer your own libraries that makes uses of the powerful framework.
  • It makes a lot of fun! Maybe the most important point. After the learning of the basics it makes really fun to develop web application. Actually I am not really a frontend developer, but the last two weeks I have done nothing else.

We don’t plan to substitute all Web Services, but all that have to do with frontends. That splits Sigimera into three parts. (1) The Frontends in Ruby  on Rails, (2) the backend services that are mainly responsible for the storage and handling of crisis and (3) the database that combines the two previous ones.

As I have said it will never be finished and week-to-week optimized a little bit more, until I am really happy with it.

 



Posted in Development, Sigimera Tagged , , , , ,

17. February 2012
19:31
by: Alex Oberhauser

Sigimera Architecture v2.0

mongodb-logo

It is not the first time that Sigimera had changed the underlying architecture. The first few were prototypes to exploit different technologies and find there weakness. After a extended exploitation phase the business logic was simplified and of course the REST API too. But more important is the switch to a scalable, but powerful, database. The following list summarizes why we have chosen MongoDB:

  • Horizontal scalability – Important for our crowd-sourcing approach. If more resources are needed we are able to mirror our whole platform (services + api + database) to a new host.
  • Schema-less document structure – Represents our internal structure; each crisis is a document, the same is true for multimedia meta data, user profiles, …
  • BSON format for internal storage - BSON is similar to JSON. This fact allows trivial transformation from internal to public data representation. In difference to Apache Cassandra this structure allows more complex nested structures, such as Arrays and or nested BSON objects.
  • Full index support - Allows to write complex queries.>
  • Geo-Spatial Queries - This type of queries are very important for our use-case. Each crisis has a related GPS coordinate tuple and in the most cases the users wants to see crisis near his/her location.
  • Perfect for logging messages - Collections can be limited by size (Capped Collection with  auto-FIFO age-out feature). This means that if the maximum size is reached the old message are overwritten with new one.
  • GridFS - Perfect for multimedia files uploaded by the users. Adds querying capabilities by meta data and distributes the files over more hosts (chunked) if needed.

MongoDB seams to be made specially for our use-case and it is additional really easy to use. In combination with our Web Service structure, that allows the splitting of single features to different hosts, this architecture should be future-proof.  To assure suitable performance the file storage is not handled by an underlying Web Service, but directly by REST API.

Sigimera is not quite finished and was not tested in a real-world scenario, but currently it looks promisingly that it could work out very well. Otherwise there will come another architecture changes…



Posted in Sigimera

6. February 2012
20:52
by: Alex Oberhauser

Sigimera Status Report and Justification

The justification of our crisis information platform is that information during and after the crisis must be available for everybody. An important aspect is the social part. Official sources can only give a broad overview about the crisis. Here we try to bridge the gap, between official sources and user generated content, all combined in one platform with the possibility to develop third-party application. The big advantage of our approach is that we are in social networks present and use this channels to propaged crisis information in nearly real time (see http://gdacs.org). This approach allows the user to communicate via this channels, but to get also additional information at our platform.

Currently we are in an exploitation phase, where the components are implemented and integrated into the overall platform. The most important part at the moment is to stabelize the REST API and to adapt our main website (http://www.sigimera.org) to fit the new architecture and integrate with different social networks.



Posted in Uncategorized

22. January 2012
13:47
by: Alex Oberhauser

Networld Blog – Style and other minor changes

Today I have changed the style from the mobile device theme (with the Freerunner as mobile device) to a more neutral theme that fits better to the character of this blog.
Additional I have remove my private website (outdated) from the links and simplified the sharing possibility. Now you can find the share button on the right side (full functionality), a subset of social networks on the left side and the +1 Button on the bottom of each post.
The author information are now on the bottom of each post and not anymore in the right side-bar.

A project page for Sigimera was added.

The About page was rewritten completely.



Posted in Networld

19. January 2012
22:51
by: Alex Oberhauser

Sigimera Architecture 1.0 is dead – Long live Sigimera Architecture 2.0

Sigimera Logo

Sigmera is a Crisis Information Platform that combines community information with information from official sources. see Sigimera

After a few months of technology exploitation and a first proof-of-concept implementation our team has decided to shift the release of the first stable version in favor of more time for a complete re-design and re-implementation with a new architecture in version 2.0.

The reason is that the inferencing and reasoning part plays no role in the current version. That allows us to switch the technology from academic to well established enterprise software. More concrete that means that we change the underlying data storage from a triplestore (OWLIM) to NoSQL (Apache Cassandra). Another big change are the loosely coupled SOAP Services that handle the business logic. Completely hidden from the user and third-party developers, this decision allows us to easily extend our current platform and not less important to distribute it over different hosts. For example the storage part could run on one cluster, the crisis crawler on one and the crisis management part with the REST API on another.

Important for the new architecture is that all frontends are based on the REST API. The REST API should use a combination of different services on different hosts to gain optimal performance. The combination of enterprise architecture for the business logic, semantic for the data representation, NoSQL for the physical data storage and Web 2.0 technology for the frontends should make Sigimera to a very easy to use and flexible platform. At least from a technology point of view.

Another important part is that we try to combine the best part from the Semantic Web Community (e.g. ontologies for interchangeable data) with well-established technology (e.g. NoSQL database to achieve horizontal scalability).

Maybe the most important part is that we shift from an academic exercise to an usable platform for everybody.

The next few months will show if we have taken the right path or if we have to find a new one.  Feel free to read more about the current changes and how the old architecture has looked liked in our Wiki. If you have questions or feedback please don’t hesitate to contact me.

Sigimera Architecture 2.0 – by Alex Oberhauser, 2011-01-19T22:51:00
Sigmera Crisis Information Platform Architecture 2.0 Review



Posted in Research, Sigimera Tagged

16. January 2012
16:03
by: Alex Oberhauser

Server Downtime

In the next days, starting from the 16. January, it is possible that the websites and services under networld.to and sigimera.org are partial offline. The reason are major changes in the server architecture as preparation for the Sigimera Release 2.0.

I apologize for any inconvenience.

EDIT:  Major changes accomplished! Not longer down times expected.



Posted in Administration Tagged

21. November 2011
09:12
by: Alex Oberhauser

Sigimera – Semantic Data Model

Sigimera Logo

The Crisis Information Platform, as described in previous blog entries on this site, consists of a SOAP backend, with an public accessible REST API. Another major part is the data representation, that is specified in RDFS/OWL. From the user representation to the representation of crisis, all data is defined as RDF Schema (partially if needed in OWL). If possible, existing ontologies are used and combined with Sigimera internally ontologies:

  • crisis.owl (prefix: crisis): Defines the schema of a crisis and related data. (Used third party ontology at the instance level: DuplinCore, FriendOfAFriend, … )
  • on-site.owl (prefix: on-site): Defines agents that act in a crisis, such as Rescuer, Victims, Drones. Currently this ontology is not used in the first release.
  • admin.owl (prefix: admin): Administration Ontology that defines Subscriber and missing properties for user handling.

The core concept is crisis:CrisisArea for a crisis entity (with crisis:DangerousZone for secondary crisis in the same area) and foaf:Agent for user that are related to one or more crisis and/or registered at our platform.

As triplestore OWLIM-Lite is used (for crisis information). The reason was the inferencing engine on default rule sets (OWL-Horst, OWL-DL, RDFS, …) and the possibility to extend this rule sets with own inferencing rules. One big challenge in the future will be the scalability of the triplestore when the number of triples and/or when the number of access per second rises. A solution is on the way…

More Information: [not-public]



Posted in Development, Research, Semantic Web, Sigimera Tagged , , , ,

17. September 2011
21:58
by: Alex Oberhauser

Sigimera – Short outlook to the first Crisis Information Platform BETA Release

Sigimera Logo

After the setup of a developer infrastructure, our team works toward a first release. The plan is to have a first beta release by the end of this year. To achieve this goal we are working on a suitable frontend, that should be easy to use, but powerful enough to have all relevant information on your fingertip. On the other side we are working on exporting the crisis data as linked data bubble and on a REST API for third party developers. The decision to release an API at this early stage should encourage developers to use the crisis services for amazing new types of applications.

More information about the current status you can find in our WIKI.



Posted in Development, Sigimera Tagged , , ,

17. July 2011
12:59
by: Alex Oberhauser

Sigimera – Crisis Information Platform

Sigimera Logo

The last few months I have worked on an approach for optimizing the information flow in crisis areas with the help of semantic technology. The outcome was the shift from a drone based framework, that aims to help on-site crisis staff, to a public available platform for everybody. The first phase of Sigimera (the project name), will be the launch of a portal that unifies different information sources, e.g. gdacs.org (nearly real time information from authorities), multimedia from the community (videos, pictures, messages), articles from newspapers, … . The main goal is to build an online community around the project, that brings people affected by crisis together with people that are able to help and inform outsider about the situation.

All information are semantically annotated and classified. This approach simplifies the evaluation of big data sets and optimizes the visualization of relevant information. The designed and build architecture allows a simplified frontend with a powerful backend. The backend allows enough room for extension, not only from a engineering perspective, but also from research side.

The following components are planned for the first BETA release:

  • Information website that simplifies the start for user, developer, researcher, …
  • User Management (with third-pary login and registration mechanism)
  • REST API for developers
  • Crisis dashboard recommendation that should give a first feeling what is planned and what could be done in future releases.
  • User feedback mechanism

The project is in a very early stage of development, although the current status gives a good overview what could be possible in the future.



Posted in Development, Networld, Research, Sigimera