Mittwoch, 19. August 2015

Building Eclipse Plugins with Maven Tycho and Travis-CI



A couple of weeks ago, we moved our open source project Yakindu Statechart Tools from google code SVN to GitHub. Until today we packaged and deployed our software using a self hosted Jenkins Server, but since Travis CI integrates seamlessly with GitHub and is free for open source projects we decided to give it a try. This blog post tries to explain how to set up Travis CI for a Maven Tycho based eclipse build.

There are plenty of blog posts out there, that stated how easy it was to setup a Travis CI build for a 10 lines of Java code 'Hello World' example. This is great, but our builds have a lot more work to do:
  • 170,000 lines of Java code
  • Code generation of Xtend classes and Xtext grammars
  • more than 1000 JUnit Tests (including UI Tests)
  • ~100 C and C++ Google Tests to run 
Let's see how well Travis can handle this.
 
Initial Setup

To get started with Travis you first have to log in to travis-ci.org with your GitHub account and grant access to your GitHub repositories This requires admin access to the repository. It is well documented in the User Guide. Next, you have to create a file .travis.yml in the root folder of your repository to tell Travis what to do. By the way, there is a really useful online syntax checker for yml files available here.

This is the simplest possible .travis.yml file for our build based on Maven Tycho:
 language: java   
 script:    
  - cd releng/org.yakindu.sct.releng   
  - mvn clean verify   

Immediately after pushing the .travis.yml file to our Git repository Travis started a new build.
and we ended up in a 5 MB log file full of exceptions like this one:

Caused by: org.eclipse.swt.SWTError: No more handles [gtk_init_check() failed]
    at org.eclipse.swt.SWT.error(SWT.java:4517)
    at org.eclipse.swt.widgets.Display.createDisplay(Display.java:908)


org.yakindu.sct.generator.genmodel.ui.SGenExecutableExtensionFactory
    at org.eclipse.swt.SWT.error(SWT.java:4517)
    at org.eclipse.swt.widgets.Display.createDisplay(Display.java:908) 

Running UI Tests

After analyzing the log files it points out that our UI dependent tests are not able to create an SWT Display object. Luckily, Travis allows the use of the virtual framebuffer xvfb for UI tests. It can be started as follows:

  language: java    
  env:  
   global:  
    - DISPLAY=:99.0  
  before_install:  
   - sh -e /etc/init.d/xvfb start - sleep 10  
  script:    
  - cd releng/org.yakindu.sct.releng    
  - mvn clean verify    

After pushing these changes to Git the build finished successful!

User Interface

Travis provides a simple yet powerful web based user interface that displays a list of all your repository builds and the console output. This is really helpful to see what is going wrong with your build. It also allows to start a build without pushing something to your repository.
What I really miss in the UI is a better integration for Unit Tests, like the Jenkins JUnit plugin. But such a feature is unfortunately not planned and one has to browse the (huge) log files for failed Unit tests.

 GitHub Integration

Per default, the master branch as well as all pull requests are build. The integration with GitHub works out of the box, the running CI jobs are shown in the GitHub UI as checks and updated automatically when the job finishes. This is really awesome for a configuration file with only 9 lines!


 Caching and container based infrastructure

Every time a new build job starts, a new Linux image is setup. To prevent that Maven downloads the internet every time a new build is started, Travis provides a container based infrastructure that allows the use of caches. One drawback of this is that the use of the sudo command is prohibited.

 sudo: false  
 language: java    
 cache:  
  directories:  
   - $HOME/.m2  
 env:  
  global:  
   - DISPLAY=:99.0  
 before_install:  
  - sh -e /etc/init.d/xvfb start - sleep 10  
 script:    
  - cd releng/org.yakindu.sct.releng    
  - mvn clean verify    

As shown above, starting the .yml configuration file with sudo: false allows the use of caches. When the build starts and a new Linux image is created, all cached directories (in our example the maven repository .m2) are restored at the beginning. This really boosts up the build time.

Performance

Our existing Jenkins CI Server is hosted on a Linux KVM virtual machine with 12 core Opteron 2,6 Ghz, 32 GB RAM and  Raid 10 HDD. The average build time is about 13 - 20 minutes.
Travis, running somewhere on Amazon cloud services, took about 10 - 11 minutes for a build. I don't know how they do it, but this is incredibly fast!


Adding google test framework for C and C++

Since Yakindu Statechart Tools is shipped with a C and C++ code generator, we have a couple of C and C++ tests that have to be run during build. This was the not-so-funny part of the configuration, because google recently decided to ship libgtest only as source code and without the static libraries. On the Linux machine we are running Jenkins it was straight forward to install google test, compile it and copy the static library to /usr/lib with the following commands:

 sudo apt-get install libgtest-dev  
 cd /usr/src/gtest  
 sudo cmake CMakeLists.txt  
 sudo make  
 sudo cp *.a /usr/lib  

Running these commands at the beginning of the Travis build would be possible, but we want to use caching so we are not allowed to use sudo, as explained above. Fortunately, there is an apt get whitelist that allows the installation of external libraries via addons.apt.packages:

 sudo: false  
 language: java  
 addons:  
  apt:  
  packages:  
  - libgtest-dev  
... 

Now the GTest source code is available on the image, but it has to be compiled. The - somehow quite hacky - solution is to copy the source into the build dir, compile it and create a environment variable GTEST_DIR that is used in our C Tests for the library lookup. Copying to usr/lib did not work because without sudo the permission is denied.
 env:  
  global:  
   - GTEST_DIR=${TRAVIS_BUILD_DIR}/gtest
  before_script:  
  - mkdir gtest   
  - cd gtest  
  - cp -r /usr/src/gtest/. .  
  - ls  
  - cmake CMakeLists.txt  
  - make   
... 

 At least this works, if you know a better solution please let me know :)
  
Publish Releases
Last but not least, we want to publish successful release builds to GitHub releases automatically.
To publish artifacts to GitHub releases a secret api_key is required for Travis to get access. These api_key should be encrypted, especially when the .travis.yml file is part of an open source project. ;-)
To set up the deploy section of the configuration, you need to install the Travis Ruby Gem.
First, download Ruby version 2.0.0 (I tried to install the Travis Gem with a newer version but it did not work) and install the ruby gem as described here.
If everything works properly, run the following command in the root folder of your GIT repository:

travis setup releases

This will setup a basic deployment section to your .travis.yml file. Do not forget to add the skip_cleanup: true, otherwise the build artifacts you want to release got deleted before the deployment step. In our case, a release is indicated with a tag that follows the naming convention release.x.x.x and we only want those tags to be automatically published, so we added a condition to the deployment section.

... 
 deploy:  
  skip_cleanup: true  
  provider: releases  
  api_key:   
   secure: BSEYtMYXInrXum0eO........  
  file: releng/org.yakindu.sct.repository/target/updatesite.zip  
  on:  
   tags: Yakindu/statecharts  
   condition: "$TRAVIS_TAG =~ ^release.*$"  

Now, every time Travis indicates a release tag, it will automatically publish the release to GitHub in case of a successful build.

Conclusion: Travis provides an incredibly powerful and customizable infrastructure to open source projects for free. Thank you very much for that and keep up the good work!