Git, Gerrit Review and Jenkins or Hudson CI Servers

This article explains how to set up Git, Gerrit and Jenkins/Hudson for team-based code review systems, as espoused in my Gerrit for iOS developers and Gerrit for Java developers presentations (and my someday... discussion, where I first argued the case for using this). The examples used assume you're using OS X or Linux, but you can run on Windows as well if you want to.

Setting up Git

Git is available on most packaging systems already, but there are installers available from the Git homepage. For Windows, the best bet is MsysGit. Note that if you've got the Apple developer tools installed (for Xcode 4) then this comes with Git binaries already. If you have problems, there are a number of very good guides at help.github.com.

Since all Git commits have an author and email address, you need to set up your name as follows, if you haven't done so already:

$ git config --global user.name "Alex Blewitt"
$ git config --global user.email "Alex.Blewitt@example.com"

You should be good to go with a Git repository. Gerrit will automatically scan Git repositories at initialisation, which is slightly easier than setting them up afterwards, so putting the Git repositories in before initialising Gerrit makes sense.

If you haven't got an existing Git repository, you can create one suitable for use:

git init --bare /path/to/gits/example.git

Gerrit

Gerrit is available from the downloads at http://code.google.com/p/gerrit/, and exists as a WAR file. There's good documentation available (currently, 2.2.1 is the latest, but this works with 2.1.7 as well) along with the installation guide.

Gerrit needs a database (to store the review information) as part of its runtime. Currently supported databases include H2, PostgreSQL and MySQL. By default, it will use H2 which needs no additional setup.

Note that Gerrit 2.2.x is moving the project configuration, rights and other metadata into Git storage, so that they are accessible and versionable through Git. This transition will continue for other types of metadata, including review notes, in the 2.2.x streams. Please see the release notes for more information.

To initialise Gerrit, run java -jar gerrit.war init -d /path/to/location to install the runtime in the given path.

If run from an interactive terminal, you are asked various questions, such as:

• Location of Git repositories [git]
• Import existing repositories [Y/n]
• Database server type [H2/?]
• Authentication method [OPENID/?]
• SMTP server hostname [localhost]
• SMTP server port [(default)]
• SMTP encryption [NONE/?]
• SMTP username
• Run as [you]
• Java runtime [/path/to/jvm]
• Copy gerrit.war to /path/to/location/bin/gerrit.war [Y/n]
• Listen on address [*]
• Listen on port [29418]
• Download and install Bouncy Castle [Y/n]
• Behind http reverse proxy [y/N]
• Use SSL [y/N]
• Listen on address [*]
• Listen on port [8080]

Most of these can be left as their default values. However, a few are worth noting the behaviour of.

• Location of Git repositories is where the git repositories live. This defaults to the 'git' directory underneath the installation location, but can be in a different location (e.g. /var/gits or similar). It's worth populating this directory with your example first, because when Gerrit starts, it will scan this directory for new projects to add.
• Listen on address is useful for hosts with multiple addresses (e.g. IPv4 and IPv6) as it allows you to constrain which address it uses. The * means any address on the local host.
• Listen on port is the port number. 29418 is the default Gerrit SSH daemon, and 8080 is the default Gerrit web daemon. However, if you have applications using port 8080 already, you might want to change the second one.
• Authentication method is how you log into Gerrit. OpenID works if you want to hook into an existing authentication provider (e.g. Google Accounts) but for testing purposes – and the ones used in the demos above – you can use development_become_any_account. Typing a ? will show a list of the available methods.

When Gerrit finishes running, you should be opened into a browser that shows you the main page. The first user to log in becomes the administrator automatically; all subsequent users that log in are non-privileged users. If you chose the development_become_any_account there's a Become link on the top of the page, which will take you to the registration/sign-in page.

Registering a user

In order to do anything with Gerrit, you need to have a registered account and an SSH keypair generated. Running ssh-keygen -t rsa -b 2048 from a command line allows you to generate a keypair, which gets put in your .ssh directory. There's more information at the GitHub Help page, although if you want more information you can see this blog post I wrote six years ago on SSH keys.

The default will be called id_rsa (which is the private key) and id_rsa.pub (which is the public key). Only ever give out your public key, never your private key.

With the key in hand, you can register a new account in Gerrit. Click on the “become” link on the top right, followed by the “New account” button, and put in your name and e-mail as they're known by Git (the one we configured above with git config). These have to match exactly (including case). You can save changes, and then pick a unique user name (click 'select username' once you've filled it with a name e.g. demo).

Email headaches Gerrit will try and send you an e-mail to verify your mail address, even in development_become_any_account mode. Without it, it doesn't like your e-mail address, and without that, you can't push code.

We can hack that shortly, so don't worry if it's not letting you register your mail address just yet.

In the 'add SSH public key' text box, add the key exactly as it is given in the .pub file. If you're on OS X, this is as easy as pbcopy < ~/.ssh/id_rsa.pub. Remember to click on “Add” to save it.

When you click Continue, you should see you logged into Gerrit's main window. So far so good. Now we can test SSH connectivity.

Typing ssh -p 29418 demo@localhost will try and talk to the Gerrit server, which will say one of three things:

1. @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
   @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
   @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
   IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
   

   This isn't nearly as bad as it looks. It just means that there's an old key in your ~/.ssh/known_hosts file. The lazy way to fix the problem is to just delete this file (which is GitHub's recommendation). The better way is to look for the line which tells you which line the error is, and delete just that line:

   Offending key in /Users/demo/.ssh/known_hosts:123
   

   So we need to delete line 123 from the known_hosts file, which you can do with any text editor you want. Given a standard UNIX setup, you can do it automatically:

   sed -i '' '123d' ~/.ssh/known_hosts
   

2.  

   The authenticity of host '[localhost]:29418 ([::1]:29418)' can't be established.
   RSA key fingerprint is e8:e2:fe:19:6f:e2:db:c1:05:b5:bf:a6:ad:4b:04:33.
   Are you sure you want to continue connecting (yes/no)? yes
   Warning: Permanently added '[localhost]:29418' (RSA) to the list of known hosts.
   Permission denied (publickey).
   

   If you get this message, it means that Gerrit doesn't recognise any of the keys you have submitted. By default, ssh will send id_rsa, but you can ensure this is the case with an entry in the .ssh/config with a line IdentityFile ~/.ssh/id_rsa at the top. You can run ssh -v which will expand on what it is sending:

   debug1: Next authentication method: publickey
   debug1: Trying private key: /Users/demo/.ssh/id_dsa
   debug1: Offering public key: /Users/demo/.ssh/id_rsa
   

   Assuming it's sending the right key, check that the key is associated with the user in Gerrit. This is at the settings - ssh public keys menu option. If it's not present, click on 'Add Key' and paste in the public key as before.

   If you get this message, and Gerrit is still complaining that you are unauthenticated, check the user name matches the username specified in the settings page. If the username is something different, try ssh -p 29418 username@localhost instead.

   Finally, to verify a specific key, run ssh -i ~/.ssh/id_rsa to explicitly choose which key to use, instead of letting it get picked automatically. If this works, but running without the -i parameter fails, then the issue is in your ~/.ssh/config file – you need to make sure that the appropriate IdentityFile is being selected.

3. 
     ****    Welcome to Gerrit Code Review    ****
   
     Hi demo, you have successfully connected over SSH.
   
     Unfortunately, interactive shells are disabled.
     To clone a hosted Git repository, use:
   
     git clone ssh://demo@localhost:29418/REPOSITORY_NAME.git
   
   Connection to localhost closed.
   

   If you see this, Gerrit is working as expected.

Fixing the email address

If you couldn't register an e-mail address in Gerrit earlier, you can do so manually. We'll stop Gerrit, then run the GSQL tool to update the columns appropriately.

$ bin/gerrit.sh stop
$ java -jar bin/gerrit.war gsql
Welcome to Gerrit Code Review 2.1.6.1
(H2 1.2.134 (2010-04-23))

Type '\h' for help.  Type '\r' to clear the buffer.

gerrit> select * from ACCOUNT_EXTERNAL_IDS;
 ACCOUNT_ID | EMAIL_ADDRESS          | PASSWORD | EXTERNAL_ID
 -----------+------------------------+----------+------------------------------------------
 1000000    | NULL                   | NULL     | uuid:ac1b8a08-2dd1-4aa1-8449-8b2994dffaed
 1000000    | NULL                   | NULL     | username:demo
(2 rows; 23 ms)
gerrit> update ACCOUNT_EXTERNAL_IDS set EMAIL_ADDRESS='alex.blewitt@example.com' where ACCOUNT_ID=1000000;
UPDATE 2; 5 ms
gerrit> select * from ACCOUNT_EXTERNAL_IDS;
 ACCOUNT_ID | EMAIL_ADDRESS          | PASSWORD | EXTERNAL_ID
 -----------+------------------------+----------+------------------------------------------
 1000000    | alex.blewitt@example.com | NULL     | uuid:ac1b8a08-2dd1-4aa1-8449-8b2994dffaed
 1000000    | alex.blewitt@example.com | NULL     | username:demo
(2 rows; 23 ms)
gerrit> \q
Bye
$ bin/gerrit.sh start

Creating a project, cloning and pushing

To get started, we need a project in Gerrit. If it didn't detect the project in your example directory, we can create one subsequently if Gerrit is running.

$ ssh -p 29418 demo@localhost gerrit create-project --name example.git

This will create a project called example, and initialise an empty repository in the gits location specified above. If you have an existing repository, it won't let you create it with the same name – but you rename it to a temporary name prior to creation and then rename back it will still work.

Having made it available in Gerrit, we can create a clone:

$ git clone ssh://demo@localhost:29418/example.git
Cloning into example...
warning: You appear to have cloned an empty repository.

We can commit and push to our repository, much like any other Git system:

$ cd example
$ echo hello > world
$ git add world
$ git commit -m "The World"
[master (root-commit) 06bf85e] The World
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 world
$ git push
No refs in common and none specified; doing nothing.
Perhaps you should specify a branch such as 'master'.
$ git push origin master
Counting objects: 3, done.
Writing objects: 100% (3/3), 217 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
To ssh://me@localhost:29418/example.git
 ! [remote rejected] master -> master (prohibited by Gerrit)
error: failed to push some refs to 'ssh://demo@localhost:29418/example.git'

What happened here? Well, Gerrit doesn't want us overwriting any branches directly in the Git repository. Instead, we must push to a different refspec, which gives Gerrit the opportunity to put the code through review. The easiest way to do that is to configure the default refspec for pushes:

$ git config remote.origin.push refs/heads/*:refs/for/*
$ git push origin
Counting objects: 3, done.
Writing objects: 100% (3/3), 217 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
To ssh://demo@localhost:29418/example.git
 * [new branch]      master -> refs/for/master

We've pushed a branch, referred to here as refs/for/master, although the actual named branch refs/changes/01/1/1, which you can see from the change,1 in Gerrit itself. You'll note that the current branch is the same hash as shown here (in my case, 06bf85e). If we were to push again, instead of overwriting the refs/for/master, we'll trigger the creation of a new branch refs/changes/02/2/1. Although it's an implementation detail, the first digits are the last two digits of the change number, and second digits are the change number, and the third digits is the patch set number. So patch set 17 of change 123 would refer to an immutable reference refs/changes/23/122/17.

If we wanted to commit amend this change, Gerrit will create a new review reference, which is not so good. You'll lose the review request comments between the two changes if that happens.

To fix that, we'll update the commit-msg to add a (Gerrit-specific) Change-Id. This will allow all subsequent commits of the same change to be associated with the same patch set. Gerrit comes with an implementation; we can just copy that out.

$ cd .git/hooks
$ scp -P 29418 demo@localhost:hooks/commit-msg .
$ cd ../..

Now, when we commit amend, we'll get a Change-Id field automatically generated. We can use this to generate multiple patches on our change set. We can fix the current commit to use the same change set by doing an amended commit and putting the one shown in the Gerrit change.

$ git commit --amend -m "Hello World
>
> Change-Id: I06bf85ed12f370212ec22dbd76c115861b653cf2
> "
[master 86a7a39] Hello World
 1 files changed, 1 insertions(+), 0 deletions(-)
$ git push
Counting objects: 3, done.
Writing objects: 100% (3/3), 260 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: (W) 86a7a39: no files changed, message updated
To ssh://me@localhost:29418/example.git
 * [new branch]      master -> refs/for/master

If you now look in Gerrit at change 1 you'll see that there is a second patch set associated with the change. The remote branch refs/changes/01/1/2 now contains the new commit message (although all the files remain the same).

Normally you won't need to add the Change-Id in place, as the commit message hook will do it for you automatically.

Reviewing and submitting

To substitute a build process, we'll create a build.sh script which trivially succeeds. Jenkins/Hudson can then check this out in order to run something. Typically this would be a Maven build process or an xcodebuild or make – to avoid a language-specific environment, we're just using a script which succeeds trivially.

$ cat > build.sh
#!/bin/sh
echo Pretending to build ...
echo done
^D
$ chmod a+x build.sh
$ git add build.sh
$ git commit -m "Adding (dummy) build script"
$ git push

The problem is, these changes are still in the review queue for Gerrit. We'll need to go there and approve them. If you go into the change, you'll see the files present and a Review button. You'll note that there's a Review button on the change, which lets you review the code. There's no submit button though ...

The reason there's no submit button is because a change needs to be reviewed with a +2 by default before it can be submitted. Since each person can add a +1, this means it's not possible to submit with a single person.

We can change this by updating the rules in Gerrit to allow an individual to submit +2. We could change this by allowing an individual to vote +2; we can also do the same with updating the rules to permit a +1.

The easiest way to change it is to go into the project admin and into the all projects access tab (which was renamed from --All Projects-- to All-Projects in Gerrit 2.2.1 and 2.1.7.2). Click on the checkbox next to the “Code review” rule, and change the “Permitted Range” to go up to +2: Looks good to me, approved. Gerrit also needs a +1 verified, which we'll set up for Jenkins/Hudson as well. Add a new rule with the following:

Category

Verified

Group Name

Non-Interactive Users

Reference Name

refs/*

Permitted Range

-1: Fails to +1: Verified

Click on “Add Access Right” to set up this permission.

We'll need to create a new user for Jenkins/Hudson and put it in this group, so we'll go to sign-in page and register a new account, buildbot. We'll need a new ssh key (call it id_rsa.buildbot) and then paste in the public key as before.

Finally, sign back in as the administrator ID (the one you created earlier) and go to the Admin - Groups tab. In the list of non-interactive users, add both the buildbot and (temporarily) the demo user.

Having set up the verified rule, we should now be able to go back into the change, in order to mark it as verified.

However, we're still missing the submit button, which is what we need to merge it on to the branch. Submit rights are a separate set of rights to the verification and review rights, so we have to go back in to the all projects access tab and add an access right as before:

Category

Submit

Group Name

Registered Users

Reference Name

refs/*

Permitted Range

+1: Submit

Click on "Add Access Right" to set up this permission.

Now, when we go back into the change, we'll see a Submit button to the review (if it's been reviewed) as well as a Publish and Submit on the comments/review page. (Note that you may see a server error when doing a Publish and Submit if the code review hasn't reached the appropriate level for submission.)

Finally, publish and submit all reviews outstanding (to make sure that the build script is in place) and do a git pull to ensure that your repository is up-to-date. They should show up in the merged tab.

Note about administration: typically, you won't give 'All Projects Access' rights universally, but rather do it on a project-by-project basis. The 'All Projects' approach is being shown here to make it easy to get up and running, but these can be configured on a per-project basis as well.

Setting up Jenkins/Hudson

The final piece of the puzzle is setting up Jenkins (or Hudson, for preference). Given that they run on port 8080 by default, as does Gerrit, you need to get one to run on a different port number. Changing Gerrit involves re-running the setup procedure, but you can change Jenkins/Hudson by command line.

$ #java -jar hudson-2.0.1.war --httpPort=1234
$ java -jar jenkins.war -httpPort=1234
...

Jenkins/Hudson can check out Git projects directly, or they can check out via the Gerrit code review. However, the check out doesn't always have the ability to customise the SSH identity (other than the default set mentioned in the .ssh/config file). It can sometimes be easier to host the Git repositories by anonymous Git protocol, which doesn't need authentication. To do this, you can run:

$ git daemon --export-all --base-path=/path/to/gits

We'll also need to install the Git plugin as well as the Git/Gerrit trigger. To do this, open Jenkins/Hudson on http://localhost:1234 and click on the Manage Jenkins/Hudson link on the top left, and go into the Manage Plugins link. Switch into the Available tab, and then install:

• Gerrit Trigger

The Install button is right down the bottom; it will restart in a few moments. The Gerrit Trigger will pull in the Git plugin. Note that there is a Gerrit plugin, which is not the one needed. (If you're using Hudson, and the update site doesn't show any content, then go to 'Advanced' and click on the 'Check now' at the bottom.)

Now we're ready to start configuring Jenkins/Hudson to do the work. To start with, we'll set up a CI job that looks out for changes to the (merged) Git repository. Do the following steps:

1. Click on the 'New Job' at the top left. It will ask for a name (e.g. Example) and the type; in this case, the free-style project.
2. Select Git as the SCM type (if it's not shown, it means that you need to install the Git plugin above). The URL will be git://localhost/example.git
3. For the build triggers, check the box next to Poll SCM and put * * * * * in it. This is a crontab-like format, which in this case equates to checking the repository every minute.
4. Scroll down to add a build step and select execute shell. Put in $WORKSPACE/build.sh (note: if your project name has a space, you may need to surround this with quotes e.g. "$WORKSPACE/build.sh")
5. Finally, click on Save and the project should be created.

This has created a build which checks out master, and if it changes, executes build.sh. Clicking on the build now link should check out the code, run the script, and report success. (If this fails, check the build log for more information and resolve before going further.)

Integration with Gerrit

We've now got Jenkins/Hudson building master whenever it changes. However, the final piece of the puzzle is getting it to build whenever a new review is submitted.

Create another job with the following:

Name

Example-Gerrit

Type

Free-style

SCM

Git

URL

git://localhost/example.git

Advanced (below URL of repository) Refspec

$GERRIT_REFSPEC

Advanced (above repository browser) Choosing strategy

Gerrit-plugin

Build Triggers

Gerrit Event

Gerrit Project

Path - ** - Path - **

This will allow the project to be triggered by Gerrit whenever a new event occurs. However, we have one final piece of the puzzle to hook up – we have to tell Jenkins/Hudson which Gerrit server to listen to.

In the Manage Jenkins/Hudson tab, an entry Gerrit Trigger will have been added. Click on this and add the following information:

Hostname

localhost

URL

http://localhost:8080

Port

29418

Username

buildbot

Keyfile

/path/to/.ssh/id_rsa.buildbot

SSH Keyfile password

...

First, save it by clicking on the “Save” button below. Then, check this works by clicking the Test Connection button. If this succeeds, click on “Restart” at the bottom of the gerrit trigger to pick up the changes, or just restart Jenkins/Hudson.

If all has gone well, then you should be able to go back to the main page where it has a Query and Trigger Gerrit Patches link on the left. Click on this, and enter is:open to see any open changes, and is:merged to see any merged changes. Once that is done, check the box and hit Trigger Selected to fire off a build against that specific change.

If you are using Hudson 2.0.0 and Gerrit 2.2.0, you might find that there are java.lang.reflect.InvocationTargetException errors in the Hudson console. This was a problem with the newly upgraded Gerrit Trigger, but upgrading to Hudson 2.0.1 fixes that issue.

Putting it all together

You should now be able to make a change in a local clone of the repository, push it to Gerrit, and have Jenkins/Hudson build it automatically for you.

To test out a failed case, edit the build.sh script and put exit 1 at the end. Commit and push that to Gerrit, and you should see Jenkins/Hudson change the state to failed, because now the build script is returning a non-zero code. Amend commit to return exit 0, push, and you'll find the build is marked as verified.

Finally, if building iOS applications with xcodebuild, then piping the build through ocunit2junit.rb script, which converts the SenTest assertions into a form which is understandable by Jenkins/Hudson, such that the failed assertions can be printed on the output build log.

About the Author

Dr Alex Blewitt is founder of Bandlem Limited, and works at an investment bank in London, but still finds the time to catch up with the latest OSGi and Eclipse news. Despite having previously been an editor for EclipseZone and a nominee for Eclipse Ambassador in 2007, his day-to-day role involves neither Eclipse nor Java. In what little time he has left over, he spends with his young family and has been known to take them flying if the weather's nice. You can follow Alex on Twitter at @alblue or his blog at alblue.bandlem.com.