This post is copied over from my internal blog inside IBM, just for posterity after I’ve left. It was first published in March 2011.

In my Baby Steps in Scala blog post I introduced the group sync problem I used to experiment with Scala as a language. After spending a bit more time learning the language features and syntax I switched focus to start looking at the ecosystem around it, including tooling and frameworks.

In my early experiments I simply used the Eclipse Scala plugin with straightforward project structures. The plugin is pretty basic and prone to crashes. The good news however is that Martin Odersky himself has been working on improving it.

As a strong believer in test-driven development I wanted to explore the ways that Scala code can be tested. I’d already managed to write Junit tests for the group sync example that called the Scala code, but I wanted to look more at specific frameworks for Scala testing. I also wanted to understand how Scala code can be integrated into a build environment.

I’ll cover the testing aspect in this post, and build in another.

A quick Google search showed there are three common testing frameworks around Scala:

• ScalaTest
• ScalaCheck
• Specs

The interesting thing is that the focus of all of them is not just “how can I test Scala code” but more on how the capabilities of the Scala language itself can be harnessed to produce tests which are much more natural and easy to define and understand. This is primarily done through Scala’s excellent support for DSLs.

All three frameworks are also more than just Unit test libraries and execution environments. They advocate specification and behaviour driven testing.

I didn’t spend much time investigating each, but based on material I’d read I chose Specs to play around with. Specs itself can make use of ScalaCheck anyway. Specs encourages up-front specification of test cases or behaviour in a very human readable form. For instance, here is the initial specification I wrote for my group sync service:

package com.adrianspender.groupsync.scala.test

import org.specs._

class GroupSyncSpecification extends Specification {

  "group sync service" should {
    "remove any cached groups if the user is no longer in any groups" in {}

    "not change the cache if the groups for the user have not changed" in {}

    "add groups to the cache if the user has been added to new groups " +
      "so that the cache matches the groups" in {}

    "remove groups from the cache if the user has been removed from groups " +
      "so that the cache matches the groups" in {}

    "update the cache when the user has been added to new groups and " +
      "removed from old ones so that the cache matches the groups" in {}
  }
}

As you can see, the specifications are written in natural language that could easily be taken from the acceptance criteria of a story I’m working on. In true TDD/BDD fashion this can be written before any coding takes place. It can also be executed to produce the following output (in this case it is being run through sbt, but I’ll cover that in the entry on building)

(click the images to view full size)

The suite contains one test, known as a group (“group sync service” should) which then defines a set of examples such as “remove any cached groups if the user is no longer in any groups”.

We then expand the examples by specifying expectations, which is really the meat of our test and what would go inside a Junit test method. To take the first example, we need to test that if the user is not in any groups, then the cache should be emptied for them. You can look back at the Baby Steps in Scala post for the actual code we are testing.

At this point, if I was implementing this test in Junit, I’d start thinking about the collaborations that the class I am testing has, and how I can provide mock implementations of them with the behaviour I need. In this case I need:

• A mock AuthoritativeSource which will return an empty set for the given user
• A mock PersistentCache on which we need to assert that the removeAllGroups() method gets called

In Java I use the EasyMock library to provide a way to define mock objects with the behaviour I want, and to validate the runtime behaviour is as expected. Specs provides support for using EasyMock (and Mockito) within Specs and all you need to do is change your Specification to add the EasyMock trait:

package com.adrianspender.groupsync.scala.test

import org.specs._
import org.specs.mock.EasyMock
import org.easymock.EasyMock._

class GroupSyncSpecification extends Specification with EasyMock {
  ...
}

This then provides us with a Scala-friendly abstraction over the EasyMock library. We can now define our mock objects, set their expected behaviour, replay them and verify them (if you are unfamiliar with EasyMock, take a look at the tutorial here) With this in place, I am now ready to implement the expectations of the specification:

"remove any cached groups if the user is no longer in any groups" in {
      val authoritativeSource = strictMock[AuthoritativeSource]
      val persistentCache = strictMock[PersistentCache]

      val groups = new java.util.HashSet[String]()
      val personId = "1234"

      authoritativeSource.getGroups(personId) returns groups
      persistentCache.removeAllGroups(personId)

      replay(authoritativeSource, persistentCache)

      val groupSyncService = new GroupSyncServiceImpl(authoritativeSource, persistentCache)
      groupSyncService.syncGroups(personId)

      verify(authoritativeSource, persistentCache)
    }

Stepping through the code we are doing the following:

1. Creating our mock objects for AuthoritativeSource and PersistentCache
2. Creating an empty set for AuthoritativeSource to return and defining the userid we will use
3. Setting the behaviour of our mocks:
• We expect the getGroups() method of AuthoritativeSource to be called with personId, and we want it to return the empty group set.
• We then expect the removeAllGroups(personId) method to be called on PersistentCache
4. We replay both mock objects, so that EasyMock understands what should happen
5. We create an instance of our GroupSyncServiceImpl – the class we are testing. The collaborators are injected through constructor injection
6. We call groupSyncService.syncGroups(personId)
7. We ask EasyMock to verify that the behaviour we expected actually happened.

And that is one part of our specification test implemented. We can then run it again:

You can see that the test we implemented is passing. We could break the test, for instance by doing this:

      val groups = new java.util.HashSet[String]()
      groups.add("group1")

So that now the group set returned from the AuthoritativeSource is not empty. Running this produces:

Here we expected one invocation of removeAllGroups() but there were none.

Let’s fix that and implement the rest of the examples. We then have a passing set of specification tests:

Finally, we have a working set of tests, but for integration into an existing Build/Continuous Integration environment, or even within Eclipse we may want to be able to execute them as JUnit tests. We can do this simply by changing the trait we use to the SpecificationWithJunit trait and annotating the class as such:

package com.adrianspender.groupsync.scala.test

import org.specs._
import org.specs.mock.EasyMock
import org.easymock.EasyMock._
import org.specs.runner.JUnitSuiteRunner
import org.junit.runner.RunWith

@RunWith(classOf[JUnitSuiteRunner])
class GroupSyncSpecification extends SpecificationWithJUnit with EasyMock {
  ...
}

Then we can simply run the whole thing as a Junit test in Eclipse:

So, Specs is a really easy way to write human readable specifications and their implementations. What’s more, it can integrate with Junit and therefore existing CI/reporting mechanisms and existing testing libraries such as EasyMock.

However, what is perhaps most impressive is that all this ease of use and power can be applied to you Java code as much as any Scala code you have. There is nothing at all to stop you using Specs, or any Scala test framework, to test Java code. This is why testing is such a great vector for starting out with Scala and building skills. You can start to introduce it into an existing Java project orthogonally through your tests rather than diving straight into writing production code in Scala.

This post is copied over from my internal blog inside IBM, just for posterity after I’ve left. It was first published in February 2011.

I’ve been spending a lot of personal time increasing my knowledge and understanding of Scala. To spur me on I build up an introduction presentation which was presented to the IBM Dublin lab recently:

However, as previously stated, the best way to learn a language is to solve real problems in it, and one vein of problems are mathematical puzzles. I’m not a great mathematician so don’t spend all my time playing with Project Euler or similar, but when somebody posed one on a UK forum I frequent I couldn’t resist having a go in Scala. What’s more bizarre is that the forum is mainly about cars (but then cars and computer geeks sometimes go hand in hand)

Here’s the problem:

Find a 9-digit integer containing only the numbers 1-9 with no duplicates that is exactly divisible by 9.
Now remove the 9th digit to leave an 8-digit integer that is exactly divisible by 8.
Now remove the 8th digit to leave an 7-digit integer that is exactly divisible by 7.
And so on, down to a 1-digit integer that is obviously divisible by 1.

What is the 9-digit number? Is there more than one answer?

The first solution posted was in C# and absolutely unreadable. You can see it on the thread linked above. Then somebody posted a Java based solution that used a little bit of recursion, but was still very imperative. He claimed the “most elegant” solution prize, so I took that as a challenge to see what I could do in Scala.

To begin with I tried to bite off more than I could chew and tried to define a lazy stream of all 9 digit numbers that were divisible by 9. However I was ignoring the restriction that numbers could only consist of the digits 1-9 with no repeated digits. I was also trying to use parts of the language that I didn’t really understand yet.

After an hour or so I switched approach, and took the previously posted Java code and just “Scalafied” it. Then I began to swap out various loops by using higher order functions such as filter, and defining it to use a function implemented with some pattern matching to recursively divide a given number according to the spec. This was all simple enough to do. I hit one snag with the best way to convert an array of Char to an integer, and settled on using an implicit conversion (not strictly necessary as the conversion is only needed once, but I’d not defined one before).

So, here is version 1:

import scala.collection.mutable.Set

object ProblemScala extends Application {

  implicit def charArray2Int(c: Array[Char]): Int = java.lang.String.valueOf(c).toInt

  val digits = Array('1', '2', '3', '4', '5', '6', '7', '8', '9')
  val permutations = Set[Int]()

  def permute(digits: Array[Char], n: Int): Unit = n match {
    case 1 => permutations += digits
    case _ => {
      for(val i <- 0 until n) {
        val j = n - 1
        swap(digits, i, j)
        permute(digits, j)
        swap(digits, i, j)
      }
    }
  }

  def swap(digits: Array[Char], i: Int, j: Int): Unit = {
    val c = digits(i)
    digits(i) = digits(j)
    digits(j) = c
  }

  def divisible(n: Int, divisor: Int): Boolean = divisor match {
    case 1 => true
    case _ => n % divisor == 0 && divisible(n / 10, divisor - 1)
  } 

  permute(digits, digits length)
  val result = permutations filter (n => divisible(n, 9))

  result.foreach(i => println("match found: " + i))
  println("total match count: " + result.size)
}

The permute and swap functions are used in building a set of all possible 9 digit numbers that can be made up of the digits array contents. This was pretty much a straight copy from the Java code, except I use pattern matching in permute rather than an if/else.

The main difference as mentioned is the divisible function and how that is used in calculating the value of ‘result’. I was quite pleased with that. Overall it read much cleaner than the Java version in my eyes. However I was not happy with this solution. It doesn’t drastically cut down the line length and there’s still too much Java-like stuff going on. It is however a good example of how Scala lets you dip your toes into the water slowly rather than throwing you in at the deep end. Just take your existing code and “Scalafy” the syntax, then start looking for the loops, vals and Units, and start refactoring them out.

Once a days work got out of the way, I started googling around for a better solution to the problem of determining the permutations of the 9 digit numbers. In the meantime, a couple more solutions had been posted in Lua and Python. Both used useful library functions for building permutations, so my thoughts turned to what I might be missing in the Scala libraries. Sadly, there is nothing directly useful in 2.8.1, and many Google results that discuss generic solutions which are quite complex and would turn my solution into an unreadable mess. I did however find reference to the fact that the as yet un-final Scala 2.9.0 does have a SeqLike.permutations function. I set up a new SBT project (more on SBT in another post) using the latest 2.9.0 nightly snapshot and got to work. The result is quite amazing:

object ProblemScala2 extends App {

  def divisible(n: Int, divisor: Int): Boolean = divisor match {
    case 1 => true
    case _ => n % divisor == 0 && divisible(n / 10, divisor - 1)
  }

  "123456789".permutations filter (n => divisible(n.toInt, 9)) foreach(n => println("match found: " + n))
}

The library function allows the String “123456789″ to be implicitly converted to a SeqLike when the permutations function is called on it. The result is an Iterator of all permutations. That can then be used as the input of the rest of my first solution. All the cruft code in the first solution to define all the permutations is completely gone and what’s left is an amazingly concise and readable solution. I think it wins the “Most elegant” prize and I’ll be surprised if anybody comes up with something better.

Scala 2.9.0 should be released in the next few months. It is not a major evolution (compared to 2.8) but this is one fun example of how the language and core libraries are getting better and better.

This post is copied over from my internal blog inside IBM, just for posterity after I’ve left. It was first published in January 2011.

As a new years resolution to myself I have started to learn Scala, a static, type-inferenced; object-oriented but yet functional programming language that runs on the JVM. After ten years of concentrating mainly on server side Java programming I figured it was time to broaden my skill set a bit, as well as stretch the brain cells. I chose Scala because at heart I am a middleware/server guy and whilst I’ve worked on a product and with a team that does a lot with Javascript, it has never really grabbed me.

I’ve started reading Beginning Scala by David Pollak via Books 24×7, as well as various web articles, including a series on DeveloperWorks. However, the best way to learn new things is always to apply them to real problems. Therefore I took one particular, fairly self contained coding problem from Lotus Connections 3.0 and re-implemented it in Scala. Here’s how.

Group Synchronization in Scala

In various places in Connections we support access control via LDAP groups. We have no way of knowing when group membership changes, so typically retrieve the list of groups a user is in when they log in. This information is then used a lot during their session so it is beneficial to cache it (typically for ten minutes before we refresh the cache) It may also be beneficial to cache this data persistently so any part of the system can use it. Ideally we would simply use a distributed in-memory cache such as WebSphere Extreme Scale, but we don’t yet, so where appropriate we cache some things in the database.

So, there is a fairly straightforward bit of code that would need to synchronize the groups the user is in with the cached set of groups we have for the user in the DB. Most of the time they will simply be the same, but sometimes they will be added to new groups, or removed from old ones. So what this code would basically do is:

1. Retrieve the list of groups the user is in from the authoritative source (e.g. LDAP)
2. If they are in no groups, ensure the cached entries (if any) are removed)
3. If they are in one or more groups, compare that with the cached version and add any new groups to the cache, and remove any groups from the cache they are no longer in.

For my Scala investigation I started out with the following assumptions:

• I need to call the implementation of this sync service from existing Java code
• I have two existing Java services for interacting with the authoritative source and the persistent cache.
• I have an existing Java interface for the sync service that I need to ensure is implemented

One of the great things about Scala is that because it runs on the JVM and is compiled down into byte code, it is fully able to interact with existing Java code. Java can invoke Scala and vice versa. Scala classes can implement Java interfaces and all sorts of goodness. You can run Scala code in WebSphere.

So, first off, here is the Java Interface of the sync service itself. It is very simple:

/**
 * Provides methods to synchronize data stored locally in a
 * persistent cache with the authoritative source.
 *
 * @author aspender
 *
 */
public interface GroupSyncService {

	/**
	 * Retrieve the groups that the given person ID is in
	 * from the definitive resource, and update the
	 * persistent cache that we use ourselves.
	 *
	 * @param personId the id of the person to sync
	 */
	public void syncGroups(String personId);

}

I also have two existing services defined by interfaces called AuthoritativeSource and PersistentCache. The former provides a single method to retrieve the groups the user is in. The latter retrieves the cached copy of the groups, as well as providing methods to update the cache.

So, as a starting point, here is an example Java implementation of the GroupSyncService:

package com.adrianspender.groupsync.java.impl;

import java.util.HashSet;
import java.util.Set;

import com.adrianspender.groupsync.java.*;

/**
 * The Java version of our group sync service.
 */
public class GroupSyncServiceImpl implements GroupSyncService {

	private AuthoritativeSource authoritativeSource;
	private PersistentCache persistentCache;

	public GroupSyncServiceImpl(AuthoritativeSource authoritativeSource, PersistentCache persistentCache) {
		this.authoritativeSource = authoritativeSource;
		this.persistentCache = persistentCache;
	}

	public void syncGroups(String personId) {

		Set<String> currentGroups = authoritativeSource.getGroups(personId);

		if(currentGroups.size() == 0 ) {
			persistentCache.removeAllGroups(personId);
		} else {
			Set<String> cachedGroups = persistentCache.retrieveAllGroups(personId);

			Set<String> addedGroups = new HashSet<String>();
			for(String groupId : currentGroups) {
				if(!cachedGroups.contains(groupId)) {
					addedGroups.add(groupId);
				}
			}
			persistentCache.addNewGroups(personId, addedGroups);

			Set<String> removedGroups = new HashSet();
			for(String groupId : cachedGroups) {
				if(!currentGroups.contains(groupId)) {
					removedGroups.add(groupId);
				}
			}
			persistentCache.removeGroups(personId, removedGroups);
		}
	}
}

I’ve deliberately left out comments and it also isn’t as defensive as i’d normally write code (e.g. no null checks – something which Scala doesn’t need so much anyway) The code is easy to follow, but note the fact there are two for loops doing the calculation of the difference between the two sets of groups (first one works out the groups that are not in the cache, the second works out the groups that should no longer be in the cache)

So, here’s my first go at a Scala version:

package com.adrianspender.groupsync.scala.impl

import com.adrianspender.groupsync.java._
import scala.collection.JavaConversions._

/*
 * The Scala version of our group sync service..
 */
class GroupSyncServiceImpl(authoritativeSource: AuthoritativeSource, persistentCache: PersistentCache) extends GroupSyncService {

	def syncGroups(personId: String) : Unit = {

		val currentGroups = authoritativeSource.getGroups(personId).toSet

		if (currentGroups.size() == 0) {
			persistentCache.removeAllGroups(personId)
		} else {
			val cachedGroups = persistentCache.retrieveAllGroups(personId).toSet

			persistentCache.addNewGroups(personId, currentGroups diff cachedGroups)
			persistentCache.removeGroups(personId, cachedGroups diff currentGroups)
		}
	}
}

This does exactly the same thing. The unit tests I wrote work exactly the same regardless of whether you use the Java or Scala version. The same Java Junit test works against both. Also note that the Scala version uses exactly the same Java implementations of AuthoritativeSource and PersistentCache. And that the Scala class implements the same GroupSyncService Java interface.

Of course, the other major thing to notice is the syntactic difference. There is no constructor – the variables of the class are defined on the class signature. There are no semi-colons. There are strange looking statements like ‘currentGroups diff cachedGroups’ which to a Java programmer may look more familiar if written as currentGroups.diff(cachedGroups) – which you could. However the syntax is still inherently familiar and easy to understand. This is certainly a big benefit of Scala. There’s much less need for nested blocks of code thanks to some of the syntactic sugar, as well as some of the capabilities of the languages and it’s libraries. The Java class is 47 lines to Scala’s 24 lines.

I’m not going to go into a line by line analysis of the Scala version, this blog post isn’t meant to be a tutorial. However one other point of note is how easy the collection libraries in Scala make working with arrays, lists, sets and maps. In this case I use the diff member of the scala.collection.immutable.Set class to extract the values from one set that are not present in the other. Simple, and something that in Java you have to loop to do. The collection capabilities really come into their power when combined with the functional aspects of Scala – for instance being able to pass a functional block of a code as a parameter to a method allows you do to complex things in single lines of code. There isn’t an example of functional programming in this particular example, but it is definitely something I need to get my head around further.