My ten-year quest for concise lambda expressions in Java
A mission to hold off the horde of for loops in Java.
A series of fortunate and unfortunate events
I started learning Java in 1997. I thought initially Java would be a fad and that Smalltalk would emerge as the victor in the battle for object-oriented programmer productivity over C++. I was wrong. In the great object-oriented battles in the 1990s between C++ and Smalltalk, Java emerged as the victor.
Y2K marked the end of my career as a professional Smalltalk developer. I stopped programming in Smalltalk professionally for the promise of a higher-paying career in Java. Over two decades later, this would turn out to be the best decision I ever made in my career. I miss Smalltalk, but it’s never too far away from my keyboard.
My journey with Java has helped me evolve as a software engineer, learner, and teacher. I know things now that if I had continued programming professionally in Smalltalk, I might have never had the opportunity to learn and share with so many other developers. My Smalltalk experience gave me a solid perspective on what Java was missing. This knowledge and experience created a great opportunity for me to become a teacher, and to learn things more deeply as a result. Teaching a Smalltalk way of thinking to thousands of Java developers globally has enabled me to achieve a higher level of appreciation for many things. I have helped many Java developers learn how to elevate their coding style so they can communicate their intent with a more expressive vocabulary. I’ve seen quite a few developers build great things with the skills they have developed. This is everything to me.
In the rest of this blog, I will explain how a dislike for reading and writing for loops became so acute that it led to me creating an open-source collections library (Eclipse Collections) for Java inspired by Smalltalk. This then set me on a quest to find a way to help Java get good syntactical support for concise lambda expressions.
Hold the for
I dislike reading for loops in application code. I see for loops as a sign that a developer failed to learn and use a higher level abstraction to communicate what they are doing with an intention revealing name. For loops are a necessity in languages like Java. With the minimal interface design approach of Java Collections, the for loop became the “Hodor” of developer communication. For anyone who hasn’t heard of “Hodor”, he is a character in the “Game of Thones” series who says only one word: Hodor! I will refer to the style of programming where all iteration patterns are written with for loops as Hofor!
By 2004, everything I was reading in Java code was Hofor!, Hofor!, Hofor!, and… Hofor!. I got paid to read Hofor! every day. I got to write a lot of Hofor! as well. This is pedestrian level programming that I would gladly welcome our new AI overlords to automagically replace with higher level abstractions employing intention revealing names.
The following example shows simple Hofor! style coding in Java. By itself, this code is easy to read and may not seem too bad. But wait for it… it will get worse.
public List<Person> getListOfPeople()
{
List<Person> people = new ArrayList<>();
people.add(new Person("Bob", "Smith"));
people.add(new Person("Sally", "Taylor"));
return people;
}
@Test
public void forLoopFindUniqueLastNames()
{
List<Person> people = this.getListOfPeople();
// Find the unique last names of the people
// Hofor!
Set<String> lastNames = new HashSet<>();
for (Person each : people)
{
lastNames.add(each.lastName());
}
Assertions.assertEquals(Set.of("Smith", "Taylor"), lastNames);
}Using a method with the intention revealing name can replace the for loop here. Extracting for loops into their own methods became a regular activity for me so I could quickly read code that communicated its intent, instead of requiring me to painstakingly translate the how to the what by reading every line in a for loop.
In the following example, I use an Extract Method refactoring to create a method named getUniqueLastNames.
@Test
public void findUniqueLastNames()
{
List<Person> people = this.getListOfPeople();
Set<String> lastNames = this.getUniqueLastNames(people);
Assertions.assertEquals(Set.of("Smith", "Taylor"), lastNames);
}
private Set<String> getUniqueLastNames(List<Person> people)
{
// Hofor!
Set<String> lastNames = new HashSet<>();
for (Person each : people)
{
lastNames.add(each.lastName());
}
return lastNames;
}This might not look too terrible to most Java developers, at least if they were programming before Java 8 was released.
As a former Smalltalk developer, this code made me want to scream. I should not have had to create the method getUniqueLastNames. I should have been able to use a method on List for this common iteration pattern known as collect, map, or transform.
It should have also been trivial to convert a List to a Set, or if efficiency was a concern, transform the elements directly into a Set. Instead, we had to use Hofor!
After four years of reading and writing code like this in Java, I was becoming increasingly motivated to do something to fix it. Reading thousands of for loops and extracting methods, knowing there is a missing level of abstraction was too much for a former Smalltalk developer to bear. I saw many former Smalltalk developers abandon Java for Ruby or Groovy or even JavaScript. I decided to take a different path and stick with Java.
I would have to slow down, in order to speed up.
Flotsam and Jetsam
The code example above may not seem bad on its own. So let’s show two more examples that should make the duplication obvious and more painful to read.
In the following example I add a second method named getUniqueFirstNames and the code in the test asserts both the unique first names and last names for the people.
@Test
public void findUniqueFirstAndLastNames()
{
List<Person> people = this.getListOfPeople();
Set<String> firstNames = this.getUniqueFirstNames(people);
Set<String> lastNames = this.getUniqueLastNames(people);
Assertions.assertEquals(Set.of("Bob", "Sally"), firstNames);
Assertions.assertEquals(Set.of("Smith", "Taylor"), lastNames);
}
private Set<String> getUniqueFirstNames(List<Person> people)
{
// Hofor!
Set<String> firstNames = new HashSet<>();
for (Person each : people)
{
firstNames.add(each.firstName());
}
return firstNames;
}
private Set<String> getUniqueLastNames(List<Person> people)
{
// Hofor!
Set<String> lastNames = new HashSet<>();
for (Person each : people)
{
lastNames.add(each.lastName());
}
return lastNames;
}You should see some clear structural similarities between getUniqueFirstNames and getUniqueLastNames.
For me once, shame on you. For me twice, shame on me.
There is one more example I will share below, in order to help drive home the point through repetition. I will add a method named getUniqueFullNames.
@Test
public void findUniqueFirstLastAndFullNames()
{
List<Person> people = this.getListOfPeople();
Set<String> firstNames = this.getUniqueFirstNames(people);
Set<String> lastNames = this.getUniqueLastNames(people);
Set<String> fullNames = this.getUniqueFullNames(people);
Assertions.assertEquals(Set.of("Bob", "Sally"), firstNames);
Assertions.assertEquals(Set.of("Smith", "Taylor"), lastNames);
Assertions.assertEquals(Set.of("Bob Smith", "Sally Taylor"), fullNames);
}
private Set<String> getUniqueFirstNames(List<Person> people)
{
// Hofor!
Set<String> firstNames = new HashSet<>();
for (Person each : people)
{
firstNames.add(each.firstName());
}
return firstNames;
}
private Set<String> getUniqueLastNames(List<Person> people)
{
// Hofor!
Set<String> lastNames = new HashSet<>();
for (Person each : people)
{
lastNames.add(each.lastName());
}
return lastNames;
}
private Set<String> getUniqueFullNames(List<Person> people)
{
// Hofor!
Set<String> lastNames = new HashSet<>();
for (Person each : people)
{
lastNames.add(each.fullName());
}
return lastNames;
}For me three times, shame on both of us. — with apologies to Stephen King
Hofor! Hofor! Hofor! The structural similarity between these three methods is screaming “Hold the for!”
Help would soon be on its way.
Fate of Sisyphus or Temporary Eye Gouging
By 2004, I had enough of this painful loop code duplication. They say necessity is the mother of invention. I needed to end this for loop code duplication in Java. I viewed it as an existential threat to my sanity as a developer.
I turned to the only tool I had available to address the problem in Java at the time. This tool was Anonymous Inner Classes (AIC). Unfortunately, every time I use an AIC in Java, I feel like I am sticking a sharp instrument in my eye. Strangely, this felt better than having to roll the same ball up a hill like Sisyphus for eternity. Java IDEs with code folding made it less painful to read AICs, so the pain was only in writing the code. That pain was also lessened by IDEs that generated the boilerplate required to implement an Anonymous Inner Class.
I will walk you through the thought process I went through to remove the duplication in the above example. I will create a single method named getUniqueValues which takes the List and a Function. Function will be an interface with a single abstract method named apply which will take a parameter and return some result. The getUniqueValues method which is single purpose here will later be converted to use a reusable method named collect and which is part of the Eclipse Collections API.
@Test
public void findUniqueFirstLastAndFullNamesWithAICs()
{
List<Person> people = this.getListOfPeople();
Set<String> firstNames =
this.getUniqueValues(people, new Function<Person, String>() {
public String apply(Person person) {
return person.firstName();
}
});
Set<String> lastNames =
this.getUniqueValues(people, new Function<Person, String>() {
public String apply(Person person) {
return person.lastName();
}
});
Set<String> fullNames =
this.getUniqueValues(people, new Function<Person, String>() {
public String apply(Person person) {
return person.fullName();
}
});
Assertions.assertEquals(Set.of("Bob", "Sally"), firstNames);
Assertions.assertEquals(Set.of("Smith", "Taylor"), lastNames);
Assertions.assertEquals(Set.of("Bob Smith", "Sally Taylor"), fullNames);
}
private Set<String> getUniqueValues(
List<Person> people,
Function<Person, String> function)
{
// Hofor!
Set<String> values = new HashSet<>();
for (Person each : people)
{
values.add(function.apply(each));
}
return values;
}By passing a Function in using an AIC, I can alter the one part of the code that was different, which is whether to use firstName, lastName, or fullName.
OMG! How is this even remotely better than the code I wrote before!?!? Why would anyone write code like this! This code is terrible. My eyes are bleeding! Please, make it stop!
Were it not for code like this with AICs that I wrote from 2004–2014 in Java, it’s possible we might not have gotten lambdas in Java as soon as we did. If Java didn’t get lambdas in Java 8 by March 2014, some of us Java developers who were exhausted from writing for loops would instead be programming in Groovy, Scala, Kotlin, C#, JavaScript or some other language that had lambda support built-in from the beginning.
I chose temporary eye gouging over the fate of Sisyphus. This was a choice of higher level abstraction or being stuck with pedestrian programming with for loops forever. Programming is a creative endeavor. Building and using higher-level abstractions to solve problems fuels our creativity as developers. The fact that Anonymous Inner Classes are hideously ugly and temporarily blind the developer who uses them could not hide the beauty and benefit of the higher level abstractions.
It’s been almost ten years since Java 8 was released. My eyes have healed, and the Anonymous Inner Classes are mostly gone, replaced by concise lambda expressions.
Rowing to the end of the known Java world
In 2004, I knew there was a better route to programming productivity. I would just have to row a boat across an entire ocean of developer time in order to get there. The years from 2004–2014 were lonely for me as a Java developer who believed lambdas in Java were an inevitability. I set out to convince others I worked with that Java needed lambdas so badly that it would have to eventually get them. I built a library inside Goldman Sachs called Caramel which was later open-sourced as GS Collections in 2012. I was successful in convincing hundreds of developers inside of Goldman Sachs that it was better to write code using higher level method abstractions and using Anonymous Inner Classes than it was to write tens of thousands of unnecessary for loops. I convinced developers in Goldman Sachs that lambdas would arrive eventually, and using Caramel would prepare them for a new lambda-enabled Java programming language. I also suggested they would likely be able to convert Anonymous Inner Classes to Lambdas through automated refactorings.
The following is an example of what I told developers would eventually be possible with Java with reasonable syntactic support for lambdas.
@Test
public void findUniqueFirstLastAndFullNames()
{
List<Person> people = this.getListOfPeople();
Set<String> firstNames =
this.getUniqueValues(people, each -> each.firstName());
Set<String> lastNames =
this.getUniqueValues(people, each -> each.lastName());
Set<String> fullNames =
this.getUniqueValues(people, each -> each.fullName());
Assertions.assertEquals(Set.of("Bob", "Sally"), firstNames);
Assertions.assertEquals(Set.of("Smith", "Taylor"), lastNames);
Assertions.assertEquals(Set.of("Bob Smith", "Sally Taylor"), fullNames);
}
private Set<String> getUniqueValues(
List<Person> people,
Function<Person, String> function)
{
// Hofor!
Set<String> values = new HashSet<>();
for (Person each : people)
{
values.add(function.apply(each));
}
return values;
}By 2006, I started getting in touch with folks at Sun and then Oracle pleading my case fervently that we needed to get Lambdas into the Java programming language. I watched the first set of Java lambda battles play out between 2006–2010 with BGGA, CICE, FCM. While these battles were happening and no one seemed to be winning, I kept quietly rowing my boat (building the Caramel library), and teaching Java developers about lambdas using the programming language I love — Smalltalk. I worked with several other amazing developers for years at Goldman Sachs building a bigger, better boat that would eventually become Eclipse Collections. I knew that once I reached land with lambdas for Java, folks on the Island of Hofor! would want to be able to leave the island for loops and experience the power of lambdas with feature-rich collections that could immediately leverage them.
I struggled at first with how to teach developers what using lambdas would be like in Java. Between 2007–2014 we taught around 200 Java developers in Goldman Sachs a week-long class in Smalltalk. The experience using Smalltalk gave them a better appreciation for OO programming and TDD. I couldn’t show these developers how to use lambdas in Java until Java 8 was released in March 2014. Instead, I was able to show them how to use lambdas with rich collections in Smalltalk, and this showed them what would be possible eventually in Java. They were able to see what I saw every day. I saw the potential for land, well, lambdas landing in Java eventually anyway.
This is how I picture the code examples above with my Smalltalk lenses on.
testFindUniqueFirstLastAndFullNames
|people firstNames lastNames fullNames|
people := OrderedCollection
with: (Person new firstName: 'Bob'; lastName: 'Smith')
with: (Person new firstName: 'Sally'; lastName: 'Taylor').
firstNames := (people collect: #firstName) asSet.
lastNames := (people collect: #lastName) asSet.
fullNames := (people collect: #fullName) asSet.
self assert: firstNames equals: (Set with: 'Bob' with: 'Sally').
self assert: lastNames equals: (Set with: 'Smith' with: 'Taylor').
self assert: fullNames equals: (Set with: 'Bob Smith' with: 'Sally Taylor').This is the syntax for an individual method in Smalltalk. The method name is testFindUniqueFirstLastAndFullNames which I added to a class called PeopleTest. The method has no parameters. Smalltalk is dynamic, so there is no need to specify a return type for the method, or types for any variables. The pipes around |people firstNames lastNames fullNames| create four local variables in Smalltalk. An OrderedCollection in Smalltalk is the equivalent of Java’s List type. I create an OrderedCollection containing two instances of the class Person. The method collect: transforms the Person instances in the OrderedCollection to String in these three cases, using the specified accessors.
I did not actually use any lambdas in the code above, as Pharo Smalltalk made it possible to simply use a Symbol (unique String marked with a #) instead. This is the closest equivalent in Smalltalk to using what we now know as Method References in Java. If I used lambdas with the collect method instead, the code above would have looked as follows. Lambdas in Smalltalk are delineated with square brackets with a pipe in the middle— [|]. The text :each to the left of the pipe defines a parameter named each. The code on the right of the pipe is the expression that will be evaluated.
firstNames := (people collect: [:each | each firstName]) asSet.
lastNames := (people collect: [:each | each lastName]) asSet.
fullNames := (people collect: [:each | each fullName]) asSet.The method asSet converts the OrderedCollection instances that are created by calling collect: to Set instances which will make sure the values are unique.
Medium doesn’t support syntax highlighting for Smalltalk syntax, so I chose the closest compatible language which is Ruby. It was a nice feeling writing a small bit of Smalltalk for this blog. I used Pharo Smalltalk 11.0 as my Smalltalk IDE to create the Person class and this supporting unit test code above.
Writing simple code like this without for loops makes me very happy. It might surprise you if I told you that Smalltalk doesn’t have for loops. It doesn’t have statements of any kind. Everything in the Smalltalk language is “object message.” Of course, this might also explain why I dislike using for loops so much in Java.
Land Ho(for)!
Around 2011, I joined the JSR 335 Expert Group. Working on the JSR 335 EG was one of the greatest experiences in my career. Our mission was to get a working specification for lambdas into the Java Language. We succeeded. Java 8 was released in March 2014 with support for Lambdas, Method References, Default Methods, and Java Streams. Java finally had decent language syntax support for lambdas! Win!
Almost two years before the successful release of Java 8, in July 2012, I gave a talk at the JVM Language Summit. I shared what I saw as the potential for lambda support in Java to improve the quality and quantity of code that developers had to write. The slides for the talk are still available online here. Slide 24 of the PDF shows the code for the pattern named collect in GS Collections (now Eclipse Collections). The collect pattern in Eclipse Collections is the equivalent of the pattern named map in Java Streams.
If we use Java lambda expressions with the Eclipse Collections collect method to solve the problem above, the code will look as follows:
@Test
public void findUniqueFirstLastAndFullNamesEclipseCollections()
{
MutableList<Person> people = Lists.mutable.with(
new Person("Bob", "Smith"),
new Person("Sally", "Taylor"));
Set<String> firstNames =
people.collect(person -> person.firstName()).toSet();
Set<String> lastNames =
people.collect(person -> person.lastName()).toSet();
Set<String> fullNames =
people.collect(person -> person.fullName()).toSet();
Assertions.assertEquals(Set.of("Bob", "Sally"), firstNames);
Assertions.assertEquals(Set.of("Smith", "Taylor"), lastNames);
Assertions.assertEquals(Set.of("Bob Smith", "Sally Taylor"), fullNames);
}As you can see, there is no need for an extra method named getUniqueValues to create a Set based on some Function applied to each element of the collection. No more Hofor! The MutableList class has the behavior collect which can transform elements in the list from one type (Person) to another (String). The toSet method then converts the MutableList to a MutableSet. MutableSet extends java.util.Set.
Eclipse Collections or Java Streams?
The answer to this question is yes. When I first began my quest for lambdas in Java, I didn’t imagine we would get anything quite like Java Streams in Java 8. I wanted eager methods on the collections themselves, and needed lambdas to reduce the verbosity. Eclipse Collections gives developers eager methods on collections, and so much more. Eclipse Collections has great integration with Java Collection Framework types and provides support for Java Streams, so there is no need to make an either/or decision. You can use Eclipse Collections and Java Streams together. They are complementary.
The following shows how to use an Eclipse Collections MutableList with Java Streams to solve the example problem. I am going to use method references here instead of lambdas to make the code even less verbose. I have a “method reference preference” in Java. While I wanted lambdas so much in Java, method references would be an extra gift in Java 8 that I now can’t imagine programming in Java without.
public static MutableList<Person> getMutableListOfPeople()
{
return Lists.mutable.with(
new Person("Bob", "Smith"),
new Person("Sally", "Taylor"));
}
@Test
public void findUniqueFirstLastAndFullNamesJavaStream()
{
MutableList<Person> people = this.getMutableListOfPeople();
Set<String> firstNames =
people.stream()
.map(Person::firstName)
.collect(Collectors.toSet());
Set<String> lastNames =
people.stream()
.map(Person::lastName)
.collect(Collectors.toSet());
Set<String> fullNames =
people.stream()
.map(Person::fullName)
.collect(Collectors.toSet());
Assertions.assertEquals(Set.of("Bob", "Sally"), firstNames);
Assertions.assertEquals(Set.of("Smith", "Taylor"), lastNames);
Assertions.assertEquals(Set.of("Bob Smith", "Sally Taylor"), fullNames);
}This code is slightly more verbose than using the eager collect method directly on the Eclipse Collections MutableList. Eclipse Collections also has custom support for lazy iteration using its own asLazy method which returns a LazyIterable type.
The following shows how to use Eclipse Collections with asLazy to solve the same problem.
@Test
public void findUniqueFirstLastAndFullNamesAsLazy()
{
MutableList<Person> people = this.getMutableListOfPeople();
LazyIterable<Person> lazyPeople = people.asLazy();
Set<String> firstNames =
lazyPeople.collect(Person::firstName).toSet();
Set<String> lastNames =
lazyPeople.collect(Person::lastName).toSet();
Set<String> fullNames =
lazyPeople.collect(Person::fullName).toSet();
Assertions.assertEquals(Set.of("Bob", "Sally"), firstNames);
Assertions.assertEquals(Set.of("Smith", "Taylor"), lastNames);
Assertions.assertEquals(Set.of("Bob Smith", "Sally Taylor"), fullNames);
}There are some subtle difference here between Java Stream and Eclipse Collections LazyIterable. A Java Stream can only be used once. A LazyIterable can be used as many times as you need. The other difference is that Eclipse Collections API has many converter methods available directly on LazyIterable (toList, toSet, toBag, toMap) whereas Java Stream has to rely on the collect method used with Collector instances, for everything other than toList, which was added to Java Stream in JDK 16.
The point I would stress is that you can use Eclipse Collections types and Java Streams very well together.
After ten years of lambdas in Java
I took a big gamble in 2004 building a collections library in Java that needed lambdas, ten years before Java would have them. I couldn’t wait for battles over lambda syntax to start fighting what I saw as the real battle, which is teaching Java developers how to code using higher levels of abstraction with rich collection interfaces that leverage lambdas. I had to get used to the temporary eye gouging with Anonymous Inner Classes. I guess ten years in programming language time can be considered temporary in a venerable language like Java, even if at the time it felt like an eternity. I loved applying automated refactorings to replace Anonymous Inner Classes with lambdas and method references on entire code bases after Java 8 was released. I gave many Java developers inside of Goldman Sachs an amazing head start on learning and using lambdas effectively in Java 8 when they became available. If any of the Java developers I worked with in Goldman Sachs whom I taught Caramel, GS Collections, or Eclipse Collections are reading this blog, I would like to say two things.
You are very welcome! Thank you for believing!
I haven’t worked at Goldman Sachs for several years now. Six years ago I created an open source Java Lambda Kata at my current employer, to help developers I worked with learn how lambdas work in Java and how to use them effectively. I was told yesterday by a developer I work with who has been out of university for two years that this same Lambda Kata was the first time she had experienced using lambdas in Java. This surprised me. I had kind of expected lambdas to have become part of the Java curriculum in universities by now. It seems Hofor! is still being taught in university.
So here I am, Java lambdas in hand with arguably the best Java collections framework available (Eclipse Collections) still fighting the battle of Hofor! with new developers. I am ok with this. I enjoy teaching developers about using lambdas with Java and Eclipse Collections. There is a growing arsenal of Eclipse Collections Code Katas that leverage lambdas that I can use to arm developers with amazing productivity tools and to defend against Hofor!
Thank you for reading my story. It only took me two decades to tell it. I hope you found it interesting and informational. I wonder what the next decade will bring. More Java with lambdas and Eclipse Collections most likely.
Best of luck with Hofor! in your Java coding adventures!
I am the creator of and committer for the Eclipse Collections OSS project, which is managed at the Eclipse Foundation. Eclipse Collections is open for contributions.
