If you've been a developer for any length of time, you'll have very likely realized that the word module is perhaps one of the most overused terms in software development. A module can mean anything ranging from a group of code entities, components, or UI types, to framework elements to complete reusable libraries. Sometimes, we use the word to imply multiple meanings in the same context!

There is a good reason for that. When writing code, we typically try to break the code base down into smaller units in order to manage complexity. For anything more than very simple programs, having a monolithic code base is not a good idea. That's why modular programming is a generally favored software design approach. There are two important goals that modularity in software development usually achieves, which are as follows:

• Divide and conquer approach

What do you do when you need to solve a large and seemingly insurmountable problem? You break it down! You'll very likely split it into smaller problems and solve them individually.

The principles of modularity encourages separating large code bases into smaller encapsulated units of functionality that are then composed to work together as a bigger unit. This aligns well with the approach we humans usually take to solve large problems. Also, once you've got a bunch of smaller modules with specialized concerns, you can use those to solve various other problems. Thus, we also achieve reusability!

• Achieving encapsulation and well-defined interfaces

When you build modules, you have the ability to hide the internal implementation from the consumers of your module. The hidden implementation details are usually referred to as being encapsulated, and what you expose to the consumers of your module is usually called the interface of your module.

Although Java developers have leveraged many different patterns and best practices over the years in order to write and structure modular and maintainable code, the language has never had native support to create modular units and build modular applications, until Java 9. With Java 9, Java developers now have the ability to create smaller units of code with a new construct called Java modules that they can group together like building blocks in order to compose larger applications. In addition to introducing this feature to the language, Java 9 also comes with what is probably the biggest overhaul to the core Java code base itself. The Java Runtime Environment (JRE) and the Java Development Kit (JDK) have been rewritten to use the concepts of modularity so that the core Java Platform itself is modularized.

When learning about Java 9 module features, it's important to understand what those new features add to the language when compared to the other features the language already has. Can't we write well organized code in Java 8? In fact, one of the benefits of object-oriented programming is indeed the idea of breaking down functionality into sub-units called objects or classes. We've been writing code like this in Java since version 1. Every Java class contains a portion of the overall application functionality that happens to belongs together. We have the ability to encapsulate some functionalities as internal to a class (as private) and some others as external (or public).

And then there's something in between with protected, thanks to the concept of packages.

Think about why we use packages in Java. We could very well write entire Java applications without creating any packages and, thereby, using just the default unnamed package. It would work! However, unless it's a simple or throwaway application, that's not a good idea. The idea of packages is to group your Java types into namespaces that signify the relationship, or perhaps a common theme among those types. It makes code easier to read, understand, and navigate.

The following diagram shows an example of classes organized in packages. Adding all classes to a single package (left) is not good practice. We typically group related classes into well-named packages that describe the nature of the classes in them (right):

There's really no rule about what types belong together in a package. However, it's generally understood that when you create a package and put a bunch of Java types in it, the types are usually related in some way. You could very well write any random set of types in the same package and the compiler wouldn't care. However, anyone else who ends up working on your code could potentially hate you forever, so this is not a wise thing to do! Having related types in common packages also has the benefit of those types being able to access the protected members of each other. This is another level of encapsulation--any protected members or methods are encapsulated within types of a package. (Although, there's an exception to this, as inherited classes are able to access private members across packages.)

So, if the idea of modular programming is to break code and functionality into encapsulated units, there's a sense in which you can do some kind of modular programming in Java well before Java 9.

The following table shows the various ways in which you can encapsulate code in Java before Java 9:

Isn't that good enough? Well, not really. The preceding table is where a limitation in the modular ability of the language becomes apparent. Notice the What to encapsulate column. Most of the encapsulation features provided by these modifiers focus on controlling access to member variables and methods. The only way you can really protect access to a type is by making it package-protected. That, unfortunately, ends up making access difficult for even your own library code to access the type, and you are forced to move all the code that accesses that type into the same package. What if you want more?

Why, you ask? There are a couple of problems with approaching modularity with just the preceding paradigm available in Java 8 and earlier. Let me explain both those problems with two stories.

Meet Jack. He's a Java developer at a medium-sized enterprise organization. He's a part of a team that writes code to do data processing. One day, Jack wrote some Java code to sort a list of usernames in alphabetical order. His code worked well without any errors and Jack was proud of his work. Since this was something that could be used by other developers in the organization, he decided to build it as a reusable library and share it with his colleagues as a packaged JAR file. Here's the structure of Jack's library:

His code belonged to two packages--acme.util.stringsorter and acme.util.stringsorter.internal. The main utility class was StringSorterUtil with one method--sortStrings. The method in turn internally called and delegated the sorting responsibility to the BubbleSortUtil.sortStrings() class  from a class in the acme.util.stringsorter.internal package. The BubbleSortUtil class used the popular Bubble Sort algorithm to sort a given list of Strings.

All that any developer had to do was to drop the jar in the classpath and call the StringSorterUtil.sortStrings() method by passing in an list of strings they needed sorting. And they did! Jack's little library became a hit! His colleagues loved the convenience that his library provided and they started using it to sort many things, such as names, tokens, addresses, and so on.

A few months later, Jack happened to talk to Daryl at the water cooler, and as usual, their conversation veered towards a discussion about their current favorite sorting algorithms. Daryl couldn't stop talking about his new-found love for hash sort. He said he found it performs much better than bubble sort, and it was unabashedly his new favorite algorithm! Jack was intrigued. He went to his desk and ran a few tests. Daryl was right! Hash sort outperformed bubble sort in most of his tests. Jack knew right then that he had to update his sorting utility to use hash sort. He added a new class, HashSortUtil in the acme.util.stringsorter.internal package and removed BubbleSortUtil.

The following is the structure of Jack's library after the change:

Thankfully, he had a separate internal class that did the sorting, so the process to invoke the StringSorterUtil.sortStrings() utility wouldn't change. Everyone could just drop in the newer version of the JAR and everything would work just fine.

But it didn't! A few of the code builds in his company started failing. It turned out the culprit was the newer version of Jack's library. Jack couldn't believe it. He didn't miss anything, did he? Well, no. All the projects that used just the StringSorterUtil class worked just fine. However, it turned out that some of the developers ended up using the BubbleSortUtil class in the internal package directly. It was available in the classpath, so they had just imported and used it. Now, since that class didn't exist in the new jar anymore, their code couldn't compile!

Jack sent out an email instructing everyone using BubbleSortUtil to update their code to use StringSorterUtil instead. However, it turned out the BubbleSortUtil class was being used in multiple places by that time, and it wasn't an easy task to change them all. "Couldn't Jack just put the BubbleSortUtil class back?" they asked. Jack yielded to their requests and the next version of the library had both the SortUtil classes (and would possibly do so well into the foreseeable future), even though it internally used only one of those two classes.

After the dust settled, Jack sat at his desk and wondered what had gone wrong. What could he have done to prevent this problem? Clearly, naming the package as internal did not prevent developers from using it. One solution would have been to write that internal bubble sort type as package-protected and move the external type to the same package. This way, he could leverage the third mechanism in the preceding encapsulation table. However, he liked the idea of separating the bubble sort class into its own type and package. Also, imagine if this were a bigger library and there was a common shared class that was supposed to be internal. In that case, pretty much all types in that library that need the internal type have to exist in the same package as that internal type! Wasn't there a better way to encapsulate the internal types?

Meet Amit, a deployment engineer at yet another enterprise technology firm. His job is to make sure that during every product release, the organization's code base is compiled and deployed properly in the production environment. During every release, he pulls in the application code and all the necessary jar files and places them in the classpath. He then starts the application that results in the Java Virtual Machine (JVM) loading all the classes and initializing execution.

One night, there was a major product feature release. There were a lot of changes to the code that were all supposed to be deployed and launched together. Amit made sure that all the new code was compiled properly and he had all the necessary jars in the classpath. He then had to start the application. Before he clicked on the button to launch the build, Amit wondered if there was some way he could make sure everything was good and that the application would work without any runtime class errors.

One thing that could potentially go wrong was if he had missed adding a certain class or jar in the classpath. Was there a way he could statically verify whether all the classes were available without actually running the application?

Each JAR bundled a set of types in a set of packages. Each type therein could potentially import other types, either from the same JAR or from other jars. To make sure he has all the classes in the classpath, he has to go to each class and verify that all its imports are in the classpath. Considering that the number of classes in his application run to thousands, it's a Herculean task.

The following diagram is a simplified version of what a sample deployed Java application looks like:

There are four jar files in the picture above, each of which contains packages and classes within them. Jar 1 is deployed in Classpath A, Jar 2 and Jar 3 in Classpath B, and Jar 4 in Classpath C. Let's assume each jar has two classes as indicated by the smaller white boxes. The three paths are configured as classpaths for the Java runtime, so the runtime knows to look at all three paths to scan and pick up classes.

After scanning all the classpaths, this is what the structure looks like to the Java runtime:

Notice that the runtime doesn't care which directory or classpath the package/type is in. It also doesn't care which jar the package/type is bundled in. As far as the Java runtime is concerned, it's just a flattened list of types in packages!

In Java, a classpath is a just set of paths. Any of those locations could have the jars and classes that the application needs to work. You can immediately see how easy it is for things to break! There's always a possibility that some of the classes that the application uses are not available in the classpath. Perhaps a missing jar or library. If the runtime doesn't have a specific class it needs, the application could start running fine, but throw a NoClassDefFoundError much later. That too, only when the execution hits a point where a missing class is actually needed.

This is a huge and very real problem in large Java applications today. There is a whole ecosystem of solutions that have sprung up to address this. For example, tools and build utilities, such as Maven or Gradle, standardize the process of specifying and acquiring external dependencies. Process-based solutions such as continuous integration aim to solve the unpredictable nature of builds across various development environments. However, all that such tools can do is make the process predictable. They cannot verify the validity or accuracy of the result that they help assemble. Once the dependencies are fetched, there's nothing that those tools can do to detect missing or duplicate types in the classpath.

Back to Amit's story. Having no way to verify whether all the classes are available up front, Amit hopes for the best and deploys the application. The application starts up fine and runs for a couple of hours without any errors. However, there's still no saying if he's got it right. Maybe there's a class in there that hasn't been executed yet, but when it has, the JVM might realize that it cannot find one of its imports. Or, maybe, there are duplicate versions of the same class in the classpath and the JVM picks up the first copy it finds. Wasn't there a better way to ensure that any given Java application will work reliably in advance?

We've seen two problems in Jack's and Amit's stories. Jack needed an effective way to encapsulate portions of his library, but couldn't. Amit needed a way to ensure reliable execution of his application without actually executing it. Both Jack and Amit didn't really have a solution to their problems because of the way classpath resolution works in Java. We may sometimes mistakenly think of a JAR file as a way to build a reusable module in Java, but that's unfortunately not the case. A JAR file is just a convenient bundle of classes. Nothing more! Once in the classpath, the JVM treats classes in a JAR no differently from separate class files all in the same root directory. At runtime, as far as the JVM is concerned, an application is just a set of classes in a flat list of packages.

What's worse is, once a class is in the classpath, it's free for all. It's incredibly easy for any developer to use a type they are not supposed to, or a type that might be available for them during compile time, but not at deployment/runtime. Or there could be multiple copies or even multiple versions of the same class in two different classpath locations, making it unpredictable which version the runtime will actually pick up during execution. There's a problem commonly called JAR hell, which refers to several issues resulting from mismatched and incorrect classes and versions in JAR files.

This problem is exacerbated in huge code bases with hundreds of thousands of classes. Imagine all those classes in your application as a flat list with no structure! It's a nightmare to maintain and organize. The bigger the code base, the bigger the problem. To illustrate this, let's take the classic example of a code base that's written in Java, that's incredibly large and complex, and has lasted for many years now. It is perhaps one of the oldest Java code bases ever, and still it continues to grow and change at a fairly rapid pace. Any guesses? Well, it's the Java platform itself!

Talk about a monolith! Java has come a long way since its first release in 1996. The first major release of the JDK had a little over 500 public classes. A far cry indeed from JDK 8 released in 2014, which counts upwards of 4,200 public classes and over 20,000 total files.

The following commands extract the rt.jar file, a library JAR file bundled in JDK 8, and count the number of classes in it. With the Java 8 version I have installed on my machine, the count is 20651:

The JDK and the runtime, the JRE, have continued to grow over the years. There are a lot of features that have been added to the language, so this growth is understandable. However, the Java language is also notorious for going to great lengths to maintain backward compatibility and for its reluctance to deprecate features unless it is absolutely necessary. So, in a way, the current size of the runtime is a little over what it could have ideally been.

Normally, most application developers wouldn't need to worry about the JDK code base. They just focus on their application code. However, the contents of the runtime does matter for application execution because of the way it is bundled. Traditionally, every JRE has had all the classes necessary for runtime bundled into a single JAR that resides in the lib directory called rt.jar. The name rt, as you might have guessed, stands for runtime.

Not only is this huge monolith of classes unnecessarily bulky in size, it also adds performance overheads for the Java Virtual Machine to manage. And that's a price the execution environment of all your applications have to pay, irrespective of whether all of those classes are being used or not.

A good example of classes that you don't need is the set of classes in the JRE related to CORBA. Ever heard of CORBA? If you haven't, don't despair. It's for a reason! It's an old technology that was introduced to the Java runtime back in version 1.4. It has mostly fallen out of popular use since then. Considering most applications don't use the CORBA technology anymore, wouldn't it be great if apps could be bundled with JREs that do not contain the unnecessary CORBA classes?

That's unfortunately not possible, again, because of rt.jar. Since everything gets bundled into a single runtime JAR, you cannot pick and choose what features you need. Everybody gets everything. And since the runtime has been increasing in size, so has the standalone deployable application. This is a more significant challenge when the runtime needs to be used on smaller devices with limited resources. If you are bundling the runtime with a simple Hello World application that uses just a handful of classes from the runtime, you have no option but to bundle a whole lot of unused classes in rt.jar with it. And, yes, even those old CORBA classes join in for the ride!

Java 8 introduced the concept of profiles, and with that, you can technically deploy smaller runtimes. But they do have some drawbacks. Also, this feature was just an initial step in the introduction of modularity features in Java 9 anyway. We'll examine compact profiles in detail in Chapter 4, Introducing the Modular JDK.

Remember the problem that Jack had with his BubbleSortUtil class? It was a Java class he wrote with the intention of it being private to his library. However, even though it started out as a private internal class, it ended up being a public class because other developers just decided to use it.

That was just a small library. Now, think about a library as big and as widely used as the Java runtime. The Java runtime obviously bundles in internal classes that are required for its functioning and aren't meant to be used by application developers. However, considering the magnitude of its usage, it isn't surprising that some of the internal classes are inadvertently used by developers anyway.

A classic example of this is a class called Unsafe in the sun.misc package. This ominous sounding class has been a part of every major JDK release for a while now. Can you guess what it does? It contains a collection of methods that perform, according to the author of the class, low-level unsafe operations. Yes, it actually says that in the comments in the class! For instance, it has a method that gets a value from a memory address. Not a typical day's work for a Java application developer! You wouldn't, and ideally shouldn't, do something like that as an application developer. This is why the class has been marked as an internal API. Want to look up its Javadoc to use it? You won't find it in there. Want to create a new instance of the class? Its constructors are marked as private. If you do somehow use it and compile your code, every Java compiler since Java 6 will give you a nasty warning, discouraging the usage of the class. And, if you still need more reasons to avoid using it, you'd be best served to just look at the name of the class!

You must have guessed what's coming by now. The sun.misc.Unsafe class has now been used in multiple projects by many developers to perform those very low-level operations, in spite of all those preventive measures that the Java runtime authors have put in place. One could argue that it implements functionality that isn't commonly available elsewhere, and for a developer who needs to do something like that, nothing beats just picking it up while it's available in the classpath and ready to use. Unsafe isn't the only internal API that is being used this way, of course. There are a few more internal classes, many in the sun.* packages, that developers have used over the years even though they shouldn't. Moreover, as long as developers continue to use these APIs, it becomes harder to remove them from the runtime. This has ended up continuing the existence of these classes in subsequent versions of the runtime, thereby allowing more developers to use them!

These limitations of the Java runtime and library system have been felt for a while now. All the problems I've outlined so far exist because of the lack of ability to create modular units of code in Java. Such a construct simply hasn't existed in the language so far. The need for it has been strongly felt in the community.

Multiple proposals for a module system for Java have been made over the years, including JSR-277 way back in 2005 (https://jcp.org/en/jsr/detail?id=277) and JSR-294 (https://jcp.org/en/jsr/detail?id=294) in 2006. After facing several hurdles, modularity is finally coming to Java with the 2017 release of Java 9 with JSR-376 (https://jcp.org/en/jsr/detail?id=376), the spec titled Java Platform Module System, as well as Project Jigsaw.

The modularity features in Java 9 are together referred to by the name Java Platform Module System (JPMS). It introduces a new language construct to create reusable components called modules. The Java Platform Module System makes it easy for developers to create contained units or components that have clearly established dependencies on other modules. With Java 9 modules, you can group certain types and packages into a module and provide it with the following information:

• Its name: This is a unique name for the module
• Its inputs: What does the module need and use? What's required for the given module to be compiled and run?
• Its outputs: What does this module output or export out to other modules?

I'll explain the input and output configuration shortly, but at a very high level, these three pieces of information are what you typically supply when you create a new module. Whenever developers need to create any components that are meant to be reusable, they can create new Java modules and provide this information to create a unit of code with a clear interface. Since a module can contain both its inputs and outputs specified formally, it adds a whole set of advantages compared to the classpath approach that we've critiqued so far.

Let's now look at the process of creating a module step by step. We'll look at it at a conceptual level now, and we'll cover the syntax in Chapter 2, Creating Your First Java Module. Let's say you want to create a reusable library and you've decided to put your code in a Java 9 module. Here are the steps you need to follow:

1. Create a module and give it a name: Every module has a name associated with it, for the obvious purpose of referring to it. You can give a module any name that you'd traditionally give to types. All the rules you are already familiar with regarding Java package names apply here (so certain characters like '/' or '-' aren't allowed, but '_' or '.' are okay). The recommended way to name a module is to use the reverse domain name convention, similar to the way you name your packages. So, for example, if someone in Acme Corp wrote an analytics module, they'd probably name the module com.acme.analytics.
2. Define the module inputs: Not many modules can realistically be self-sufficient. You'll often need to import types that aren't in your module. This is where the module input configuration comes into play. When you create a module, you explicitly need to declare which other modules you need for your code to work. You do that specifying which modules your module requires.
3. Define the module outputs: We've seen that in a traditional JAR file system, placing Java types in a JAR file doesn't really mean anything and every public type is accessible to every other type in the classpath, irrespective of which JAR it is in. A module behaves differently. By default, every Java type you place in a module is accessible only to other types in the same module. Even if the type is marked public! In order to expose types outside the module, you need to explicitly specify which packages you want to export. From any module, you can only export packages that are in that module. Once you've exported a package, all types in that package are potentially accessible outside the module. This enables every Java module to clearly separate and hide internal packages that are to be used only inside the module and expose only types that are intended to be used externally. If a Java type is in a package that isn't exported, then no other type outside the module can import it, even if the type is public!

Note the difference between the things you export from a module (which are packages) and the things you import or require (which are other modules). Since we are exporting types from a module at a package level, why not require packages too? The reason is simple. When a module requires another module, it automatically gets access to all the packages that that module exports. This way, you don't have to specify every package that your module needs. Just the name of the module you depend on will suffice. We'll look at the access mechanisms in much more detail in Chapter 3, Handling Inter-Module Dependencies.

The following diagram illustrates the input and output definitions of a typical module:

JPMS was designed with two primary goals in mind:

• Strong encapsulation: We've seen the dangers of having every public class accessible to every other class in the classpath. Since every module declares which packages are public and isolates those which are internal, the Java compiler and runtime can now enforce these rules to make sure that none of the internal classes are being used outside the module.
• Reliable configuration: Since every module declares what it needs, the runtime can check whether every module has what it needs well before the application is up and running. No more wishing and hoping that all the required classes are available in the classpath.

You can guess how happy Jack and Amit would be to hear this! Thanks to strong encapsulation, Jack would just need to put all of his StringSorter code in a module and export just his public package. Thus, his internal package would be hidden and not accessible by default. And, thanks to reliable configuration, Amit can always confidently say whether a given set of modules have all their dependencies met before running the application.

In addition to these two core goals, there has been another important goal that the module system was designed for--to be scalable and easy to use even on huge monolithic libraries. As a validation of that, the Java 9 team went ahead and modularized what's pretty much the oldest and biggest Java code base they could get their hands on--the Java Platform itself. This task, something that ended up involving significant effort, was performed under the name Project Jigsaw.

Alan Bateman, a member of the Java Platform Group at Oracle said this is in his talk in Java One in September 2016:

No matter what the application is about, there's one set of libraries that every Java program is guaranteed to use without a doubt--the Java Platform. For Java developers to be writing modular Java code, it's essential for the core Java platform and the JDK library to be modular as well. Before Java 9, all the classes and types in the JDK had such complicated inter-dependencies that they resembled a big bowl of spaghetti.

Not only is the final rt.jar bundle unnecessarily large, it makes the JDK code base itself harder to change and evolve. Considering how any type in such a huge code base could be used by any of the other thousands of types in the platform, I wouldn't want to go in there and make any major changes to that code. Another problem with the platform is that it has always lacked ways to hide and encapsulate internal platform APIs such as sun.misc.Unsafe. The platform itself could very well use the same strong encapsulation and reliable configuration benefits that JPMS gives us.

With Java 9, we've finally got a modular JDK to build on top of. Various different sets of related JDK classes are bundled into separate modules, each with its own imports and exports. For example, SQL related types are in a module called java.sql. XML functionality has gone into the java.xml module, and so on. We'll be looking at these out-of-the-box modules in more detail in Chapter 3, Handling Inter-Module Dependencies.

The following is an illustration of a subset of the new Java 9 platform modules. Don't worry about the individual names. We'll cover platform modules in detail in Chapter 4, Introducing the Modular JDK:

Project Jigsaw claims the following as its primary goals. It's important to keep this in mind as you learn about the impact of the modularization of the platform:

• Scalable platform: Moving away from a monolithic runtime and enabling the ability to scale the platform down to smaller computing devices.
• Security and maintainability: Better organization of the platform code to make it more maintainable. Hiding internal APIs and better modular interfaces to improve platform security.
• Improved application performance: Smaller platform with only the necessary runtimes, resulting in faster performance.
• Easier developer experience: The combination of the module system and the modular platform to make it easier for developers to create applications and libraries.

What does this mean for application developers? The most immediate difference is that not all types in the JDK are now accessible in your code. The same mechanisms we saw apply to our modules work with the Java modules too. Any time you depend on a platform class, you'll have to import into your module the right platform module that contains that class. And, even then, you'll be able to use the class only if it has been exported from the module and is public.

This way, the JDK code base also gets all the advantages of the strong encapsulation and reliable configuration that the JPMS promises. There are potential backward compatibility issues though. What if you used a JDK class in JDK 8 or earlier that's now an encapsulated class in a module? That code wouldn't work in Java 9! The platform uses the encapsulation features to protect certain internal JDK classes from external use. So, any code that depends on such classes in Java 8 or earlier cannot be migrated to Java 9 without removing that dependency first. There are a few challenges associated with moving code from Java 8 or earlier to Java 9. We'll look at Java 9 migration-related challenges and best practices in Chapter 9, Module Design Patterns and Strategies.

Another important aspect of modularity that most modular platforms have to deal with, and we haven't covered so far, is versioning. Are modules versionable? Can you declare dependencies between modules that specify which versions of the modules need to work together? You cannot! Java Platform Module System does not support versioning today. We'll briefly examine the reasons why in Chapter 3, Handling Inter-Module Dependencies.

In this chapter, we looked, at a high level, at some limitations of the traditional way of building reusable components in Java using JAR files. We saw how packaging libraries in JAR files doesn't allow developers to encapsulate inner APIs and types. There's also no way to reliably figure out whether a given application has all the necessary classes in the classpath. We learned how these problems that developers face in their code are not only present in the JDK code base itself, but are actually an issue on a much bigger scale. We understood the Java Platform Module System and the two primary goals that it set to achieve--strong encapsulation and reliable configuration. We learned about Project Jigsaw and the effort to modularize the core JDK using the same modular paradigm that's available to developers to use in their code.

At this time, you are probably wondering how the concept of modularity manifests in the Java language. What does a Java module look like?

In the next chapter, we'll answer these questions by creating our first Java 9 module, and get started on our sample application project which we'll be working on throughout this book.