Using Multiple Class Loaders

Introduction

Whenever one executes a program written in Java, by default, they are using three class loaders. The most basic class loader is the bootstrap class loader. It loads core java classes (those found in rt.jar and i18n.jar, for example) from jars in the lib directory of your Java installation. This class loader is implemented in native code (i.e. C, C++ or assembly) rather than in Java. Unlike all the other class loaders, if you call getClassLoader() on a class loaded from the bootstrap class loader, you will get null. The next class loader is known as the extension class loader. It loads classes from extensions stored in jars in the lib/ext folder of your Java installation, as well as extensions classes from operating system specific jars stored in an operating system specific folder. The final class loader that you use by default is known as the system class loader. It loads classes that are found on the class path in the order that they exist on that path.

Hierarchy of Class Loaders

A natural question arises: With so many class loaders by default, how does a particular class know which one to use? The simple answer is that a class uses the same class loader as the object that references it. Thus, if class B is referenced by class A, then B will try to use the same class loader as A. In the case of all classes referenced in the main method where execution starts, the class loader that is used is the system class loader. (This is because it is the system class loader that loads the object that contains the main method.) Thus, by default, all classes try to use the system class loader. They use the system class loader either because they are loaded by the main method, or because their ancestor was loaded with the system class loader, which occurred because the ancestor was loaded in the default manner by the main method.

So, if by default, if all classes are loaded with the system class loader, how do the bootstrap class loader or the extension class loader ever come into play? The answer is that they come into play because the system class loader chooses to defer to them. The first thing that the system class loader does (besides checking to see if the class is already loaded, in which case there is no need to reload the class) is let its parent, the extension class loader, load the class. Only if the extension class loader fails to load the class does the system class loader try to load the class using the class path. The extension class loader engages in the exact same behavior. Before trying to load the class, it asks its parent, the bootstrap class loader, to try to load the class. Only if the bootstrap fails to load the class (because the class is not a class core to the Java system) does the extension class loader examine the jars in the lib/ext folder to determine if this is a standard extension that it should load.

In this way, a hierarchy is established. First the bootstrap class loader gets a crack at loading the class. If that fails, then the extension class loader tries. If that fails, finally the system class loader will look at the class path and try to load the class from there. But from the perspective of the class that is being loaded, it is always the system class loader that is loading the class. It is the system class loader the chooses to defer to the extension class loader and the extension class loader that choose to defer to the bootstrap class loader.

Limitations of Default Class Loading

For smaller projects involving one or a small number of developers, or for projects that are highly centralized where a powerful organizational body dictates precisely what third-party software is permissible and dictates that a particular version be used throughout all modules, the default class loading mechanism works adequately in most circumstances. However, when you have a system that has multiple modules and you wish to allow independent modules to use different versions of the same third-party software, the default class loading system falls apart. Each class that is loaded into the virtual machine is uniquely identified by three things. Its name, its package, and its class loader. So, for example, if the extension class loader loads a class, that class will be uniquely identified by references to its class name, package name, and a reference to the extension class loader. Third-party software loaded from jars is typically handled by the system class loader via a reference to the class path. Without a custom class loading mechanism (or without messy hacks involving the misuse of the extension class loader), it is impossible to fully load multiple versions of the same jar into the same virtual machine for two reasons. First, the system class loader is going to stop when it finds the first instance of a class with a particular name and package on the class path. So, when two jars contain a class with the same name and package, that class will always be loaded from the first jar on the class path. Second, even if the system class loader were to continue its search along the class path, ultimately two classes would be loaded with the same class loader (i.e. the system class loader), the same package, and the same name. This would violate basic architectural constraints (each loaded class MUST be uniquely identified by its name, package, and class loader) and would not be allowed.

As a caveat, it should be noted that it is in fact possible to make use of classes from the different versions of the same jar on the class path. If version 1 of A.jar contains class X from package P1, and version 2 of A.jar contains class X, and class Y both from package P1, and version 1 of A.jar is on the class path before version 2 of A.jar, then a request for class X will load from version 1 of A.jar and a request for class Y from will load from version 2 of A.jar. This is a bad situation. If class Y refers to class X, it clearly wishes to interact with version 2 of class X. But what it will in fact be interacting with is version 1 of class X. This could lead to some nuanced and difficult to discover errors. In short, great care should be used when constructing class paths. Care that has heretofore been almost entirely lacking in the management of the Kepler project, which in fact has different versions of the same jar messily stuffed into different locations of the lib/jar folder.

In any case, the basic problem is clear. If two relatively independent modules wish to use different versions of the same jar (i.e. classes with the same name and same package) this is impossible to do with the default class loading mechanism. This problem has been acknowledged by Sun Microsystems, which has developed Java Specification Request 277, a specification for a Java Module System to deal with it. However, this system is not slated to be incorporated until Java 7 is released, which is currently scheduled for January 2009. The Mac OS X release is sure to be even later. In the meantime, we have defined a lightweight custom module and class loading system that cleanly solves this problem.

The Solution

Our solution is to use an optional module that defines a custom class loader that will be used instead of the system class loader. To minimize overhead, this class loader behaves in a manner identical to the system class loader in most cases. That is, it simply defers requests to load a class to the system class loader, which in turn defers to the request to the extension class loader, which in turn defers the request to the bootstrap class loader. However, in special cases where a module developer wants to use a different version of a jar that is already in use in Kepler, he or she can specify in a file that a separate class loader be used for all classes and all unique jars defined in that module.

There are two types of jars that a module might contain. Ones that are unique to that module and ones that are not unique. Unique jars include jars where the class and package name combination have not been used elsewhere in the system (where the "system" is defined as all lower priority modules, including Kepler and Ptolemy). Unique jars also include those where the class and package names have been used elsewhere in the system, but where a different version of the jar is in use. In contrast, non-unique jars are all jars where the class and package names have already been used in the system, and those arise from the same version of the jar.

Those modules that are specified to use a separate class loader handle unique and non-unique jars differently. If the jar is non-unique, the module class loader simply defers to the class loader in the system that has already has been slated to handle the precise class that is being requested. If the jar is unique, it handles the loading itself.

The end result is this. It is now possible to have different modules depend on different versions of the same jar. That is, module X can use version 1 of A.jar and module Y can use version 2 of A.jar and these modules can happily coexist in the same system. But, there is also a limitation, one that is inherent to Java and cannot be overcome by any system whatsoever. If different versions of the same class are loaded by different class loaders, then modules that use those different class loaders cannot exchange objects of that type. That is, if module Y tries to call a method in module X and that method returns a type that is defined in both version 1 and version 2 of A.jar, then a class cast exception will occur, because although the package and name of the class are identical, they are loaded from different class loaders.