Extension build system overview

This document provides an overview of the effort to develop a new build system for facilitating developing extensions to Kepler.  It lists the goals for the project and describes at a high level how the system meets these goals.

Goals

G1)  Facilitate modularization of Kepler and independent development of new extensions to Kepler.

• Background:  Kepler has in the past been primarily developed in monolithic fashion.  Source code for technology-specific and domain-specific actors and support classes were stored in the same repository module as general system features.  All of this code was compiled as a single unit by the build system, and released in Kepler distributions as a single unit.  Kepler stakeholders developers have requested that the Kepler code base be modularized and that development of the kernel of code used by all Kepler users be decoupled from development of extensions developed by and for specific communities.
  
• Approach:  The new build system is based on the notion of modules.  The Kepler kernel is a module, Ptolemy II is a module, and any number of system extensions can be developed as modules.  The Kepler kernel currently contains everything that was included in the Kepler 1.0 release, actors and all, but over time much of the code can be migrated to clearly defined modules outside of the kernel.
  

G2)  Enable developers to choose which version of Kepler their extensions are meant to work with.

• Background:  As long as all of the code associated with Kepler was stored, versioned, built, and distributed as a single unit it was clear what version of Kepler any actor or domain-specific capability was meant work with.  It was required to work with the rest of the current revision of Kepler at the HEAD of CVS (or on the trunk in SVN).  Because the current revision of Kepler was a moving target there was no guarantee that the current version of any actor would work with a past version of Kepler.  However, our stakeholders need to be able to develop, distribute, and support actor packages and system extensions for versions of Kepler already in the field. 
• Approach:  The new build system decouples building and release of different modules.  The system allows one to select which version of the Kepler kernel is to be used when building and testing extensions.  It also allows one to select which other extensions to include in a build or run of the system.  Ultimately this will enable developers to package and distribute their extensions independently of the Kepler kernel.  Note that one consequence of our approach is that the build system itself is, in a sense, a module too.  One checks outs the build system independently of the Kepler kernel.  Another consequence is that it generally will not be useful to check out all the Kepler modules onto a single machine; it would be unusual to find that all modules in the repository are meant to be built against the same version of the Kepler kernel.
  

G3)  Provide a way to flexibly version, share, and maintain source code for Kepler extensions.

• Background:   The previous build system compiles all source code in the main Kepler source tree, and only that code.  In the past this made it impossible to store code meant to be build against a different version of Kepler in the main code repository.  Code that worked with a previous version could break the current build of Kepler.  Consequently, projects developing code based on the Kepler beta releases or the 1.0 release had to store their code in other repositories.  The result was that it was challenging for such projects to share code with the rest of the community and stay informed about parallel efforts. 
  
• Approach:  The new build system does not require that all code be stored in a single source tree.  Rather, a developer configures the build system installed on a local machine to build any number of distinct modules stored anywhere on their machine.  Each of these modules can be stored and versioned anywhere, but we encourage projects to store their extensions in a new area in the Kepler svn repository (https://code.kepler-project.org/code/kepler/modules/) set aside for these modules to make it easier for projects to collaborate.  Each Kepler extension can be tagged and branched independently of the Kepler kernel and one another, allowing extension developers to maintain (and fix bugs) in distinct versions of their extensions meant to run on different versions of Kepler.  Again, this means that it will be rare to check out all code in the modules area at one time, as this would grab every tag and every branch of every extension.
  

G4)  Enable developers to experiment with alternative implementations of classes in the Kepler kernel without creating a complete branch of the kernel.

• Background:  Much development in the Kepler community centers around not simply developing new actors and workflows for Kepler but significantly enhancing the workflow system itself or customizing the user interface for specific communities.  Many of the chief stakeholder projects in Kepler are funded to enhance the system in such major ways.  In the past these projects had to choose how to develop these enhancements and provide their users with versions of Kepler that included them.  One option was to do all work at the HEAD of cvs (now the trunk of svn) and wait for the next release of Kepler before delivering a solution to a user community.  Another option, very commonly taken, was to take a snapshot of Kepler and work on it in a project-specific source code repository, effectively branching Kepler system development (at least temporarily) and decoupling release of project-specific enhancements from the official Kepler release schedule.  The problem with the first approach was that for a new distribution of Kepler to be released, everyone working on the centrally maintained Kepler repository had to be ready to make a release at the same time.  The second approach, on the other hand, made it a lot more work to share these major enhancements with the rest of the Kepler community, to get feedback on them during development, and to later integrate those changes to the centrally maintained version of Kepler.
• Approach:  The new build system explicitly supports development of radical new capabilities of Kepler and decoupling of project-specific software release schedules from the official Kepler release schedule.  Besides allowing modules to be developed against specific versions or releases of Kepler, it allows Kepler extensions to override classes in the Kepler release it is meant to work with.  The system automatically orders the class path at run time to respect the module priority order defined in a simple text file (modules.txt).  When more than one module defines the same class, the version of the class included in the module listed earlier in modules.txt takes priority at run time.  This approach applies the familiar Java classpath approach to Kepler modules, and makes it easy for projects to develop new major features and provide new options in the graphical user interface.  These extensions can be delivered to users who need  the feature immediately, and because the entire Kepler source tree has not been branched and stored in a separate repository, the approach makes it clear to the entire Kepler developer community (including the Kepler Framework Team) what classes are being overridden, by whom, and why. The Kepler Framework Team then has the option of incorporating these contributions into the Kepler kernel or adding well-defined extension points for supporting optional features demonstrated by overrides, thus making particular overrides unnecessary in future official releases of Kepler.  This makes it easy for projects to contribute their innovations to the official distribution of Kepler without coupling their release schedules to every other project developing with Kepler.
  

G5)  Eliminate the need for the exp directory in the Kepler trunk along with the development and build system complication it entails.

• Background: Kepler builds on the Ptolemy II system and faces challenges similar to those faced by Kepler stakeholders needing to extend Kepler in significant ways. Namely, Kepler often needs to override capabilities in Ptolemy for all the reasons given above (see G4), sometimes on a schedule that does not fit the Ptolemy release schedule.  In the past these overrides of Ptolemy classes have been stored in the src/exp directory on the trunk of Kepler. This approach introduced complexities in configuring IDEs such as Eclipse.
• Approach: The new extension build system treats Ptolemy II like any other module and allows Kepler (and its extensions) to override Ptolemy classes as needed using the approach outlined under G4 above.  The build system also automates the setup of three popular IDEs (see G8, below).  Finally, because the system decouples development of Kepler extensions from active development on the Kepler trunk, it eliminates the need to frequently update the local copy of Ptolemy on a developers' computer.  Like Kepler, Ptolemy is held constant when developing extensions for versions of Kepler that have been released and are already in the field.
  

G6)  Support extensions to Kepler that use versions of third-party libraries (i.e., jars) different from those used in the Kepler kernel or in other extensions.

• Background:  In the past, third-party jars required by the various contributions to Kepler were stored (sometimes deeply) below a single directory tree, lib/jar.  All of these jars were expected to be loaded at run time.  One consequence of this was that all contributors to Kepler have had to agree on the version of every 3rd-party jar on which Kepler depended.  It has not been uncommon, however, for different projects to required different versions of these libraries in their contributions to Kepler, and not to have the resources to upgrade (or downgrade) their contributions to an alternative version of the jar.
• Approach:  The new build system allows a module to be configured to optionally run in its own Java class loader. A modules running in its own class loader can employ  versions of 3rd-party jars differing from those used by the Kepler kernel and modules running in the same classloader as the kernel.  Modules that must run in their own class loader are listed in a simple text file, classloaders.txt, that is read by the build system. Each module-specific class loader loads only those jars unique to the module or differing in version from those in the main class loader such that jars are not loaded redundantly and so that all modules can continue to interact in the normal Java fashion, except with respect to the classes loaded from alternative versions of the same jars.
  

G7)  Make it easy to declare what 3rd party libraries are required by particular Kepler extensions and to package these libraries with the extensions.

• Background:   Kepler currently depends on a very large number of 3rd-party jars.  At the moment it is difficult to tell what parts of Kepler depend on which of these jars.  As we modularize Kepler it would be useful to be able to easily specify what modules depend on what jars, and to distribute and install only those jars required by a specific set of modules when installing or configuring a deployment of Kepler.
  
• Approach:  The new extension build system allows modules to include the jars they require in a lib directory specific to the module.  The build system loads each jar only once even if the same jar is included in multiple modules (except when multiple versions of the same jar are required, see G6 above).  Thus, as modules of domain-specific functionality are factored out of the kernel and into their own modules, the multitude of jars on the Kepler trunk but no longer needed by the Kepler kernel can be moved into module-specific lib directories. Ultimately, this approach also will allow jars needed by a particular module to be packaged and distributed easily with the module.
  

G8)  Make it easy for new developers to get started developing with Kepler and to set up their development environments.

• Background:   Getting started as a new Kepler developer can be time-consuming task especially if one intends to develop with an IDE.  The module system introduced here has the potential to further complicate straightforward development tasks unless the build system automates setup and reconfiguration of development environments.
  
• Approach:  The new extension build system automates checking out of the appropriate versions of the Kepler kernel, Ptolemy, and modules of interest to a particular developer.  It also organizes these files in a uniform way automatically on the developer's machine such that all modules are stored in a consistent manner locally.  Finally, the build system has targets for preparing project files for three IDEs (IntelliJ, Eclipse, and NetBeans) in addition to supporting full operation from the command line.