The NSS Build System:
History and Future Directions
Main technical contact: Bob Relyea
Manager: Wan-Teh Chang
- Special Considerations
- iPlanet Requirements for the NSS Build System
- Mozilla Requirements for the NSS Build System
Network Security Services (NSS) started out as
the base security code used by the original Netscape Navigator products.
In the beginning, this code was packaged and built as part of the monolithic
client build system. The NSS build system was disentangled from the client
build system starting with Communicator. Since then, it has gone through
a series of modifications in response to the changing needs of NSS customers,
whose products now include the following:
- Mozilla open source browser and related browser products, including Netscape browsers.
- iPlanet server products, including but not limited to the following:
- iPlanet Directory Server
- iPlanet LDAP C SDK
- iPlanet Certificate Management System
- iPlanet Web Server
- iPlanet Messaging Server
- Custom client-server applications built by other NSS customers.
Beyond the goal of merely building without error on all platforms, the NSS team has the goal of building efficient, high-performance crypto code on all platforms. NSS and its current build system contain much embedded knowledge of how to get optimal performance out of specific compilers on specific platforms, and of pitfalls (including compiler deficiencies) to avoid on specific platforms. Code that merely builds correctly on these platforms may be far from optimal for some NSS customers.
This document describes the history of the NSS build system, iPlanet and Mozilla requirements for a new build system, and some proposals for how to meet those requirements. Please send feedback on these proposals to mozilla.dev.tech.crypto.
For more background information on NSS, see the NSS Project Page.
This document uses the following terms related to the Netscape, Mozilla,
and NSS build systems:
autoconf: A build system framework that is popular in the open source community.
config: The original build system for Netscape Navigator.
coreconf: The build system, based on config, that was developed and is still used for NSS.
svbld: The internal iPlanet wide automated build and release system.
- The browser team frequently "froze" builds based on the original monolithic
build system while browser problems were being resolved. When this happened,
the NSS team was blocked and could not address the needs of its other customers
at the time, the Netscape servers.
- NSS provides the core security APIs.
- Netscape Portable Runtime (NSPR) provides a platform-neutral API for system level and libc like functions
- DBM provides low-level data-base management services.
- The client and server teams could not agree on a binary object-naming convention.
- The client team never implemented a binary import scheme. Instead, client builds were accomplished by pulling the various components from a stable NSS release tag (representing the source used to produce the latest QA'd binary release).
- A single set of Makefiles worked on all platforms (with the exception of Macintosh).
- The system could easily build separate components by importing binary versions of dependent components. It was no longer necessary to build all the dependencies prior to building one's one product. This greatly improved product development time.
- The system ran with minimal external tool dependencies: ns/tools for Windows, standard Unix commands for Unix plus either a native compiler/linker or gcc.
Therefore, as part of the modularity work on Communicator, Netscape engineers disentangled three components from the monolithic Communicator build system to be built as separate binary components. These components represented the basic code that has been shared by Netscape clients and servers since the beginning:
To faciliate this change, we created a hybrid version of the client and server build systems that made it possible to build, release, and import binary NSS releases. At this point two important things happened:
These closely related build systems had three major advantages over the client build system:
- Netscape announced the historic decision to develop all its browser
products as open source. Because of patent restrictions on proprietory
code and US export rules, we could not ship the source to our security
code at that time. Instead, we patched the client security calls with "hieroglyphs"
to obscure the security links in Communicator, which allowed us to publish
the rest of the source code.
- work with a varienty of clients, each with its own version of NSPR (or no version of NSPR)
- provide the same UI for each client
- allow multiple applications to share a common set of crytographic "tokens" and certificates at the same time.
- All the object files could be installed in a parallel tree (a feature that the Windows version of Communicator had, but other platforms did not).
- The ability to build cross plaform.
- Parallel building.
The original code contained the Netscape client's config build system (on which coreconf was originally based). Over the next couple of years, Mozilla developers converted the Unix build code to the autoconf system (used by Linux and other open source developers). It started as wrappers, and slowly moved to complete autoconf usage. At this time NSPR was integrated with a dual build system, because NSPR still needed to build and maintain binary releases as a component of the iPlanet server product releases. A separate copy of DBM was converted to the autoconf build system.
The NSS team went on to solve the problem of how we could produce a security module that would
At the same time, the Mozilla team made (or acquired) enhancements to their build system to support the following capabilities:
- In the wake of signals that the US Government was considering easing
the export regulations governing crypto software, the NSS team began the
process of sanitizing the NSS code and examining the impediments to building
a fully open source version of NSS. One issue we examined was the possibility
of converting to the autoconf build system. We attempted to build DBM as
a component without pulling the monolithic Mozilla build system. The proposal
was to give DBM its own autoconf system, which could be called separately
from the monolithic autoconf system.
The Mozilla team rejected this suggestion. No alternative was available and neither the Mozilla team nor the NSS team had the resources to address the problem, which was not a high priority for either team. Coupled with the difficulties the NSPR team was having with the dual build system, this situation led the NSS team to maintain the coreconf build system for NSS.
Because of the RSA patent, NSS used components built from proprietary licensed source code that could not be publicly disclosed. For this reason, and also because of export controls, the NSS team built PSM binaries internally and distributed them only as a complete executable program.
Another important event in 1999 was the acquisition of Netscape by AOL. Soon after, the NSS engineering team, along with the other Netscape server teams, became part of iPlanet E-Commerce Solutions.
- The RSA patent expired in September 2000, and the NSS team immediately
developed replacements for all the proprietary code on which NSS previously
depended. As a result of these changes, all NSS source could start being
posted to Mozilla. By December the PSM team had created a set of Makefiles
that could be attached to the mozilla build system.
- Changes in export regulations allowed Netscape and Sun to ship NSS
in the form of shared libraries, so the NSS team modified NSS to ship as
shared libraries in addition to static libraries. In the process we added
.def files to control what symbols are exported from our shared
To understand the situation today (April 2001), it's important to recognize that, unlike other Mozilla components, PSM, NSS, and NSPR each operate under their own release schedules and product requirement documents (PRDs). Because of this, tree locking and checkin rules will always differ between the NSS build system and the Mozilla build system.
In the future, as embedding becomes more important, more Mozilla components will start branching off on their own schedules as well. In the long term, iPlanet and other customer requirements for a separate NSS build system converge with Mozilla requirements for separate components. The only difference between NSS and other components in this respect is that NSS is a more mature component with a broader customer base.
At the same time, the Mozilla team may well have its own requirement to be able to tweak the NSS branch of the Mozilla build system without worrying about NSS release schedule or NSS team approval. Neither of these requirements have ever been stated, but it would be foolish not to recognize that they exist. These are the same requirements that led to the initial bifurcation of coreconf in 1997.
In addition to meeting the different needs of various customers and engineering
teams, the current NSS build system takes account of various specialized
customer needs. Future work on the build system must take these considerations
The main consideration involves performance. Most products built with autoconf have code that is conditionally compiled based on the presence or absence of some symbol in a system header file, such as a #define or a certain function declaration. The objective of those measures is to produce code that correctly compiles on all platforms, not using symbols or calling functions that do not exist on all systems.
Some people have wondered why NSS uses information about the particular system for which it is being built as a basis for deciding what exactly to build. The answer is that such information is vital to obtaining maximum performance in NSS. Information that is system-dependent and cannot be gleaned from header files includes the following:
- whether loop unrolling improves speed on that system or not
- whether byte writes are signficantly slower than word writes on that platform.
- whether byte reads are slower than word reads, shifting and masking
- whether the system has a FPU that is capable of doing 64-bit integer arithmetic
- whether using the FPU is faster than using the integer ALU for multiplication.
- compiler flaws--for example, does the compiler's preprocessor truncate all values to 32 bits for evaluating #if expressions?
On many platforms, assembler code is necessary to get optimal performance in time-critical functions. Such code is dependent on the type of the target system and also the tools used to build. For example, gnu assembler syntax is totally different from any system's native assembler. Such assembler code is either present or not in the product's sources, and it is built, or not, based on whether it is appropriate to the system and tools being used. Such files will continue to be built based on information about the system for which they are being built.
The iPlanet NSS engineering team's customers and build requirements are
different from those of the Mozilla team. Unlike other Mozilla components,
PSM, NSS, and NSPR operate under their own release schedules and PRDs.
Because of this, tree locking and checkin rules will always differ from
those for Mozilla. The NSS team needs to be able to make adjustments to
its build system, whether based on coreconf or autoconf, without gates
or controls by the Mozilla team. This means NSS can never be part of a
monolithic build system.
iPlanet Requirements for the NSS Build System
The iPlanet engineering team has the following general requirements for the new NSS build system, in order of priority:
- Build NSS without pulling parts of Mozilla that iPlanet products do not need or use and that are not being pulled in present NSS builds for iPlanet.
- Preserve and where possible improve the current level of performance, which is important for many iPlanet server products.
- Build NSS with minimal external tools dependencies. The bigger these dependencies, the bigger pressure on our release engineering team. When these dependencies vary greatly from the dependencies of other components built by our release team, iPlanet management starts asking nasty cost/value questions.
- Build all NSS platforms (except Macintosh, see note below) with a single build system and a single set of Makefiles.
- Build from imported binary components. Production, QA'd versions of NSS should be supplied by building with production, QA'ed, binary versions of NSPR and DBM.
- Build as a component reliably on all platforms (except Macintosh, see below).
- Build from a single checked out tree on all NSS platforms simultaneously (requirement of our svbld system).
We believe that Mozilla has the following requirements for the new NSS
build system, in order of priority:
Mozilla Requirements for the NSS Build System
- NSS should build reliably when invoked from the Mozilla build system.
- NSS should build from a read-only tree.
- NSS should build cross-platform.
- NSS should not interfere with the -j flag (that is, NSS should not force serialization of a parallel build).
- NSS should be built by some standard autoconf invocation.
We are considering two proposals for a new NSS build system. Please send
comments to mozilla.dev.tech.crypto.
- The NSS engineering team integrates the Mozilla build requirements
into coreconf. Mozilla Requirements 1 through 3 are relatively easy to
do in coreconf. It takes one day to get cross-plaform code working (there
is only one command in the coreconf build that needs to be native). coreconf
is already OBJDIR driven. It would take a couple of days to allow it to
build its objects into a separate object tree.
Mozilla Requirement 4 requires more understanding about what parallelized builds mean.
Mozilla Requirement 5 is actually the hardest, although it seems to be the lowest priority.
In the meantime, the Mozilla team is working at its own pace toward a componentized version of autoconf that can be used to produce a version of NSS as simply and reliably as the current NSS 3.2 Build Instructions.
This proposal meets all the listed requirements without using autoconf.
- The NSS engineering team continues using its own build system, and
the Mozilla team develops a parallel build system that is decoupled from
NSS (preferably in a separate tree or tag). Developers can overlay the
parallel build system on an existing NSS tree to produce an autoconf tree
that the mozilla team may use to build NSS from. This build system is maintained
by the Mozilla community.
The C code for this special tree would still be pulled from a stable, tested version of NSS and not modified. This suggests a tree where autoconf is overlayed on top of an existing NSS system. There is an example of this already in security/dbm, where the security Makefiles pull in the mozilla DBM source but use the coreconf build system to build DBM.
The disadvantage of this proposal is we put off the work componentizing the Mozilla build system until necko, gecko, or someone else needs it (which is not necessarily a bad resource trade-off).