Building Large Java Projects Faster: Multicore javac and Makefile integration

Building Large Java Projects
Faster: Multicore javac and
Makefile integration
Fredrik Öhrström
Principal Engineer & Patent Liaison

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.3
The following is intended to outline our general product direction. It is intended
for information purposes only, and may not be incorporated into any contract.
It is not a commitment to deliver any material, code, or functionality, and should
not be relied upon in making purchasing decisions. The development, release,
and timing of any features or functionality described for Oracle’s products
remains at the sole discretion of Oracle.

What is the output from javac?
From a single Java source file, javac can generate:
 Zero, to an infinite number of classes with unpredictable file names.
(package-info.java, inner classes and package private classes
not matching the file name)
 If annotation processors are in use, then anything can happen.
(New sources, new packages, new xml-files, png-files etc)

How javac (re)compiles code
Javac requires explicit source files on the command line (or @file)
 The explicit sources are always compiled and linking dependencies
outside the explicit sources, are read from the classpath.
(eg java.lang.Object, org.antlr.runtime.CharStream, etc)
 If the to be linked dependencies are not yet compiled,
then use -sourcepath srcroot to link to source code.

Unfortunately, the default behavior for to be linked sources found through
the -sourcepath, is to emit classes for these sources. Unless:
 If -implicit:none is supplied on the command line
 Or the corresponding class in the (boot)classpath or -d directory is newer than
the source.
If you develop jaxp (which is part of the jdk), then a jdk installed after
your source was written, will override your source in the sourcepath,
even though your source is probably newer.

Compiling, Headers and Linking in the Java world.
 At runtime, linking of imports is done lazily when the class is loaded.
 At compile time, usage of imports are checked against the class files
in the (boot)classpath.
 At compile time, usage of imports are checked against the source
files in the sourcepath or explicit sources to be compiled.
But these checks will trigger a full compile of the referenced
classes. Not only their public apis.
 This makes it very easy to create circular dependencies.

Implicit generation of classes and accidentally not recompiling your
own source code, is usually not a problem. Because most developers
do not develop code that resides in the jdk.
 But implicitly generated classes contribute to the confusion of what
javac outputs!

How a makefile works
imperative statements
goal: requirements
action that generates goal
# A makefile is executed imperatively to create
# a dependency tree from goals to requirements

How a makefile works
FLAGS += Os
app: app.c app.h
gcc $(FLAGS) o $@ $<
# Then a sequence of actions are calculated
# to generate the goal given on the command line.
# A goal with a newer timestamp than its
# requirements is considered up to date.

How a makefile works, macro calls
$(eval $(call SetupJavaCompilation,BUILD_JAXP,
     SETUP:=GENERATE_NEWBYTECODE,
     SRC:=jaxp/src,
     BIN:=output/classes,
     SRCZIP:=output/src.zip),
     JAR:=output/classes.jar))
all: output/src.zip output/classes.jar

A javac command line
javac bootclasspath classes d classes src/java/lang/Object.java
src/java/lang/CharacterData.java src/java/lang/OutOfMemoryError.java
src/java/lang/Character.java src/java/lang/Override.java
src/java/lang/CharacterName.java src/java/lang/packageinfo.java
src/java/lang/CharSequence.java src/java/lang/Package.java
src/java/lang/ClassCastException.java src/java/lang/ProcessBuilder.java
src/java/lang/ClassCircularityError.java src/java/lang/Process.java
src/java/lang/ClassFormatError.java src/java/lang/Readable.java
src/java/lang/Class.java src/java/lang/ClassLoader.java
src/java/lang/ClassNotFoundException.java
src/java/lang/ReflectiveOperationException.java
src/java/lang/ClassValue.java src/java/lang/Runnable.java
src/java/lang/Cloneable.java .....
and ca 8000 more files....
 You cannot really use the command line for large projects.
 An @-file can be used to work around the command line length problem.
 We need to use an external tool.

An ant build file
<project name="MyProject" default="dist" basedir=".">
  <property name="src" location="src"/>
  <property name="build" location="build"/>
  <property name="dist"  location="dist"/>
  <target name="init">
    <tstamp/>
    <mkdir dir="${build}"/>
  </target>
  <target name="compile" depends="init" description="compile it>
    <javac srcdir="${src}" destdir="${build}"/>
  </target>
  <target name="dist" depends="compile" description=”generate jar”>
    <mkdir dir="${dist}/lib"/>
    <jar jarfile="${dist}/lib/MyProject${DSTAMP}.jar"
         basedir="${build}"/>
  </target>
</project>

An attempt at makefile
SRC:=.../src
BUILD:=.../build
DIST:=.../dist
DSTAMP:=$(shell date +'%Y%m%d')
SOURCES:=$(shell find $(SRC) name ”*.java”)
$(DIST)/lib/MyProject$(DSTAMP).jar : $(BUILD)/the_build
    mkdir p $(@D)
    (cd $(BUILD) ; jar cf $@ *)
$(BUILD)/the_build : $(SOURCES)
    mkdir p $(@D)
    javac $(SOURCES) d $(BUILD)
    touch $@
 No incremental compile
 No incremental jar update
 Limit on the number of source files!

The major obstacles to writing makefiles for Java
 unpredictable output from javac
 for any decent size Java project, the filenames of the Java sources
do not fit on the command line!

Whither ant? Whither make?
It does not matter if you prefer make or ant. You cannot do proper
dependency tracking and incremental builds in Java because:
 Javac lacks the necessary features! This cannot be fixed, lest you
write your own Java compiler, which is a lot of work.
It does not matter if you prefer make or ant. You cannot get proper
multi core support for speeding up compilation times for large Java
projects, because:
 Javac lacks the necessary features! This cannot be fixed externally.

Whither ant? Whither make?
But what about ant depend?
 Limited to what it can see in class files, eg cannot see where final
statics came from.
But what about ant fork=yes?
 Does not work within a single javac task. Manual splitting of sources
is needed.

Whither IDEs?
 IDEs like Eclipse and Netbeans do keep a database of the source
dependencies in memory. Needed for quick and efficient incremental
compiles while editing!
 But very limited support for multi core compiles.
 We need ant/make for batch compiles on build servers anyway.

A first attempt at improving javac
 added a server to javac
 added -XDdepfile=......
 added -XDnativeapifile=.....
 added -XDpubapifile=....
 Relied on make (sic!) to split compile into multiple cores with a javac
invocation for each package.
 This worked, sort of, but lets look at the makefile.

A horribly complex (tiny) part of a makefile
$2 : $$($2_PACKAGE_SRCS)
    $(MKDIR) p $$($2_PACKAGE_BDIR)
    $(RM) $2.tmp
    $$(call ListPathsSafely,$2_PACKAGE_SRCS,n, >> $2.tmp)
    $(ECHO) $$($2_PACKAGE_BDIR)*.class | $(GREP) v *.class | $(TR) ' ' 'n' >
$$($2_PACKAGE_BDIR)_the.package.prev
    $(RM) $$($2_PACKAGE_BDIR)*.class $$($2_PACKAGE_BDIR)*.notify $$($2_PACKAGE_BDIR)*.deleted
    $(ECHO) Compiling `$(WC) $2.tmp | $(TR) s ' ' | $(CUT) f 2 d ' '` files in package $(patsubst $4/%/,%,
$(dir $2.tmp))
    $9 $$($2_REMOTE) $$($2_DEPS) $$($2_PUBAPI) $$($2_NATIVEAPI) $(10) implicit:none sourcepath
"$$($2_SRCROOTSC)" d $4 @$2.tmp
    $(ECHO) $$($2_PACKAGE_BDIR)*.class | $(GREP) v *.class | $(TR) ' ' 'n' >
$$($2_PACKAGE_BDIR)_the.package.now
    ($(GREP) xvf $$($2_PACKAGE_BDIR)_the.package.now $$($2_PACKAGE_BDIR)_the.package.prev >
$$($2_PACKAGE_BDIR)_the.package.deleted;true)
    $(ECHO) $1_CLASSES += `$(CAT) $$($2_PACKAGE_BDIR)_the.package.now` |
$(SED) 's/$$$$/$$$$$$$$/g' > $$($2_PACKAGE_BDIR)_the.package.d
    $(ECHO) $1_JAVAS += $$($2_PACKAGE_SRCS) >> $$($2_PACKAGE_BDIR)_the.package.d
    $(ECHO) $2_NOTIFIED:=true > $$($2_PACKAGE_BDIR)_the.package.notify
    $$($2_APPEND_DEPS)
    $$($2_COPY_FILES)
    $(MV) f $2.tmp $2

A better way, move the makefile logic into a smart
javac wrapper (sjavac) written in Java.
 sjavac -src sources -d classes
 Compiles all sources below sources. (Expects standard package file
structure)
 Creates a classes/javac_state file that contains the necessary
information for incremental compiles.
(timestamps, artifacts, dependencies and public apis)
 Forces -implicit:none
 Supports multiple cores.
 javac must provide hooks for this wrapper to work.

A much better part of a makefile (plain javac)
(common/makefiles/JavaCompilation.gmk)

$$($1_BIN)/_the.batch: $$($1_SRCS) $$($1_DEPENDS)
                $(MKDIR) p $$(@D)
                $$(call ListPathsSafely,$1_SRCS,n, >> $$($1_BIN)/_the.batch.tmp)
                $(ECHO) Compiling $1
                $$($1_JVM) $$($1_JAVAC)
                        @$$($1_BIN)/_the.batch.tmp
                        $$($1_FLAGS)
                        d $$($1_BIN) $$($1_HEADER_ARGS)
Apropos the ListPathsSafely macro.... my birthday present wish is a new function in
GNU make:
$(write_to_file the.batch.tmp $(SOURCES))
Such a function can be used to solve any command line length limitation!

A much better part of a makefile (sjavac)
(common/makefiles/JavaCompilation.gmk)

$$($1_BIN)/javac_state: $$($1_SRCS) $$($1_DEPENDS)
                $(MKDIR) p $$(@D)
                $$(call ListPathsSafely,$1_SRCS,n, >> $$($1_BIN)/_the.batch.tmp)
                $(ECHO) Compiling $1
                $$($1_JVM) $$($1_SJAVAC)
                        $$($1_REMOTE) $$($1_SJAVAC_ARGS) permitunidentifiedartifacts
                        mfl $$($1_BIN)/_the.batch.tmp
                        $$($1_FLAGS)
                        d $$($1_BIN) $$($1_HEADERS_ARG)
Apropos the ListPathsSafely macro.... my birthday present wish is a new function in
GNU make:
$(write_to_file the.batch.tmp $(SOURCES))
Such a function can be used to solve any command line length limitation!

Your future makefile
SJAVAC:=java jar javac.jar com.sun.tools.sjavac.Main
SOURCES:=$(shell find src name ”*.java”)
bin/javac_state : $(SOURCES)
$(SJAVAC) src src h jniheaders d bin
 Very fast source file timestamp scan, courtesy make
 Proper incremental build, including removal of class files.
 Proper generation and removal of C-header files for JNI
 Multi core support
 Oops, jar is not incremental....yet.

Your future ant build.xml
just replace the task javac with sjavac
 Not so fast source file timestamp scan, but ok.

Your future ant build.xml
R src src d bin
P com.sun.java_cup.internal.runtime
S src/com/sun/java_cup/internal/runtime/Scanner.java
1347955522000
D java.lang
D java.util
I TYPE com.sun.java_cup.internal.runtime.Scanner
I   METHOD public abstract
com.sun.java_cup.internal.runtime.Symbol next_token(
) throws java.lang.Exception
I   VAR public int sym
A bin/com/sun/java_cup/internal/runtime/Scanner.class
1349217684000

Why is it hard to make javac multi core?

One way to do it.....

Optimistic splitting compiling
 3 cores give a 20% speedup.
 Object.java is always compiled thrice....
Obvious improvements:
 Share already compiled classes
 Share loaded classes from classpath
 Do emit on all cores

Optimistic splitting compiling
 Need a new warning -Xlint:auxiliaryclass
Catches references to classes hidden inside sourcefiles
in such a way that they cannot be found on the
through the -sourcepath. Since this breaks multi core
compilation (and is generally a nuisance).

Lets investigate the new build system for the OpenJDK
 Which was in fact the cause for developing sjavac

Status quo april 2011
Too speed up the OpenJDK build we faced a formidable problem!
 Javac quirks and limitations
 The Java language tendency for circular dependencies
 Undocumented side effects of compiles
 No working dependency propagation Java --> C++ --> Java
 Lack of proper dependency propagation makes it hard to
parallelize the build!
 Of course all the above are relevant for incremental builds too!

The OpenJDK is a complex product
 It has a Java compiler
 It has a JVM containing C++ code and Java code.
 It has the JDK, containing C/C++ code and Java code.
 It has complex intra-dependencies:
C++ code depends on C-headers generated from Java code.
Java code generated from C-headers from the OS.
Generated Java/C and C++ code from several different sources.
The new Java code sometimes needs the new Java compiler
The new Java compiler sometimes needs the new JDK

The Old OpenJDK build system was troublesome
 Controlled through environment variables and command line
variables
 No working dependency tracking
 Slow compilation because of inefficient usage of build computer
 Multiple almost identical copies of the same Makefiles
 The source code has platform specific files in share/classes
and vice versa
 A huge technical debt

The Old OpenJDK build system was troublesome
 Each and every developer created their own build setup script.
 This knowledge was shared through a human peer to peer network.
 We encoded this knowledge into a configure script.
 The configure api is a well known standard.

How to configure and build using the new build system.
hg clone http://hg.openjdk.java.net/buildinfra/jdk8
cd jdk8
sh get_source.sh
cd common/makefiles
bash ../autoconf/configure –enablesjavac
make
...........a lot of output........
cd ../..
./build/linuxx86normalserverrelease/jdk/bin/java version
make images
ls ./build/linuxx86normalserverrelease/images/j2sdkimage
make install

Contents of the generated spec.gmk file.
SRC_ROOT:=/work/jdk8
...
OUTPUT_ROOT:=/work/jdk8/build/linuxx86normalserverrelease
...
BOOT_JDK:=/usr/lib/jvm/java7openjdki386
...
CC:=CCACHE_COMPRESS=1 CCACHE_SLOPPINESS=time_macros
/usr/local/bin/ccache /usr/bin/cc
...
USE_EXTERNAL_LIBJPEG:=true
USE_EXTERNAL_LIBGIF:=false
USE_EXTERNAL_LIBZ:=false
...
SYS_ROOT:=/

The examples of the configure api.
configure help
withbootjdk=/../jdk7
enablesjavac
disablestaticlinkstdc++
host=i686unknownlinuxgnu
enabledebug
withjvmvariants=client,server

The first build of the OpenJDK with the new
build system and multi core sjavac
....
...
.
Compiling 2985 files in 181 packages (com.oracle.net to java.util.concurrent)
Compiling 2973 files in 168 packages (java.util.concurrent.atomic to sun.nio.ch.sctp)
Compiling 3022 files in 559 packages (sun.nio.cs to sun.util.xml)
....
...
.
Use “make VERBOSE=” to see all command lines.

The actual command line to compile the JDK!
sjavac –server:portfile=javacservers/server.port,poolsize=8,
id=GENERATE_JDKBYTECODE,javac=sjavac
x com.sun.pept.* x com.sun.tools.example.trace.*
xf *SolarisFileStore.java xf *SolarisFileSystem.java
... many more filter rules ...............
src jdk/src/share/classes:jdk/src/solaris/classes:gensrc
permitunidentifiedartifacts
mfl ...._the.batch.tmp
bootclasspath jdk/classes
source 7 target 7 encoding ascii
XDignore.symbol.file=true
Xlint:all,....,auxiliaryclasses
implicit:none
d jdk/classes
h jdk/gensrc_headers

An incremental build of the OpenJDK
touch ../../jdk/src/share/classes/com/sun/tools/jdi/SDE.java
make
########################################################################
########################################################################
##### Entering jdk for target(s) all                               #####
########################################################################
Compiling BUILD_JDK
Compiling com.sun.tools.jdi(80)
Note:
/home/fohrstro/jdk8/jdk/src/share/classes/com/sun/tools/jdi/EventRequestMa
nagerImpl.java uses unchecked or unsafe operations.
Note: Recompile with Xlint:unchecked for details.
########################################################################
##### Leaving jdk for target(s) all                                #####
########################################################################
##### Build time 00:00:08 jdk for target(s) all                    #####
########################################################################

An incremental build of the OpenJDK
emacs nw ../../jdk/src/share/classes/javax/net/SocketFactory.java
make
...
..
.
Compiling javax.net(2)
Package javax.net pubapi has changed!
Compiling com.sun.jndi.ldap(46)
Compiling javax.net.ssl(37) javax.rmi.ssl(2)
...
..
.

Old
New +sjavac
no ccache
New no sjavac
primed ccache
New +sjavac
empty ccache
New +sjavac
primed ccache
Langtools 37s 30s 29s 31s 31s
Corba 1m32s 7s 16s 7s 7s
Jaxp 18s 7s 15s 7s 7s
Jaxws 23s 9s 19s 9s 9s
Hotspot 3m01s 3m05s 1m10s 5m18s 1m7s
Jdk 9m52s 1m23s 2m9s 1m25s 50s
Total 15m43s 5m21s 4m38s 7m37s 2m51s
OpenJDK Build times: GNU/Linux 64bit 8cores 16GB
 sjavac doubles the compilation speed! This is due to the server and multi core
support. There are several opportunities for improving the multi core support!
We have only skimmed the surface so far.

Summary
 Initial support for multi core javac, more to come.
 Incremental Java builds from the command line with proper
dependency tracking.
 The new javac -h option to generate C-headers for JNI
 A new lint warning -Xlint:auxiliaryclass
 A much more inviting development environment for every new
developer who wants to contribute to the OpenJDK!

TODO
 Push into tl/jdk8/langtools
now it is inside build-infra/jdk8/langtools
 Incremental jar updates
 Write sjavac task for ant
 Add public api tracking for classes and jars

Building Large Java Projects Faster: Multicore javac and Makefile integration

More Related Content

What's hot (20)

Similar to Building Large Java Projects Faster: Multicore javac and Makefile integration (20)

Recently uploaded (20)

Building Large Java Projects Faster: Multicore javac and Makefile integration