Introduction

One of the cool features available in Java 6.0 (Mustang) is the ‘Java Compiler API’. This API is a result of the JSR (Java Specification Request) 199 which proposes that there must be a standard way to compile java source files. The result of the JSR is the new ‘Java Compiler API’ and one can use this new feature to compile java source files from within java files. Previously developers were depending on the low-level issues like starting a process representing the javac.exe. Though this feature is not intended to every one, Editors or IDE (Integrated Development Environment) can make much use of this new feature for compiling Java source files in a better manner.

Compiler API

All the API (the client interfaces and the classes) needed by the developers for playing with the Java compiler API is available in the new javax.tools package. Not only this package represents classes and methods for invoking a java compiler, but it provides a common interface for representing any kind of Tool. A tool is generally a command line program (like javac.exe, javadoc.exe or javah.exe).

also read:

• New Features in Java 6.0 – Part 1
• New Features in Java 6.0 – Part 2
• New Features in JDBC API (Java 6.0)
• What is New in Java 8.0

Instead of looking into all the classes and interfaces that are available in the javax.tools package, it makes more sense to go through some sample programs, then examining what the classes and the methods are doing.

Compiling a java source file from another Java source

Following is the small sample program that will demonstrate how to compile a Java source file from another java file on the fly.

[All the examples given here are written and tested with Mustang build 1.6.0-b105, and it seems that more API changes and restructuring of classes and methods are occurring in the newer builds].

[code lang=”java”]MyClass.java:

package test;
public class MyClass {
public void myMethod(){
System.out.println("My Method Called");
}
}

Listing for SimpleCompileTest.java that compiles the MyClass.java file.

SimpleCompileTest.java:

package test;
import javax.tools.*;
public class SimpleCompileTest {
public static void main(String[] args) {
String fileToCompile = "test" + java.io.File.separator +"MyClass.java";
JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
int compilationResult = compiler.run(null, null, null, fileToCompile);
if(compilationResult == 0){
System.out.println("Compilation is successful");
}else{
System.out.println("Compilation Failed");
}
}
}[/code]

The entry point for getting a compiler instance is to depend on the ToolProvider class. This class provides methods for locating a Tool object. (Remember a Tool can be anything like javac, javadoc, rmic, javah etc…) Though the only Tool available in Mustang build (1.6.0-b105) is the JavaCompiler as of now, it is expected that many more tools are to be added in the future.

The getSystemJavaCompiler() method in the ToolProvider class returns an object of some class that implements JavaCompiler (JavaCompiler is an interface that extends Tool interface and not a class). To be more specific, the method returns a JavaCompiler implementation that is shipped along with the Mustang Distribution. The implementation of the Java Compiler is available in the tools.jar file (which is usually available in the <JDK60_INSTALLATION_DIR>\lib\tools.jar).

After getting an instance of the JavaCompiler, compilation on a set of files (also known as compilation units) can be done by invoking the run(InputStream inputStream, OutputStream outputStream, OutputStream errorStream, String … arguments) method. To use the defaults, null can be passed for the first three parameters (which correspond to System.in, System.out and System.err), the fourth parameter which cannot be null is a variable argument that refers to the command-line arguments we usually pass to the javac compiler.

The file that we are going to compile is MyClass.java which is the same test package as of the SimpleCompileTest. The complete file name (along with the directory ‘test’) is passed as 4th argument to the run method. If there are no errors in the source file (MyClass.java, in this case), the method will return 0 which means that the source file was compiled successfully.

After compiling and running the SimpleCompileTest.java, one can see the following output message in the console.

‘Compilation is successful’

Let us modify the source code by introducing a small error by removing the semi-colon after the end of the println() statement and see what happens to the output.

[code lang=”java”]MyClass.java:
package test;

public class MyClass {
public void myMethod(){
System.out.println("My Method Called")
// Semi-colon removed here purposefully.
}
}
Now, running the SimpleCompileTest.java leads to the following output,

test\MyClass.java:5: ‘;’ expected
System.out.println("My Method Called")
^
1 error
Compilation Failed[/code]

This is the error message one normally sees when compiling a java file using javac.exe in the command prompt. The above represents the error message(s) and since we have made the error output stream point to null (which defaults to System.err, which is the console) we are getting the output error messages in the console. If instead we have pointed the errorStream to something like this,

[code lang=”java”]FileOutputStream errorStream = new FileOutputStream("Errors.txt");
int compilationResult = compiler.run(null, null, errorStream, fileToCompile);[/code]

a new file called Errors.txt will be created in the current directory and the file would be populated with the error messages that we saw before.

Compiling multiple files

One might be tempted to think that the following code will work for compiling multiple java files (assuming that the two files that are to be compiled are One.java and Two.java).

[code lang=”java”]
String filesToCompile = new String("One.java Two.java") ;[/code]

But surprisingly, when you try this, you will get a ‘Compilation Failed’ error message in the console.

The answer is JavaCompiler needs to extract individual options and arguments and these options and arguments should not be separated by spaces, but should be given as individual strings.

So, this won’t work at all.

[code lang=”java”]compiler.run(null, null, null, “One.java Two.java”);[/code]

But, the below code will work nicely.

[code lang=”java”]compiler.run(null, null, null, “One.java”, “Two.java”);[/code]

One reason for forcing this kind of restriction is that sometimes the complete java file names (which includes directory path as well) itself can have white-spaces , in such a case it would be difficult for the parser to parse all the tokens correctly into options and arguments.

Following is a sample code that compiles multiple java files.

[code lang=”java”]MyClass.java:
package test;

public class MyClass {
}

MyAnotherClass.java:
package test;

public class MyAnotherClass {
}

MultipleFilesCompileTest.java:
package test;
import javax.tools.*;

public class MultipleFilesCompileTest {
public static void main(String[] args) throws Exception{
String file1ToCompile = "test" + java.io.File.separator + "MyClass.java";
String file2ToCompile = "test" + java.io.File.separator + "MyAnotherClass.java";
JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
int compilationResult = compiler.run(null, null, null, file1ToCompile, file2ToCompile);
if(compilationResult == 0){
System.out.println("Compilation is successful");
}else{
System.out.println("Compilation Failed");
}
}
}[/code]

The above program compiles fine with the output message ‘Compilation is successful’.

Do remember, that the final argument is a variable argument and it can accept any number of arguments.

[Starting with Java 5.0, variable argument is a new feature where the callers (the calling method) can pass any number of arguments. To illustrate this concept, look at the sample code.

[code lang=”java”]public int addNumbers(int …numbers){

int total = 0;
For(int temp : numbers){
Total = total + temp;
}
return total;
}[/code]

A variable argument is represented by ellipsis (…) preceding the variable name like this
int … numbers. And , one can call the above method in different styles, like the below

[code lang=”java”]addNumbers(10, 10, 30,40); // This will work.
addNumbers(10,10) // This also will work.[/code]

type must be the last one. Variable arguments are internally treated as arrays. So, this is also possible now.

[code lang=”java”]addNumbers(new int[]{10, 34, 54});[/code]

So, great care should be exercised when passing multiple options along with values to the run method. As an example, the ideal way to pass options along with its option values might me,

[code lang=”java”]compiler.run(null, null, null, ”-classpath”, ”PathToClasses”, ”-sourcepath”, ”PathToSources”, ”One.java”, ”Two.java”);[/code]

Ass one can see, even the option and its option values must be treated as a separate string.

As you are aware of the fact, when invoking the java compiler with the ‘verbose’ option specified, the javac will output messages that will occur during the compilation life-cycle (like parsing the input, validating them, scanning for the paths for both source and class files, loading all the necessary class files, then finally creating the class files in the specified destination directory). Let us achieve the same effect through the following sample code.

[code lang=”java”]SimpleCompileTestWithVerboseOption.java
package test;
import java.io.FileOutputStream;
import javax.tools.*;
public class SimpleCompileTestWithVerboseOption {
public static void main(String[] args) throws Exception{
String fileToCompile = "test" + java.io.File.separator + "MyClass.java";
JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
FileOutputStream errorStream = new FileOutputStream("Errors.txt");
int compilationResult = compiler.run(
null, null, errorStream, "-verbose", fileToCompile);
if(compilationResult == 0){
System.out.println("Compilation is successful");
}else{
System.out.println("Compilation Failed");
}
}
}[/code]

May be a bug in Mustang

One might see that the errorStream (3rd argument) has been pointed out to a file to collect the output messages instead of the outputStream (2nd argument).

The following code was tried for capturing the verbose output, but this code failed, in the sense, the output was still written to the standard console, though the code seems to re-direct the output to a file.

[code lang=”java”]FileOutputStream outputStream = new FileOutputStream("Output.txt");
int compilationResult = compiler.run(
null, outputStream, null, "-verbose", fileToCompile);[/code]

The Java Compiler API is treating the output messages (in this case, the output messages obtained by specifying the ‘verbose’ option) as error messages, and so even though the output is pointing to the file (‘Output.txt’), it is spitting all the output messages to the console.

Also, the documentation for the run method is unclear; it tells that any diagnostics (errors, warnings or information) may be written either to the output stream or the error messages.

Advanced Compilation

In the above section, we saw how to compile files using the JavaCompiler tool. For advanced compilation related stuffs, the JavaCompiler depends on two more services namely the file manager services and the diagnostics services. These services are provided by the JavaFileManager and the DiagnosticListener classes respectively.

JavaFileManager

The JavaFileManager (being given implementation as StandardJavaFileManager) manages all the file objects that are usually associated with tools. This JavaFileManager is not only to JavaCompiler, but instead it can work with any kinds of objects that conform to the standard Tool interface. To understand why JavaFileManager is so important, let us understand what could be the things that may happen during compilation process.

[code lang=”java”]javac MyClass.java[/code]

When we issue this command in the command prompt, so many things will happen. The very first thing is that the compiler will parse all the options that are specified by the user, have to validate them, and them have to scan the source and class path for java source files and the jar files. It then has to deal with the input files (in this case it is MyClass.java) and output files (MyClass.class).

So, JavaFileManager which is associated with any kind of tool (normally all tools have some kind of input and the output files for processing), deals with managing all the input files and output files. By managing, we mean that JavaFileManager is responsible for creating output files, scanning for the input files, caching them for better performance. One such implementation given to the JavaFileManager is the StandardJavaFileManager.

A file being managed by JavaFileManager doesn’t mean the file is necessarily a file in the hard-disk. The contents of the file managed may come from a physical file in a hard-disk, may come from in-memory or can even come from a remote socket. That’s why JavaFileManager doesn’t deal with java.io.File objects (which usually refer to the physical files in the operating system File System). Rather, JavaFileManager manages all the files and their contents in the form of FileObject and JavaFileObject which represents the abstract representation for any kind of files by managed by the JavaFileManager.

FileObject and JavaFileObject refer to the abstract file representation that are being managed by the JavaFileManager and also have support related to reading and writing the contents to the right destination. The only difference between a FileObject and a JavaFileObject is that a FileObject can represent any kind of FileObject (like text file, properties file, image file etc), whereas a JavaFileObject can represent only java source file (.java) or a class file (.class). If a FileObject represents a java source file or a class file, then implementations should take care to return a JavaFileObject instead.

Diagnostics

The second dependent service by the JavaCompiler is the Diagnostic Service. Diagnostic usually refers to errors, warnings or informative messages that may appear in a program. For getting diagnostics messages during the compilation process, we can attach a listener to the compiler object. The listener is a DiagnosticListener object and its report() method will be called with a diagnostic method which contains many a more information like the kind of diagnostics (error, warning, information or other), the source for this diagnostics, the line number in the source code, a descriptive message etc.

CompilationTask

Before going into the sample code to clarify things like JavaFileManager, Diagnostic,and the DiagnosticListener classes, let us have a quick review on the class CompilationTask. From its name, obviously one can tell that it represents an object that encapsulates the actual compilation operation. We can initiate the compilation operation by calling the call() method in the CompilationTask object.

But how to get the CompilationTask object?

Since, CompilationTask is closely associated to a JavaCompiler object, one can easily say JavaCompiler.getCompilationTask( arguments ) to get the CompilationTask object. Let we now see the parameters that are needed to pass to the getCompilationTask(…) method.

Almost all the arguments can be null (with their defaults, which is discussed later), but the final argument represents the list of java objects to be compiled (or the compilation units) which cannot be null. The last argument is Iterable<? Extends JavaFileObject> javaObjects.

So, how can one populate this argument?

[Iterable is a new interface that was added with jdk 5.0 which is use to iterate (or traverse over a collection of objects. It has one method called iterator() which returns an Iterator object, and using it one can traverse over the collection by using the combinational hasNext() and the next() methods.

Considering type safety which started from Java 5.0, it is the role of the developer to specify what exactly is the type of object to be iterated. Iterable<? Extends JavaFileObject> means that this argument is an iterable that is acting on any class that implement the JavaFileObject interface].

JavaFileManager has 4 convenience methods like getJavaFileObject(File … files) , getJavaFileObjects(String … filenames), getJavaFileObjectsFromFiles(Iterable(? extends JavaFileObject) and getJavaFileObjectFromString(Iterable<? extends String> filenames) that returns javaObjects in the form of Iterable<? Extends JavaFileObject> type.

So, we can construct the 4th arguments using any of the convenience methods.

With more bits of theory in the last few sections, let us see a sample program that incorporates all the classes and the methods that we saw listed.

Purposefully we create a java file called MoreErrors.java that has some error code in it. The listing for MoreErrors.java is shown below.

[code lang=”java”]MoreErrors.java:
package test;
public class MoreErrors {
public void errorOne ()
// No open braces here. Line a
}
public void errorTwo(){
System.out.println("No semicolon") // Line b
// No semicolon at the end of the statement.
}
public void errorThree(){
System.out.prntln("No method name called prntln()"); // Line c
}
}[/code]

As one can see, the statement above line a, line b and line c has errors.

Let us look into the AdvancedCompilationTest.java that uses all the classes and the methods that we discussed above.

[code lang=”java”]AdvancedCompilationTest.java:
package test;
import java.io.*;
import java.util.*;
import javax.tools.*;
import javax.tools.JavaCompiler.*;
public class AdvancedCompilationTest {
public static void main(String[] args) throws Exception {
JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
// Line 1.
MyDiagnosticListener listener = new MyDiagnosticListener(); // Line 2.
StandardJavaFileManager fileManager =
compiler.getStandardFileManager(listener, null, null); // Line 3.
String fileToCompile = "test" + File.separator + "ManyErrors.java";
// Line 4
Iterable fileObjects = fileManager.getJavaFileObjectsFromStrings(
Arrays.asList(fileToCompile)); // Line 5
CompilationTask task = compiler.getTask(null, fileManager, listener, null, null, fileObjects); // Line 6
Boolean result = task.call(); // Line 7
if(result == true){
System.out.println("Compilation has succeeded");
}
}
}
class MyDiagnosticListener implements DiagnosticListener{
public void report(Diagnostic diagnostic) {
System.out.println("Code->" + diagnostic.getCode());
System.out.println("Column Number->" + diagnostic.getColumnNumber());
System.out.println("End Position->" + diagnostic.getEndPosition());
System.out.println("Kind->" + diagnostic.getKind());
System.out.println("Line Number->" + diagnostic.getLineNumber());
System.out.println("Message->"+ diagnostic.getMessage(Locale.ENGLISH));
System.out.println("Position->" + diagnostic.getPosition());
System.out.println("Source" + diagnostic.getSource());
System.out.println("Start Position->" + diagnostic.getStartPosition());
System.out.println("\n");
}
}[/code]

Let us explore the above code in greater detail.

Line 1 is essentially creating an object of type JavaCompiler using the ToolProvider class. This is the entry point.

[code lang=”java”]JavaCompiler compiler = ToolProvider.getSystemJavaCompiler()[/code]

Line 2 and Line 3 is making the compiler to make use of the FileManager and the Diagnostic’s Services. To rewind the theory a bit, JavaFileManager object is used to manage the input and the output files that a tool normally deals with. And Diagnostic’s objects refer the diagnostics (errors, warnings or information) that may occur during the compilation process within a program.

[code lang=”java”]MyDiagnosticListener listener = new MyDiagnosticListener();[/code]

Line 2 creates an object of Type DiagnosticListener, since we want to monitor the diagnostics that may happen during the process of compilation (diagnostics, in the form of error messages will always happen in our case, as we have purposefully done some errors in the java code). We have overridden the report(Diagnostic) method and have extracted all the possible information. Since diagnostics can happen in any kind of file object, how can we specifically tell that this Diagnostics is for a java file object?

The answer has become easy because of Java 5.0 generics. One can notice that MyDiagnosticListener is a typed class (having some typed parameter) meaning that it can act on any kind of object that has diagnostics properties; here we are explicitly telling that the diagnostics is for JavaFileObject and not for any other file object by mentioning the JavaFileObject in the class declaration and the method declaration (shown in bold).

[code lang=”java”]class MyDiagnosticListener implements DiagnosticListener{
public void report(Diagnostic diagnostic) {
System.out.println("Code->" + diagnostic.getCode());
System.out.println("Column Number->" + diagnostic.getColumnNumber());
….
….
}
}[/code]

In Line 3, we are associating the Diagnostics listener object to the compiler object through the standard java file manager by making this method call.

[code lang=”java”]StandardJavaFileManager fileManager =
compiler.getStandardFileManager(listener, null, null);[/code]

This method tells to attach the diagnostics listener object to this compiler object, so whenever this compiler object executes a compilation operation, and if any diagnostics related errors or warnings have occurred in a program, then the diagnostics being encapsulated by the Diagnostic object will be passed back to the report() method of the DiagnosticListener interface.

The last 2 arguments refer the locale and charset arguments for formatting the diagnostic messages in a particular locale and by using the specified charset which can be null.

Line 4 and Line 5 populates the file objects to be compiled by using the convenience objects in the StandardJavaFileManager class.

[code lang=”java”]String fileToCompile = "test" + File.separator + "ManyErrors.java";
Iterable fileObjects = fileManager.getJavaFileObjectsFromStrings(
Arrays.asList(fileToCompile));[/code]

Line 6 gets an instance of the CompilationTask object by calling the getCompilationTask() method and by passing the fileManager, listener and the fileObjects objects. The null arguments refer to the Writer object (for getting the output from the compiler), list of options (the options that we pass to javac like –classpath classes, -sourcepath Sources…) and classes (for processing the custom annotations that are found in the source code).

Finally, the actual compilation operation is done by calling the call() method which returns true if all the files (compilationUnits) succeed compilation. If any of the files have errors in it , then the call() method will return false. Anyway, in our case, we have some error code in the MoreErrors.java file, so the report method will be called printing all the diagnostics information.

DiagnosticCollector

In the previous program, we saw that we have a customized class called MyDiagnosticListener. Its sole purpose it to collect and print all the diagnostic messages to the console. This class can be completely eliminated since Mustang already has a class called DiagnosticCollection<SourceObject> that does the same thing. It has a method called getDiagnostics() which returns a list , through which we can iterate and can output the diagnostic messages to the console.

The following code achieves the same using DiagosticCollector class.

[code lang=”java”]AdvancedCompilationTest2.java:
package test;
import java.io.*;
import java.util.*;
import javax.tools.*;
import javax.tools.JavaCompiler.*;
public class AdvancedCompilationTest2 {
public static void main(String[] args) throws Exception {
JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); // Line 1.
DiagnosticCollector diagnosticsCollector =
new DiagnosticCollector();
StandardJavaFileManager fileManager =
compiler.getStandardFileManager(diagnosticsCollector, null, null); // Line 3.
String fileToCompile = "test" + File.separator + "ManyErrors.java"; // Line 4
Iterable fileObjects = fileManager.getJavaFileObjectsFromStrings(Arrays.asList(fileToCompile)); // Line 5
CompilationTask task = compiler.getTask(null, fileManager, diagnosticsCollector, null, null, fileObjects); // Line 6
Boolean result = task.call(); // Line 7
List> diagnostics = diagnosticsCollector.getDiagnostics();
for(Diagnostic d : diagnostics){
// Print all the information here.
}
if(result == true){
System.out.println("Compilation has succeeded");
}else{
System.out.println("Compilation fails.");
}
}
}[/code]

Compilation of Java Source from a String object

Having discussed about the various ways of compiling java file sources, it’s now time to look at how to compile a java source that is encapsulated in a string object. As previously mentioned, it is not mandatory that the contents of a java source must reside in hard-disk, it can even reside in memory. By saying that compiling a java source from a string object, we are implicitly saying that the java source is residing in memory, more specifically, the contents are residing in RAM.

For this to happen, we have to encapsulate a class the represents the java source from a string. We can extend this class from the SimpleJavaFileObject (a convenient class that overrides all the methods in the JavaFileObject with some default implementation). The only method to override is the getCharContent() that will be called internally by the Java compiler to get the java source contents.

[code lang=”java”]JavaObjectFromString.java:
package test;
import java.net.URI;
class JavaObjectFromString extends SimpleJavaFileObject{
private String contents = null;
public JavaObjectFromString(String className, String contents) throws Exception{
super(new URI(className), Kind.SOURCE);
this.contents = contents;
}
public CharSequence getCharContent(boolean ignoreEncodingErrors) throws IOException {
return contents;
}
}[/code]

Since the SimpleJavaFileObject has a protected two argument constructor that accepts an URI (the URI representation of the file object) and a Kind object (a special type that tells what kind is this object, the Kind may be a Kind.SOURCE, Kind.CLASS, Kind.HTML, Kind.OTHER, in our case, it is Kind.SOURCE, since we are inferring a Java source object), we have defined a two argument constructor in JavaObjectFromString class and delegates the control back the base class. The getCharContent() method has to be overridden (since this is the method that will be called by the JavaCompiler to get the actual java source contents) to return the string (remember, String implements CharSequence) that represents the entire java source (which was previously saved in the constructor).

The code that uses this JavaObjectFromString object looks like this.

[code lang=”java”]AdvancedCompilationTest3.java:
package test;
import java.io.*;
import java.util.*;
import javax.tools.*;
import javax.tools.JavaCompiler.*;
public class AdvancedCompilationTest3 {
public static void main(String[] args) throws Exception {
JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
DiagnosticCollector diagnosticsCollector =
new DiagnosticCollector();
StandardJavaFileManager fileManager =
compiler.getStandardFileManager(diagnosticsCollector, null, null);
JavaFileObject javaObjectFromString = getJavaFileContentsAsString();
Iterable fileObjects = Arrays.asList(javaObjectFromString);
CompilationTask task = compiler.getTask(null, fileManager, diagnosticsCollector, null, null, fileObjects);
Boolean result = task.call();
List> diagnostics = diagnosticsCollector.getDiagnostics();
for(Diagnostic d : diagnostics){
// Print all the information here.
}
if(result == true){
System.out.println("Compilation has succeeded");
}else{
System.out.println("Compilation fails.");
}
}
static SimpleJavaFileObject getJavaFileContentsAsString(){
StringBuilder javaFileContents = new StringBuilder("" +
"class TestClass{" +
" public void testMethod(){" +
" System.out.println(" + "\"test\"" + ");" +
"}" +
"}");
JavaObjectFromString javaFileObject = null;
try{
javaFileObject = new JavaObjectFromString("TestClass", javaFileContents.toString());
}catch(Exception exception){
exception.printStackTrace();
}
return javaFileObject;
}
}[/code]

Conclusion

Before the release of Mustang, the compiler API related interfaces and classes were maintained in some non-standard packages (i.e inside com.sun.javac.tools.javac). But with Java 6.0, the designers of java have given great heights to compilation API by generalizing them in the javax.tools package. As already mentioned, this API is not for everyone. Web and Application servers can depend on this API exhaustively to provide compilation activities of the dynamically created Java source files. Although it is heard that more and more bugs related to Compiler API are there, they are expected to be fixed soon in the future builds.