Category Archives: Compiler Construction

Java 8 Nashorn Script Engine

For a side project I am developing in Java I needed a good JavaScript parser but the publicly documented Nashorn interface is all about compiling when I only needed an intermediate representation. It is currently possible as of JDK8u40 to use the parser to get the AST as a JSON encoded string either from a JS file being executed by the ScriptEngine or from within Java using the non-public Nashorn API.

The below image should convey to you how a simple assignment statement is broken into a stream of tokens then an AST is generated as JSON on the right. This is why symbols like * + – are called binary operators because they take two operands, ! is the logical negation and an unary operator becaues it takes only one operand. The operands can also be expressions because expressions are defined recursively in terms of expressions which can be literals, terms, or other expressions. This is how we end up with tree’s which when coupled with additional semantic information such as keywords, types, and identifiers help us do code generation. This enables you to take in one language, say GML, and spit out a completely different one like C++ which, if you don’t already know, is exactly what ENIGMA’s compiler does.

Abstract Syntax Tree

An abstract syntax tree is produced by a parser after the lexer phase breaks code into a stream of tokens.

Wikipedia has additional information on abstract syntax trees if you would like to know more.
https://en.wikipedia.org/wiki/Abstract_syntax_tree
The following StackOverflow post provides clarification between an AST and a parse tree.
http://stackoverflow.com/questions/5026517/whats-the-difference-between-parse-tree-and-ast

This example shows you how to get the AST as JSON from Java. This was my own discovery from studying the Nashorn source code.

String code = "function a() { var b = 5; } function c() { }";

Options options = new Options("nashorn");
options.set("anon.functions", true);
options.set("parse.only", true);
options.set("scripting", true);

ErrorManager errors = new ErrorManager();
Context contextm = new Context(options, errors, Thread.currentThread().getContextClassLoader());
Context.setGlobal(contextm.createGlobal());
String json = ScriptUtils.parse(code, "<unknown>", false);
System.out.println(json);

This example should give the following JSON encoded AST as I executed it on Java 8u51. This JSON encoding provided by Nashorn is compliant with the community standard JavaScript JSON AST model popularized by Mozilla.
https://developer.mozilla.org/en-US/docs/Mozilla/Projects/SpiderMonkey/Parser_API

{“type”:”Program”,”body”:[{“type”:”FunctionDeclaration”,”id”:{“type”:”Identifier”,”name”:”a”},”params”:[],”defaults”:[],”rest”:null,”body”:{“type”:”BlockStatement”,”body”:[{“type”:”VariableDeclaration”,”declarations”:[{“type”:”VariableDeclarator”,”id”:{“type”:”Identifier”,”name”:”b”},”init”:{“type”:”Literal”,”value”:5}}]}]},”generator”:false,”expression”:false},{“type”:”FunctionDeclaration”,”id”:{“type”:”Identifier”,”name”:”c”},”params”:[],”defaults”:[],”rest”:null,”body”:{“type”:”BlockStatement”,”body”:[]},”generator”:false,”expression”:false}]}

It is important to note however that this interface may change because it’s not well documented and is new to the JSE. Additionally the OpenJDK project is developing a public interface for Java 9 that allows AST traversal in a more standard and user friendly way.
http://openjdk.java.net/jeps/236

Limited documentation for the existing public Nashorn classes in Java 8 can be found below.
https://docs.oracle.com/javase/8/docs/jdk/api/nashorn/allclasses-noframe.html

The following link provides a list of all of the parser and compiler options that I set above. However it is important to note that the syntax is different when setting the options inside Java where – is replaced with a period.
http://hg.openjdk.java.net/jdk8u/jdk8u-dev/nashorn/file/tip/docs/DEVELOPER_README

The Nashorn source code can be found on GitHub and also on BitBucket. I prefer the BitBucket version as the GitHub version seems to be missing some classes.
https://github.com/uditrugman/openjdk8-nashorn
https://bitbucket.org/adoptopenjdk/jdk8-nashorn/src/096dc407d310?at=default