When a Java program is compiled, it is not translated to executable machine code but rather, the javac compiler yields Java bytecode, which serves as an intermediate format that describes a program to the Java virtual machine. Despite their shared names, the Java virtual machine has no notion of the Java programming language, but exclusively processes bytecode instructions.
One of the original intentions of Java bytecode was to reduce a Java program’s size. As an emerging language in the fledgling days of the World Wide Web, applets for example, would require a minimal download time. Thus, sending single bytes as instructions was preferred to transmitting human-readable words and symbols. But despite that translation, a Java program expressed as bytecode still largely resembles the original source code.
Over time, developers of languages besides Java created compilers to translate those languages into Java bytecode. Today, the list of language-to-Java-bytecode compilers is almost endless and nearly every programming language became executable on the Java virtual machine. Moreover, the recent introduction of the invokedynamic bytecode instruction in Java 7 has increased the capabilities of efficiently running dynamic languages on the JVM.
However, beyond the invokedynamic instruction, Java bytecode remains fairly Java specific. As of this writing, the set of bytecode instructions largely reflects the feature set of the Java programming language. For example, any value in a Java program remains strictly typed, even though a dynamic JVM language might suggest the opposite. In addition, the virtual machine strictly enforces the object-oriented paradigm. Therefore, any function of a JVM language that is not semantically associated with an object, still needs to be represented as a method inside of some class.
Nevertheless, the Java virtual machine grows increasingly polyglot. And despite its Java-centric nature, the virtual machine still supports the composition of bytecode instructions that a Java compiler would reject. After briefly introducing the fundamentals of the bytecode format. this article explores a few capabilities of the JVM that cannot be expressed as Java source code.
Java bytecode fundamentals
Few developers ever work with Java bytecode directly, and at first glance, this virtual machine language might seem overly complicated. But in fact, the bytecode format is quite trivial to understand. As its name suggests, the Java virtual machine represents a computer that does not normally exist as actual hardware but only virtually as a program of its own. Java bytecode serves as the set of legal machine code instructions to this virtual computer.
Every program maintains an internal stack of operands, and any bytecode instruction operates on that operand stack for the currently executing method. To process any values, those values must first be pushed onto the stack. Once on the stack, those values can serve as inputs to an operation. Let’s look at an example: suppose we want to add the numbers one and two. Both values must first get pushed onto the operand stack, by executing the bytecode instructions iconst_1 and iconst_2. With these two numbers on the stack, the iadd instruction can now consume both values by popping them off the top of the stack. As a result, the instruction pushes the sum of the consumed values back onto the stack.
Similarly, a method gets executed by first pushing its arguments onto the operand stack. When executing the method, the Java virtual machine then pops all arguments off the stack to hand them to the invoked method. Non-static methods additionally receive the instance of the invoked method as an implicit first argument. After returning, a non-void method finally pushes back its return value onto the operand stack.
To make sure that no method is invoked with incompatible arguments, the Java virtual machine executes a verifier whenever it loads new bytecode. Among other things, the verifier ensures that the virtual machine never arrives at an inconsistent state where an executed bytecode instruction expects different values than those found on the operand stack. If the verifier cannot guarantee such consistency, it rejects the loaded class and throws a VerifyError. The Java compiler would of course never create such bytecode. Nevertheless, implementers of other language compilers are equally bound by this static consistency check and must not generate bytecode that would fail the verifier’s consistency check.
Partly erased types
However, the verifier does not audit all rules that are imposed by the Java programming language. A famous example for such a rule is the runtime erasure of generic types. When a generic type is translated during compilation, it is reduced to its most general boundary. Of course, this narrows down the capabilities of the verifier when asserting bytecode for interacting with generic types. As these generic types were erased during compilation, the verifier can only assure assignability to the erasure of a generic type. Because this compromises the capability of the JVM to verify loaded code, the Java compiler explicitly warns about potentially unsafe usage of generic types.
Note that generic types are not fully erased but are embedded as meta information into a class file. Several frameworks extract this meta information via the reflection API and change their behavior according to the discovered information. After all, generic types are only hidden from the Java virtual machine and not erased completely.
Unchecking checked exceptions
A lesser-known difference between the JVM and the Java programming language is the treatment of checked exceptions. For a checked exception, the Java compiler normally assures that it is either caught within a method or explicitly declared to be thrown. However, this is only a convention of the Java compiler and not a feature of the Java virtual machine. At runtime, a checked exception can be thrown independently of any declaration.
By abusing the mentioned erasure of generic types, it is even possible to trick the Java compiler into throwing a checked exception. This can be accomplished by casting a checked exception to a runtime exception. To prevent this casting from producing a type error, it is conducted using generic types, which are removed when translating a method to bytecode. The following example demonstrates how to implement such a generic casting:
static void doThrow(Throwable throwable) {
class Unchecker<T extends Throwable> {
@SuppressWarnings("unchecked")
private void uncheck(Throwable throwable) throws T {
throw (T) throwable;
}
}
new Unchecker<RuntimeException>().uncheck(throwable);
}
To throw such an exception in your code, you would just call the static doThrow method, supplying the root Throwable, without having to declare an explicit throws clause. The uncheck method is defined to throw a generic exception T, which the compiler must allow since the T generic parameter, being a subclass of Throwable, might be a RuntimeException.
Since the generic information is erased during compilation, the casting to T does not translate into a bytecode instruction. The Java compiler warns about this unsafe use of generics but here this warning is intentionally ignored. This unsafe operation can now be used to trick the compiler into “casting” any Throwable to a runtime exception, thereby obviating any explicit throws declaration.
At first glance, this might not seem very useful. However unchecking checked exceptions can be an interesting option when dealing with lambda expressions. Most functional interfaces that ship with the Java class library do not declare checked exceptions. Therefore, checked exceptions need to always be caught within a lambda expression, preempting the benefits of concision generally associated with such functional expressions.
This is especially inconvenient if the method using a lambda expression intends to escalate a checked exception to its caller. For achieving such escalation, the checked exception needs to be wrapped inside of an unchecked exception. This wrapper exception needs then to be caught outside of the lambda such that the wrapped exception can be re-thrown. By unchecking the checked exception using the trick from above, it is however possible to largely avoid such boilerplate.
interface ExceptionConsumer<T> extends Consumer<T> {
void doAccept(T t) throws Throwable;
@Override
default void accept(T t) {
try {
doAccept(t);
} catch (Throwable throwable) {
doThrow(throwable);
}
}
}
Given this helper interface that unchecks a checked exception, it is now possible to operate with methods that throw checked exceptions even inside of a lambda expressions.
void doSomething() throws Exception {
Arrays
.asList("foo", "bar")
.forEach((ExceptionConsumer<String>) s -> doSomethingWith(s));
}
void doSomethingWith(String s) throws Exception;
As demonstrated in the above code, unchecking the checked exception allows its propagation to the outer method. On the downside, the Java compiler is no longer able to verify that the lambda expression’s outer method declares the checked exception. Therefore, unchecking checked exceptions needs to be handled with great care.
Return type overloading
The Java programming language does not consider a method’s return type to be part of that method’s signature. Therefore, it is not possible to define two methods with the same name and parameter types within the same class even if they return a value of a different type. The rationale behind this decision is a situation where a method is invoked for its side effect while ignoring its return value. In this case, the resolution of a method call would become ambiguous if a Java method was overloaded by its return type.
In Java bytecode, any method is instead identified by a signature that does include the method’s return type. Therefore, a Java class file can include two methods that only differ by their return types. Consequently, bytecode must refer to a specific return type when invoking a method. For this reason, a Java program needs to be recompiled if the return type of an invoked method changes, even if the Java source code does not require any changes. Without this recompilation, the JVM is not able to link a call site to the changed method because the bytecode instruction no longer references the correct return type.
A hybrid type system
Because methods are referred to by their exact signature, a compiled Java class needs to preserve the types that are defined by the Java programming language. However, when executing a method, the Java virtual machine applies a slightly different type system for primitive types than the Java programming language. In a way, the JVM applies a hybrid type system where it distinguishes types for describing values from types that exist during execution.
When loading a value onto the operand stack, the JVM treats primitive types that are smaller than an integer as if they were integers. Consequently, it makes little difference to a program’s bytecode representation if a method variable is represented by, for example, a short instead of an int. Instead, the Java compiler inserts additional bytecode instructions that crop a value to the allowed range of a short when assigning values. Using shorts over integers in a method’s body can therefore rather result in additional bytecode instructions rather than presumably optimizing a program.
This equalization does not apply however when storing values on the heap, either in the form of a field or as the value of an array. However, boolean values still do not exist even on this level but are rather encoded as the single byte values zero and one representing false and true. This is attributable to the fact that most hardware does not allow explicit access to a single bit, and so boolean values are rather represented as bytes.
Breaking the constructor chain
The Java compiler requires any constructor to invoke another constructor as its first instruction, either implicitly or explicitly. To be valid, this invoked constructor must be declared by the same class or the direct superclass. Within Java bytecode, this restriction is only partially enforced. Instead, the JVM’s verifier asserts that another valid constructor is called eventually. Additionally, it verifies that any method calls and field reads on the constructed instance occur only after this constructor was called. Other than that, it is perfectly legal to execute any code before invoking another constructor. Also, it is even possible to write values to fields of the constructed instance before calling another constructor.
For making this rule less erratic, the JVM’s type system includes a special type named undefined. In addition, the construction of an object is split into two separate bytecode instructions where the first instruction creates the undefined object and the second instruction completes the definition by calling a constructor on it. In bytecode, a constructor is represented as an instance method named <init> which is also displayed in stack traces.
As long as an object is still considered to be undefined, the JVM’s verifier forbids any meaningful interaction with the instance besides writing to its fields. Other than that, to the Java virtual machine it is nothing more than calling an ordinary method. When fully deactivating the JVM’s verifier, the virtual machine is even capable of not invoking a constructor or invoking constructors multiple times on the same instance.
Final-ish field
Another convention of constructors that is strictly enforced by the Java programming language is the one and only one time assignment to final fields. A final field’s value cannot be changed once it is assigned. At the same time, a final field must be defined within a constructor’s body for a Java program to compile.
While the JVM is aware of final fields, the JVM’s verifier enforces different rules. So it is legal in bytecode to reassign a final field any number of times as long as this reassignment is conducted within a constructor call of the class that declares the field. If a constructor invokes another constructor of the same class, it is even possible for the other constructor to reassign a final field. Thus, the following Java class can be legally expressed as byte code:
class Foo {
final int bar;
Foo() {
this(43);
System.out.println(bar);
bar = 42;
}
Foo(int value) {
bar = value;
}
}
After invoking the no-argument constructor of the above class, the only field is assigned the value 42 but the value 43 is printed to the console.
At the same time, it is also possible to skip assigning a value to a final field altogether. In this case, the field is initialized with a default value. Interestingly, such rules are not enforced at all by the bytecode verifier when dealing with static final fields. Static final fields can be reassigned arbitrarily within the declaring class.
Delaying compilation decisions until runtime
The introduction of the invokedynamic instruction ushered several major changes to the implementation of the Java virtual machine. Many Java developers did not perceive the introduction of this new bytecode instruction as very significant. One reason for this view on invokedynamic is probably due to the inability to explicitly define dynamic method invocations using the Java programming language.
With explicit bytecode generation, it is however possible to use invokedynamic to delay the decision of selecting the invocation target of an arbitrary call site until its first execution. This is especially useful for dynamic languages where the information about which method should be invoked cannot be determined until the method is actually executed. At first glance, using invokedynamic can often appear as an equivalent to using Java’s reflection API. However, by using dynamic method invocation, it is possible to explicitly link a method invocation to a call site. With Java 9, this linkage will yield a stack trace that completely hides the dynamic invocation. Besides allowing for a more efficient treatment of primitive types, invokedynamic also allows for a better handling of security as the JVM only allows for a binding of invocation targets that are visible to the class that declares a dynamic call site.
Whenever an invokedynamic call site is created in bytecode, it references a so-called bootstrap method for deciding what method should be invoked. A bootstrap method is implemented in plain Java and serves as a lookup routine for locating the method that should be called for the invokedynamic instruction. This lookup is only performed a single time. It is however possible to manually rebind a dynamic call site at a later time.
By using some pseudo syntax, an invokedynamic instruction can be used for implementing a call site that either calls a method foo or bar, depending on a randomized state.
void someMethod() {
invokedynamic[Bootstrapper::bootstrap]
}
class Bootstrapper {
static CallSite bootstrap(MethodHandles.Lookup lookup, Object... args) {
String name = new Random().nextBoolean() ? "foo": "bar";
MethodType methodType = MethodType.methodType(void.class);
return new ConstantCallSite(lookup.findStatic(Bootstrapper.class,
name, methodType));
}
static void foo() {
System.out.println("foo!");
}
static void bar() {
System.out.println("bar!");
}
}
When the method containing the invokedynamic instruction is called for the first time, the referenced bootstrap method is invoked. As a first argument, it receives a lookup object for looking up methods that are visible to the class enclosing an invokedynamic call site. Depending on the outcome of the randomized name assignment, the invokedynamic call site is either bound to invoking foo or bar by returning the appropriate method. After binding the method, the bootstrap method is never again called for the same call site. At the same time, the invokedynamic call site is from then on handled as if the method call was hard-coded to the formerly dynamic call site.
About the Author
Rafael Winterhalter works as a software engineer in Oslo, Norway. He is a proponent of static typing and a JVM enthusiast with particular interests in code instrumentation, concurrency and functional programming. Rafael blogs about software development, regularly presents at conferences and was pronounced a JavaOne Rock Star. When coding outside of his workplace, he contributes to a wide range of open source projects and often works on Byte Buddy, a library for simple runtime code generation for the Java virtual machine.
