IL Generation in .NET with Sigil

Sigil is a library for generating Common Intermediate Language (CIL). It wraps ILGenerator in a finer-grained interface, automates some optimizations and provides validations for the generated IL. InfoQ reached out with Sigil's creator Kevin Montrose, team lead at StackOverflow, to get a better understanding of ILGenerator and Sigil.

InfoQ: What are the most suited scenarios for generating IL? Is it possible to replace any piece of code using reflection with ILGenerator/Sigil?

Kevin Montrose: Typically IL is generated as a faster alternative to reflection, so tasks like (de)serialization, type mapping, mocking, etc. for which reflection is often used can benefit from Sigil/ILGenerator.

There are a few use cases Sigil doesn't support (which are possible with ILGenerator): fault blocks and emitting code that uses generic (ie. List<T> rather than List<int>) types.  Fault blocks aren't really a thing in C#, and aren't legal in DynamicMethod anyway.  Unbound generic types complicate validation and generation*, and since in the typical case all types are known when using Sigil there's been little demand for to support them.

* Generic types, in the absence of constraints, could be either structs or reference types which has to be reflected in the emitted IL and handled in validation.

InfoQ: What are the challenges of emitting IL at runtime?

KM: When using ILGenerator, there are two big stumbling blocks.

The first is the interface provided.  Most opcodes in ILGenerator are emitted with a call to one of Emit's overloads, the problem is that Emit(...) does no checking that the instruction is actual sensible.

For example

1: var dyn = new DynamicMethod("foo", typeof(void), new Type[0]);
2: var il = dyn.GetILGenerator();
3: il.Emit(OpCodes.Ldc_I4_0);
4: il.Emit(OpCodes.Add, typeof(object));
5: il.Emit(OpCodes.Pop);
6: il.Emit(OpCodes.Ret);
7: var del = (Action)dyn.CreateDelegate(typeof(Action));
8: del();

doesn't fail to compile, even though line #4 is nonsensical (the Add opcode takes no immediate values).

The second issue with ILGenerator is that it does no validation of the correctness of the emitted IL, correctness is only checked when the JIT gets it.  This typically means the first time a delegate or method is invoked (line #8 in the above example), which can be very far removed from the actual mistake.  The raised exception will also lack detail, rarely saying more than "Common Language Runtime detected an invalid program." or "Operation could destabilize the runtime.".

Another, less significant, difficulty is generating ideal IL such as using shorter forms when possible (Br_S instead of Br, Ldc_I4_0 instead of Ldc_I4 & 0, etc.), and reusing locals (rather than use additional slots on the stack).

InfoQ: How does Sigil make IL Generation easier?

KM: Sigil provides a less error prone interface, and (by default) validates the IL stream as it is emitted.

The previous example would be rendered as:

1: var e = Emit<Action>.NewDynamicMethod("foo");
2: e.LoadConstant(0);
3: e.Add(typeof(object));
4: e.Pop();
5: e.Return();
6: var del = e.CreateDelegate();
7: del();

This fails to compile, as Add() takes no parameters.  "Correcting" that to

1: var e = Emit<Action>.NewDynamicMethod("foo");
2: e.LoadConstant(0);
3: e.Add();
4: e.Pop();
5: e.Return();
6: var del = e.CreateDelegate();
7: del();

does compile, but throws a SigilVerificationException at runtime on line #3 "Add expects 2 values on the stack".

If instead we "corrected" the example to

1: var e = Emit<Action>.NewDynamicMethod("foo");
2: e.LoadConstant(0);
3: e.LoadConstant(typeof(object));
4: e.Add();
5: e.Pop();
6: e.Return();
7: var del = e.CreateDelegate();
8: del();

we'd get another SigilVerificationException at runtime - this time on line #4 "Add expected a by ref, double, float, int, long, native int, or pointer; found System.RuntimeTypeHandle".

There are cases, usually around branches, where Sigil can't determine if an IL stream is valid at exactly the erroneous line - but Sigil still fails as soon as it can (always before the JIT is involved), and with more context than ILGenerator provides.

Basically Sigil makes it harder to make common mistakes at compile time, and gives better feedback at runtime when mistakes are made anyway. Sigil also automates opcode selection, using the shortest form possible.

Sigil makes it easier to reuse locals.  In Sigil, locals (variables on the stack) are represented by a Local class which implements IDisposable.  When a Local is requested via Emit.DeclareLocal(), Sigil will reuse any previously disposed Locals' slot if the types match.

For example

var e = Emit<Action>.NewDynamicMethod("foo");
using(var a = e.DeclareLocal<int>()) 
{
  e.LoadConstant(20);
  e.StoreLocal(a); 
}
using(var b = e.DeclareLocal<int>())
{
  e.LoadConstant(30);
  e.StoreLocal(b); 
}
e.Return();
var del = e.CreateDelegate();
del();

will generate

ldc.i4.s 20
stloc.0
ldc.i4.0
stloc.0
ldc.i4.s 30
stloc.0
ret

which reuses the first slot on the stack (as show by stloc.0 there).

InfoQ: What is the performance cost of using Sigil compared to .NET ILGenerator? Does it yield a significant size increase in generated code output?

KM: Sigil does slow down the act of _emitting_ IL, as it's doing considerably more work.  Validation can also impose non-trivial memory pressure during IL emission.  Both are a consequence of the validation failing as quickly as it can, and both tend to grow in proportion to the number of branches in the emitted code.

Sigil does support disabling validation for when you just want a nicer interface for ILGenerator by passing "doVerify:false" when creating an Emit instance.  This considerably speeds up Sigil, though ILGenerator will still be a bit faster.

The IL Sigil generates is no different than what ILGenerator would create (assuming you do by hand all the little optimizations Sigil does for you), so there's no difference in generated code size.

InfoQ: Sigil was recently ported to .NET Core. Did it require significant work?

KM: All that work was actually done by my colleague Marc Gravell in a pull request.  As I understand it, most of the difficulty was in finding the appropriate methods and classes for getting type information and removing anything that isn't possible in .NET Core from that build. 

InfoQ: Sigil is used by Jil, a speed focused JSON serializer you created at Stack Overflow. Are you aware of other projects using Sigil?

KM: Nothing big and open source. It gets some use in other internal Stack Overflow projects, and I suspect digging into Github will reveal some other users.

InhoQ: Are there any new feature you would like to add in Sigil?

KM: I've toyed around with adding fault and generic type support, more for completion's sake than anything else.

A requested feature that would be quite useful, but is very difficult, would be support for saving generated types or delegates to a DLL for future use.  This would allow Sigil to be used as part of a build - which is handy for platforms that don't support runtime code generation (like Xamarin), or apps that can't afford the overhead of using Sigil "in production".

Sigl is an open source project, available on GitHub.For refencing the library, the Nuget package supports .NET 2.0 to 4.5 and CoreCLR.