Simplifying F# Type Provider Development

Type providers are one of the most interesting and empowering features of the F# 3.0 release. Properly written type providers make data access virtually frictionless in F# applications as they eliminate the need for manually developing and maintaining the types which correspond to the underlying data structures. This aspect is particularly important for data exploration tasks where many competing data access technologies require a fair amount of configuration before they're useful.

For all their strengths, type providers tend to be a bit of a black box; once referenced, they usually just work. Not being the type of developer that settles for magical incantations, I recently spent some time delving into their depths.

Getting started with creating a new type provider wasn't nearly as difficult as I expected. Once I found the FSharp.TypeProviders.StarterPack NuGet package which conveniently wraps up the provided types from the F# 3.0 Sample Pack to simplify type provider creation, I was able to write a type provider rather quickly.

Despite the convenience afforded by the provided types, something that has really irked me about them is despite being distributed as F# source files (.fs) they were designed in a highly imperative, object-oriented manner and don't really follow any F# idioms. This results in code that looks out-of-place among the surrounding F# code. Consider the following code from an ID3 type provider which defines a new ProvidedProperty instance and attaches some XML documentation before attaching the property to a provided type identified as ty:

let prop =
  ProvidedProperty(
    "AlbumTitle",
    typeof<string>,
    GetterCode =
      fun [tags] ->
        <@@ (((%%tags:obj) :?> Dictionary<string, ID3Frame>).["TALB"]).GetContent() |> unbox @@>)
prop.AddXmlDocDelayed (fun () -> "Gets the album title. Corresponds to the TALB tag.")
ty.AddMember prop

A similar pattern applies to creating provided methods as shown here:

let method =
  ProvidedMethod(
    "GetTag",
    [ ProvidedParameter("tag", typeof<string>) ],
    typeof<ID3Frame option>,
    InvokeCode =
      (fun [ tags; tag ] -> <@@ let tagDict = ((%%tags:obj) :?> Dictionary<string, ID3Frame>)
      if tagDict.ContainsKey(%%tag:string) then Some tagDict.[(%%tag:string)]
      else None @@>))
method.AddXmlDocDelayed (fun () -> "Returns an ID3Frame object representing the specific tag")
ty.AddMember method

The syntax for both examples is straightforward — particularly for those with an object-oriented background. But to the F# programmer it's tedious; it requires intermediate bindings, doesn't play nicely with pipelining or function composition, and generally doesn't fit the spirit of the language. By writing a few simple functions that take advantage of F#'s statically resolved type parameters we can greatly improve the type provider development experience.

Although each of the provided type classes have their place within a type provider, it seems that the most commonly used are the ProvidedConstructor, ProvidedProperty, ProvidedMethod, and ProvidedParameter classes, so the remainder of this article will focus specifically on those types.

We begin by defining some simple factory functions to wrap calls to the respective constructors as follows: (These functions should be placed in a separate module. Consider decorating the module with the AutoOpen attribute for convenience.)

let inline makeProvidedConstructor parameters invokeCode =
  ProvidedConstructor(parameters, InvokeCode = invokeCode)

let inline makeReadOnlyProvidedProperty< ^T> getterCode propName =
  ProvidedProperty(propName, typeof< ^T>, GetterCode = getterCode)

let inline makeProvidedMethod< 'T> parameters invokeCode methodName =
  ProvidedMethod(methodName, parameters, typeof< 'T>, InvokeCode = invokeCode)

let inline makeProvidedParameter< ^T> paramName =
  ProvidedParameter(paramName, typeof< ^T>)

There isn't much to these functions but there are a few things to note. Foremost is that by writing these as curried functions it's trivial to compose specialized functions that better convey the intent of the provided members via partial application. For example, in our hypothetical ID3 tag type provider example, we could expose individual properties for each ID3 tag. Rather than repeating the code for each tag, changing only the property and tag names, we could compose a new makeTagProperty function that sets the property type to string, accepts the tag and property name, and automatically builds the getter code expression.

Next, each of the functions include the inline modifier. This instructs the compiler to insert the method body at the call site in place of the function call, thus eliminating the associated overhead. The inline modifier is often used in conjunction with operators but can be useful for this type of function as well.

The most interesting aspect of each of the functions (except the makeProvidedConstructor function) is found in their use of generics. In each case generics are used to partially abstract away how the provided member's return type is specified. I prefer this approach as it provides consistency with other parts of the .NET Framework and isolates the calls to typeof to these functions. More important than the abstraction the generics provide is that rather than relying on regular type parameters the generics here use statically resolved type parameters as indicated by the ^ prefix rather than the standard apostrophe.

This distinction is of particular importance because it affects how the type provider is compiled. With regular generics the type parameters are resolved at run time as is always the case in C# and Visual Basic. With F#'s statically resolved type parameters, the parameter types are resolved at compile time which generally results in more efficient code. Furthermore, statically resolved type parameters also allow a variety of additional constraint types which are not allowed on regular type parameters. One such constraint type is the member constraint which will be highlighted shortly.

Thus far all we've really achieved with our helper functions is wrapping some constructors. While these provide some extra convenience, they do little to advance us toward our goal of writing more idiomatic F# code. To that end, let's turn our focus to the AddXmlDocDelayed method from the initial code sample.

It would be nice to attach the XML documentation to the members as part of a pipeline or composition chain. Despite the fact that the AddXmlDocDelayed method exists on each of the provided types we've discussed so far (except ProvidedParameter) in each case it stands alone — there is no single, unifying interface which we can reference in a new function. In fact, each of the provided member types simply derive from a corresponding MemberInfo implementation and don't reference any other interfaces. This leaves us with but a few options:

• Create individual functions for each provided member type,
• Use reflection,
• Use dynamic type-test patterns to get the appropriate type and corresponding method, or
• Channel some black magic from F#'s statically resolved type parameters to write a function constrained to executing against only those types that include an AddXmlDocDelayed method.

None of the first three options are appealing. Writing separate functions for each supported provided type is likely to cause future maintenance issues. Reflection is viable but requires additional error handling and won't provide any compile time support. Dynamic type-test patterns are more idiomatic F# and will certainly provide us with compile time checking but we must provide match cases for each allowable type. Only by leveraging a member-constrained statically resolved type parameter can we get compile time checking without being explicit about the types we want to work with.

Here is the function in its entirety:

let inline addDelayedXmlComment comment providedMember =
  (^a : (member AddXmlDocDelayed : (unit -> string) -> unit) providedMember, (fun () -> comment))
  providedMember

Excluding the signature, the addDelayedXmlComment function is a mere two lines of code. Unlike the previous factory functions, we've allowed the compiler to infer more about the function by omitting the explicit type parameter from the function's signature, instead placing those details in the function body.

The function body's first line can be likened to using reflection to obtain a reference to a MethodInfo instance representing the type's AddXmlDocDelayed method and calling its Invoke method except here the resolution is occurring at compile time. Here, the type we're "reflecting" upon is denoted as ^a, which indicates that it's a statically resolved type parameter. Next is the member constraint indicating that ^a must have a member named AddXmlDocDelayed which accepts a function (unit -> string) and returns unit. Finally we pass in the arguments in tupled form with the first argument, providedMember, being akin to the object parameter supplied to MethodInfo.Invoke and the second argument being the function that AddXmlDocDelayed will use to generate the comment.

The second line of the function simply returns the provided member. This allows the provided member to continue being passed along the function chain.

With these new helper functions in place, the original imperative provided property code can be rewritten as follows:

"AlbumTitle"
|> makeReadOnlyProvidedProperty<string>
    (fun [tags] ->
      <@@ (((%%tags:obj) :?> Dictionary<string, ID3Frame>).["TALB"]).GetContent() |> unbox @@>))
|> addDelayedXmlComment "Gets the album title. Corresponds to the TALB tag."
|> ty.AddMember

And the provided method can be rewritten like this:

"GetTag"
|> makeProvidedMethod<ID3Frame option>
     [ makeProvidedParameter<string> "tag" ]
     (fun [ tags; tag ] -> <@@ let tagDict = ((%%tags:obj) :?> Dictionary<string, ID3Frame>)
                               if tagDict.ContainsKey(%%tag:string) then Some tagDict.[(%%tag:string)]
                               else None @@>)
|> addDelayedXmlComment "Returns an ID3Frame object representing the specific tag" |> ty.AddMember

Here it is apparent through pipelining that we're defining a read only string property or method, attaching some XML documentation, and adding the member to the provided type.

Given that most MP3 files have multiple string tags, it seems likely that the provided property code would be repeated for each of those tags. Rather than duplicating the code, changing only the tag and comment text, we can further leverage partial application of our helper functions to compose a specialized factory function:

let inline makeTagPropertyWithComment tag comment =
  let expr =
    fun [tags] ->
      <@@ (((%%tags:obj) :?> Dictionary<string, ID3Frame>).[tag]).GetContent() |> unbox @@>
  (makeReadOnlyProvidedProperty<string> expr) >> (addDelayedXmlComment comment)

The makeTagPropertyWithComment function uses the forward composition operator to compose a new function that first creates the provided property then adds the delayed XML comment. The function's return value is the provided property as evidenced by its signature:

string -> string -> (string -> ProvidedProperty)

As a result, we're free to pass the resulting provided property on to another function. By using this function, our tag property can be further reduced to:

"AlbumTitle"
|> makeTagPropertyWithComment tag "Gets the album title. Corresponds to the TALB tag."
|> ty.AddMember

The difference between this version and the original, imperative version is astounding. Through some simple inline wrapper functions with statically resolved type parameters and member constraints, we've managed to reduce the code necessary for creating a provided property, adding delayed XML documentation, and attaching the property to a type by approximately 60%. In the process the code has become more idiomatic F# by eliminating the intermediate variable (prop) and replacing most direct method invocations with pipelined functions. What's more, extending these examples to provide additional functionality from the various provided types follows the same patterns we've just covered.

It would be nice to see some of these techniques make it into future versions of F# or even the type provider starter pack. Perhaps it's time to submit a pull request!

About the Author

Dave Fancher has been developing software with the .NET Framework for more than a decade. He is a familiar face in the Indiana development community as both a speaker and participant in user groups around the state. In July 2013, Dave was recognized as a Microsoft MVP (Most Valuable Professional) for Visual F#. When not writing code or writing about code at davefancher.com, he can often be found watching a movie or gaming on his Xbox One.