ezyang's blog

Safety first: FFI and threading

Update. While this blog post presents two true facts, it gets the causal relationship between the two facts wrong. Here is the correction.

Attention conservation notice. Don’t use unsafe in your FFI imports! We really mean it!

Consider the following example in from an old version of Haskellwiki’s FFI introduction:

{-# INCLUDE <math.h> #-}
{-# LANGUAGE ForeignFunctionInterface #-}
module FfiExample where
import Foreign.C -- get the C types

-- pure function
-- "unsafe" means it's slightly faster but can't callback to haskell
foreign import ccall unsafe "sin" c_sin :: CDouble -> CDouble
sin :: Double -> Double
sin d = realToFrac (c_sin (realToFrac d))

The comment blithely notes that the function can’t “callback to Haskell.” Someone first learning about the FFI might think, “Oh, that means I can put most unsafe on most of my FFI declarations, since I’m not going to do anything advanced like call back to Haskell.”

Oh my friend, if only it were that simple!

Recall that when you create a thread in Haskell with forkIO, you’re not creating a real operating system thread; you’re creating a green thread that Haskell’s runtime system manages across its pool of operating system threads. This is usually very good: real threads are heavyweight, but Haskell threads are light and you can use a lot of them without paying too much. But here’s the rub:

In particular, when you invoke an unsafe FFI import, you effectively suspend everything else going on in the system: Haskell is not able to preempt it (in particular unsafe indicated that there was no need to save the state of the RTS), and the foreign code will keep running by itself until it finishes.

Don’t believe me? Try it out yourself (I conducted my tests on 6.12.1). You’ll need a few files:

/* cbit.c */
#include <stdio.h>
int bottom(int a) {
    while (1) {printf("%d\n", a);sleep(1);}
    return a;
}

/* cbit.h */
int bottom(int a);

And UnsafeFFITest.hs:

{-# LANGUAGE ForeignFunctionInterface #-}

import Foreign.C
import Control.Concurrent

main = do
    forkIO $ do
        safeBottom 1
        return ()
    yield
    print "Pass (expected)"
    forkIO $ do
        unsafeBottom 2
        return ()
    yield
    print "Pass (not expected)"

foreign import ccall "cbit.h bottom" safeBottom :: CInt -> IO CInt
foreign import ccall unsafe "cbit.h bottom" unsafeBottom :: CInt -> IO CInt

Compile and run the relevant files with:

gcc -c -o cbit.o cbit.c
ghc -threaded --make UnsafeFFITest.hs cbit.o
./UnsafeFFITest +RTS -N4

The output you see should be similar to this:

ezyang@javelin:~/Dev/haskell/unsafe-ffi$ ./UnsafeFFITest +RTS -N2
1
"Pass (expected)"
2
1
2
1
2

The first call played nice and let Haskell move along, but the second call didn’t. Some things to try for yourself include swapping the order of the forks, using forkOS (which many people, including myself, incorrectly assumed creates another operating system call) and changing the RTS option -N.

What does this mean? Essentially, only if you’re really sure Haskell will never have to preempt your C call (which I would not be comfortable saying except for the smallest, purest C functions), don’t use unsafe. It’s not worth it. Safety first!

Postscript. Thanks #haskell for helping me hash out this line of thought (I’d run into this behavior earlier, but it hadn’t occurred to me that it was bloggable.)

Postscript 2. Thanks to Simon Marlow for clarifying some mistakes that I made in my original treatment of the topic. If you’re interested in more details about the interaction of concurrency and the FFI, check out the paper he pointed to: Extending the Haskell Foreign Function Interface with Concurrency.

Groom: human readable Show for Haskell

Tapping away at a complex datastructure, I find myself facing a veritable wall of Babel.

“Zounds!” I exclaim, “The GHC gods have cursed me once again with a derived Show instance with no whitespace!” I mutter discontently to myself, and begin pairing up parentheses and brackets, scanning the sheet of text for some discernible feature that may tell me of the data I am looking for.

But then, a thought comes to me: “Show is specified to be a valid Haskell expression without whitespace. What if I parsed it and then pretty-printed the resulting AST?”

Four lines of code later (with the help of Language.Haskell)…

Ah, much better!

How to use it. In your shell:

cabal install groom

and in your program:

import Text.Groom
main = putStrLn . groom $ yourDataStructureHere

Update. Gleb writes in to mention ipprint which does essentially the same thing but also has a function for putStrLn . show and has some tweaked defaults including knowledge of your terminal size.

Update 2. Don mentions to me the pretty-show package by Iavor S. Diatchki which also does similar functionality, and comes with an executable that lets you prettify output offline!

Little’s law

A short thought from standing in line at the World Expo: Little’s law is a remarkable result that relates the number of people in a queue, the arrival rate of people to the queue, and the time spent waiting in the queue. It seems that it could be easily applied to a most universal feature of theme parks: waiting queues. Instead of such unreliable methods as giving visitors tokens to test how long it takes to traverse some portion of the line and then eyeballing the wait time from there, it would be a simple matter to install two gates: one to count incoming visitors and one to count outgoing visitors, and with this data derive an instantaneous “wait time in queue” figure based on a smoothed running average of queue size and arrival rate. Added benefit for being electronic, which means you can easily beam it to information boards across the park!

It occurs to me that I don’t have any evidence that theme parks aren’t already using this technique, except maybe for the fact that their posted wait times are wildly inaccurate.

MVC and Purity

Attention conservation notice. Purely functional programming demonstrates the same practices recommended by object-oriented MVC practice.

Model-View-Controller is a widely used object-oriented design pattern for organizing functionality in an application with a user interface. I first ran across it in my early days programming web applications. The Model/View separation made deep intuitive sense to me as a PHP programmer: without it, you’d end up with spaghetti templates with HTML print statements interleaved with MySQL queries. But Controller was always a little wishy-washy. What exactly did it do? It was some sort of “glue” code, the kind of stuff that bound together the Model and View and gave them orders. But this was always a sort of half-hearted answer for me (where should input validation go?), and soon I left the world of web applications, my questions unanswered.

Having been exposed to purely functional programming, I now believe that the controller and model/view separation is precisely the separation between side-effectful code (IO) and pure code.

The controller depends on the model and the view, but the model and view should not (directly) depend on the controller. Pure code and impure code don’t mix freely. In particular, you’re not allowed to reference impure code from pure code (unless you use unsafePerformIO). However, impure code can call pure code (although there may be some technical details involved), and the resulting code is impure. So, if the Controller is impure code and the Model/View is pure code, separating the two is simply making sure that if we have any code that is impure, we’ve extracted as much of the pure computation out of it as possible. Stated differently, if I have a function that reads and writes data, and there are lines in it that don’t have anything to do with IO, I should move them into their own function. Maybe those lines are the templating system, in which case it’s View; maybe those lines are running some complicated equation, in which case it’s Model. Pure/impure doesn’t capture the model/view distinction.

The controller receives input and initiates a response. So, the controller is input-output, i.e. IO.

The controller handles events that affect the model or the view. Pure code sort of lives in a vacuum: it can do computation, but it can’t do anything useful, since it can’t have any side effects and thus has no way for us to tell it what to compute, or to view the result of the computation. Impure code is the way we get anything done by handing off this information to our pure code.

There are several possible objections to this division. Here are a few of them:

Most object-oriented models are stateful, and state is not pure! There is a common misconception that state isn’t pure, possibly arising from the fact that both IO and State are monads. However, I can turn a state monad into a single, pure value by simply running the state machine: code that is stateful is monadic, but it is also pure, since it doesn’t have any external side effects. Shared state is a bit trickier, and usually not pure.

Controller code doesn’t have to be impure and here’s an example. Here I’ll indulge in a bit of prescriptivism: I bet you have a model, but one that is only tangentially related to your core business logic. If you have code that parses binary strings into message objects (but doesn’t actually handle transmitting or receiving those binary strings on the network), you have a mini-model of network messages. You should probably keep it separate from your real model, but for testability you should also keep it separate from your network code. Separation of concerns may be malleable, but the little IO in your type signature is always honest.

Some parting words about the purity hair-shirt: it is fairly widely acknowledged that busting out the MVC pattern makes your application more complex initially, and in a purely functional language, you’re forced to respect the distinction from the very beginning. Thus, writing small programs can be frustrating in a purely functional language because you don’t want to use the bulky but scalable engineering practice yet, and the language is making you do so from the very beginning. Haskell gives you a lot of rope to make it pleasant again, but it takes a while to get used to. On the plus side, when your program grows, this separation will continue to be enforced, and a messy refactoring may be averted.

Call and fun: marshalling redux

This part six of a six part introduction to c2hs. We finally talk about what ostensibly is the point of c2hs: calling C functions from Haskell. c2hs, due to its knowledge of the C headers, can already do the work for generating FFI imports. The call hook simply tells c2hs to generate the FFI import, while the fun hook generates another Haskell function which performs marshalling.

Call. The format of call is quite simple, because like get and set, it is meant to be interleaved with other Haskell code. If I would like to invoke the readline function from readline/readline.h, a {#call readline #} would suffice; c2hs will then generate the FFI import with the correct signature and transform the call directive into the name of the FFI import.

Of course, readline doesn’t call back to Haskell, so we could add unsafe: {#call unsafe readline #}. And if you’re sure that the C function has no side-effects, you can add pure: {#call pure unsafe sin #}. If you have multiple calls to the same function using the same FFI declaration, their flags need to be consistent.

By default, the cid will be use precisely to determine the name of the FFI import; if it is not a valid Haskell identifier for a function (i.e. is capitalized) or the C function name would conflict with another, you’ll need to specify what the FFI will import as. Common conventions include prefixing the function with c_, or you can use ^ for c2hs’s capitalization conversion. {#call FooBar_Baz as ^ #} will convert to fooBarBaz (with an appropriate FFI declaration).

Fun. Because the signature of the FFI declarations will all be C types, and Haskell programs tend not to use those, and because it is a frequent operation to convert to and from the C types, there’s a little bit of automation to help you out with the fun directive. Unlike call, it’s intended to standalone as a definition, and not be embedded in code. Note that you don’t have to use fun; gtk2hs doesn’t use it, for example. However, many people find it useful.

A fun starts off much like a call: you first specify if it’s pure and/or unsafe, specify the C identifier, and the specify the Haskell name. Since the majority of your code will refer to the Haskell name, it’s usually best to specify ^ for a consistent naming convention.

From here, we need to specify what the end type of the desired Haskell function is, and how to go from those types to the C types (the marshalling functions). The c2hs tutorial has a bit to say on this topic, so we’ll take a more example oriented approach.

Primitive C types. The integral, floating point and boolean (usually an integer under the hood) primitive C types are so prevalent that c2hs will automatically use the cIntConv, cFloatConv and cFromBool/cToBool functions to marshal them if none are specified. These functions work in both directions. This directive:

{#fun pure sinf as ^
  { `Float' } -> `Float' #}

generates:

sinf :: Float -> Float
sinf a1 =
  let {a1' = cFloatConv a1} in 
  let {res = sinf'_ a1'} in
  let {res' = cFloatConv res} in
  (res')

You can see that a bunch of (ugly) generated code is added to run the marshalling function on the argument, pass it to the FFI, and then another marshalling function is called on the result. Idiomatic Haskell might look like:

sinf = cFloatConv . sinf'_ . cFloatConv

If you’d like to use a different name for the marshalling function, you can specify it before the type of an argument (an “in” marshaller), or after the result (an “out” marshaller), as such:

{#fun pure sinf as ^
  { myFloatConv `Float` } -> `Float` myFloatConv

and you can just replace the relevant function calls in the generated Haskell.

String arguments. Strings also hold a special place in c2hs’s heart; null-terminated and strings needing explicit length information specified are handled with ease. Consider these two function prototypes:

void print_null_str(char *str);
void print_explicit_str(char *str, int length);

We can write the following c2hs directives:

{#fun print_null_str as ^ { `String' } -> `()' }
{#fun print_explicit_str as ^ { `String'& } -> `()' }

and they will be automatically be marshalled with withCString* and withCStringLen*.

There are several interesting things happening here. We represent a void return type using () (the empty type in Haskell). Additionally, the String parameter in print_explicit_str has an ampersand affixed to it; this means that the marshaller should produce a tuple of arguments which will be passed to the function as two separate arguments. Sure enough, withCStringLen results in a (Ptr CChar, Int), and c2hs use a slight variant withCStringLenIntConv which converts the Int into a CInt. (Note that if you need more complicated multi-argument ordering, fun is not for you.)

But perhaps the most interesting thing is the * affixed to the input marshaller, which has two effects. The first is to indicate that the input marshalling function is the IO monad, for example, the type of withCString is String -> (CString -> IO a) -> IO a. But furthermore, it indicates a function that follows the bracketed resource pattern “with”. We did not useString -> CString, since this could result in a memory leak if we don’t free the CString later! The code generated is then:

printNullStr :: String -> IO ()
printNullStr a1 =
  withCString a1 $ \a1' -> 
  printNullStr'_ a1' >>= \res ->
  return ()

printExplicitStr :: String -> IO ()
printExplicitStr a1 =
  withCStringLenIntConv a1 $ \(a1'1, a1'2) -> 
  printExplicitStr'_ a1'1  a1'2 >>= \res ->
  return ()

which makes use of hanging lambdas to keep the layout consistent.

Marshalling struct arguments. While the c2hs documentation claims that there is a default marshaller if you have the following situation in C:

struct my_struct { int b; int c; };
void frob_struct(struct my_struct *);

and in Haskell:

data MyStruct = MyStruct Int Int
instance Storable MyStruct where ...
{#pointer *my_struct as MyStructPtr -> MyStruct #}

So you should be able to write:

{#fun frob_struct as ^ { `MyStruct' } -> `()' #}

Where, the input marshaller is with*. Unfortunately, I could never get that to work; furthermore, c2hs thinks that with is a reserved word, so you’ll need to rename it in order to use it. :

withT = with
{#fun copy_struct as ^ { withT* `MyStruct' } -> `()' #}

Opaque pointer arguments. When you don’t want to perform any tomfoolery on a pointer when in Haskell, you can simply specify that the pointer is the argument and use id as the marshaller. In the previous example, copy_struct could have alternately been defined as:

{#fun copy_struct as ^ { id `MyStructPtr' } -> `()' #}

A convention is to omit Ptr from the name of the pointer type if you are only dealing with opaque pointers.

Out marshalling input arguments. A frequent pattern in C code is using pointer arguments to permit a function to return multiple results. For example, strtol has the following signature:

long int strtol(const char *nptr, char **endptr, int base);

endptr points to a pointer which will get set to the pointer at the end of the portion of the string in nptr we parsed. If we don’t care about it, we can set endptr = NULL.

Obviously, we don’t want our Haskell function to do this, and we have much easier ways of returning multiple results with tuples, so c2hs has a notion of an outmarshaller for an input argument. It also has the notion of a “fake” input argument which the user doesn’t have to pass, in case our function is completely responsible for allocating the memory the pointer we pass to the function points to.

Here’s a first attempt at writing a fun hook for strtol:

{#fun strtol as ^ {id `Ptr CChar', id `Ptr (Ptr CChar)', `Int'} -> `Int` #}

We’ve eschewed the default string marshalling because otherwise endptr won’t give us very interesting information. This version is a carbon copy of the original.

To improve this, we consider Ptr (Ptr CChar) to be a way of returning Ptr CChar. So, after the function is run, we should peek (dereference the pointer) and return the result:

{#fun strtol as ^ {id `Ptr CChar', withT* `Ptr CChar' peek*, `Int'} -> `Int' #}

peek is in IO, so it needs the asterisk, but for out marshallers it doesn’t result in any fancy bracketing usage. Now, the Haskell return type of this function is not Int; it’s (Int, Ptr CChar). :

strtol :: Ptr CChar -> Ptr CChar -> Int -> IO (Int, Ptr CChar)
strtol a1 a2 a3 =
  let {a1' = id a1} in 
  withT a2 $ \a2' -> 
  let {a3' = cIntConv a3} in 
  strtol'_ a1' a2' a3' >>= \res ->
  peek a2'>>= \a2'' -> 
  let {res' = cIntConv res} in
  return (res', a2'')

Since we’re overwriting the original contents of the pointer, it doesn’t make much since to force the user of our function to pass it to us. We can suffix our input marshaller with - to indicate that it’s not a real Haskell argument, and use alloca instead:

{#fun strtol as ^ {id `Ptr CChar', alloca- `Ptr CChar' peek*, `Int'} -> `Int' #}

Notice that we got rid of the *; it’s one or the other. Now we have a usable function:

strtol :: Ptr CChar -> Int -> IO (Int, Ptr CChar)
strtol a1 a3 =
  let {a1' = id a1} in 
  alloca $ \a2' -> 
  let {a3' = cIntConv a3} in 
  strtol'_ a1' a2' a3' >>= \res ->
  peek a2'>>= \a2'' -> 
  let {res' = cIntConv res} in
  return (res', a2'')

or, in idiomatic Haskell:

strtol nptr base = alloca $ \endptr -> do
  result <- strtol'_ nptr endptr (cIntconv base)
  end <- peek endptr
  return (result, end)

Error handling. There is one last piece of functionality that we haven’t discussed, which is the - flag on an out marshaller, which causes Haskell to ignore the result. By itself it’s not ordinarily useful, but when combined with * (which indicates the action is in IO), it can be used to attach functions that check for error conditions and throw an exception if that is the case. Recall that the default output marshaller for () is void-, ignoring the output result of a function.

Thinking about talk

This one’s for the MIT crowd.

I will unfortunately not be in Boston over IAP, so I won’t be able to do a redux of the class I taught last year, Advanced Typeclasses in Haskell. However, since I will be in Boston for September, it might be a good time to do cluedump for SIPB this year. I love talking about Haskell, and so I could do another talk in a similar vein (maybe something that covers rank-2 types and existential quantification?) I’ve also been thinking that doing an architectural overview of Scripts would also be good.

What would you like to see me talk about?

Marshalling with get and set

This part five of a six part introduction to c2hs. Today, we explain how to marshal data to and from C structures.

An important note. There is a difference between struct foo and foo; c2hs only considers the latter a type, so you may need to add some typedefs of the form typedef struct foo foo in order to get c2hs to recognize these structs.

Get. The Haskell FFI has no knowledge of C structs; Haskell’s idea of reading a member of a struct is to peek at some byte offset of a memory location, which you calculated manually. This is horrid, and hsc2hs has #peek to relieve you of this non-portable drudgery. c2hs has something even simpler: you can specify {#get StructName->struct_field #} and c2hs will replace this with a lambda that does the correct peek with the correct type: (\ptr -> do {peekByteOff ptr 12 ::IO CInt}) (in the IO monad!) Note the following gotchas:

• You will need to manually convert the resulting primitive C type into a more friendly Haskell type, and
• The left hand side of the expression is a type or a struct name, not the Haskell variable containing the pointer/struct you want to peek at. That will usually go to the right of the lambda.

The get directive is actually more general than just struct access: it can dereference pointers (*StructName) or access a member without dereferencing (StructName.struct_field).

Set. The opposite of get, set lets you poke values into arbitrary memory locations. Unlike get, the value passed in is required to be a pointer (and the syntax uses periods). {#set StructName.struct_field #} expands to (\ptr val -> do {pokeByteOff ptr 12 (val::CInt)}); the pointer is the first argument and the value is the second. You also need to marshal the input value manually.

Defining Storable. If you’re not individually getting and setting fields in the struct in an opaque pointer, creating a Storable instance is a good thing to do. However, since all of the lambdas that get and set create are in the IO monad, composing them can be slightly tricky. Judicious use of monadic lifting and applicative instances can make the code a lot simpler, however:

data StructName = StructName
  { struct_field1'StructName :: Int
  , struct_field2'StructName :: Int
  }
instance Storable StructName where
  sizeOf _ = {#sizeof StructName #}
  alignment _ = 4
  peek p = StructName
    <$> liftM fromIntegral ({#get StructName->struct_field1 #} p)
    <*> liftM fromIntegral ({#get StructName->struct_field2 #} p)
  poke p x = do
    {#set StructName.struct_field1 #} p (fromIntegral $ struct_field1'StructName x)
    {#set StructName.struct_field2 #} p (fromIntegral $ struct_field2'StructName x)

The odd naming convention in StructName is to account for the fact that different structures can share field names, while Haskell field names may not.

Note. c2hs recently got support added for an alignment directive, which computes the alignment for a C datastructure. Unfortunately, as of 0.6.12, this has not yet been released to the general public.

Request. The paper describing c2hs states the following: “[Marshaling of compound C values to Haskell values] is more generally useful; however, often we do not really want to marshal entire C structures to Haskell.” Unfortunately, current incarnations of c2hs do not offer any optional functionality to reduce the drudgery of writing the “straightforward” Storable instance, which would be absolutely lovely. bindings-dsl and GreenCard appear to fare better in this respect.

Next time. Call and fun: marshalling redux

First steps in c2hs

This is part four of a six part tutorial series on c2hs. Today we discuss the simple things in c2hs, namely the type, enum, pointer, import and context directives.

Prior art. All of the directives c2hs supports are tersely described in the “tutorial” page (which would perhaps be more accurately described as a “reference manual”, not tutorial.) There is also (paradoxically) a much more informal introduction for most of the directives in c2hs’s research paper.

Type. C code will occasionally contain macro conditionals redefining a type depending on some build condition (the following is real code):

#if       defined(__ccdoc__)
typedef platform_dependent_type ABC_PTRUINT_T;
#elif     defined(LIN64)
typedef unsigned long ABC_PTRUINT_T;
#elif     defined(NT64)
typedef unsigned long long ABC_PTRUINT_T;
#elif     defined(NT) || defined(LIN) || defined(WIN32)
typedef unsigned int ABC_PTRUINT_T;
#else
   #error unknown platform
#endif /* defined(PLATFORM) */

If you wanted to refer to write FFI code that referenced functions that used ABC_PTRUINT_T, you might have to have performed some guess on what the value truly is in Haskell or used the C preprocessor to reimplement the conditions. With c2hs you can retrieve the true value of a typedef with type:

type ABC_PTRUINT_T = {#type ABC_PTRUINT_T #}

Consider the case of a 64-bit Linux system (such that __ccdoc__ is undefined and LIN64 is defined), then the result is:

type ABC_PTRUINT_T = CLong

Enum. Enums show up frequently in well-written (i.e. eschewing magic numbers) C code:

enum Abc_VerbLevel 
{
   ABC_PROMPT   = -2, 
   ABC_ERROR    = -1, 
   ABC_WARNING  =  0, 
   ABC_STANDARD =  1, 
   ABC_VERBOSE  =  2 
}; 

However, underneath the hood, these are really just ints, so Haskell code that wants to pass an enum value to a function has to:

1. Create a new datatype to represent the enumeration, and
2. Write a function that maps from that datatype to the C integer and back again for the Enum instance.

We can have c2hs do all the work for us:

{#enum Abc_VerbLevel {underscoreToCase} deriving (Show, Eq) #}

which becomes:

data Abc_VerbLevel = AbcPrompt | AbcError | AbcWarning | AbcStandard | AbcVerbose
  deriving (Show, Eq)
instance Enum Abc_VerbLevel
  fromEnum AbcPrompt = -2
  -- ...

Note that, as ABC_PROMPT is a very unsightly constructor in Haskell, we transform the names using the underscoreToCase algorithm as mentioned before. You can also explicitly list out the renamings:

{#enum Abc_VerbLevel {AbcPrompt, AbcError, AbcWarning, AbcStandard, AbcVerbose} #}

or change the name of the datatype:

{#enum Abc_VerbLevel as AbcVerbLevel {underscoreToCase} #}

There are two other transforms (which can combine with underscoreToCase: upcaseFirstLetter and downcaseFirstLetter, though I’m not sure when the latter would result in working Haskell code.

Pointer. Unlike C primitives, which are specified in Foreign.C.Types, Haskell needs to be told how to map pointer types (foo*) into Haskell types. Consider some struct:

struct foobar {
  int foo;
  int bar;
}

It is quite conceivable that there exists a data Foobar = Foobar Int Int in the Haskell codebase, in which case we would like Ptr Foobar to represent a struct foobar* in the original C code. c2hs has no way of deriving this information directly, so we give it this information:

{#pointer *foobar as FoobarPtr -> Foobar #}

This generates the code:

type FoobarPtr = Ptr Foobar

But more importantly, allows c2hs to place more specific types in the signatures it writes for FFI bindings (which we will see in the next post in the series.)

Some variations on the theme:

• If you want to represent an opaque pointer whose contents will not be marshalled, you can either do empty data declarations:

  data Foobar
  {#pointer *foobar as FoobarPtr -> Foobar #}
  

  or you can have c2hs generate code using the newtype trick:

  {#pointer *foobar as FoobarPtr newtype #}
  

  I prefer empty data declarations, since there’s no need to wrap and unwrap a newtype in that case: the newtype will generate:

  newtype FoobarPtr = FoobarPtr (Ptr FoobarPtr)
  

  which, for any code expecting Ptr a, needs to be unwrapped.

• If you do not care for the name FoobarPtr and would just like to explicitly say Ptr Foobar, you can tell c2hs not to emit the type definition with nocode:

  {#pointer *foobar -> Foobar nocode #}
  

• If no Haskell name mapping is specified, it will simply use the C name:

  -- if it was struct Foobar...
  {#pointer *Foobar #}
  

• If you would like to refer to a typedef in C which is already a pointer, just omit the asterisk:

  typedef struct Foobar*   FoobarPtr
  {#pointer FoobarPtr #}
  

• c2hs also has limited support for declaring pointers as foreign or stable, and generating code accordingly. I’ve not used this, except in one case where I found the generated bindings for the pointer were not flexible enough. Your mileage may vary.

Import. A C library that contains multiple header files will probably have some headers including others to get vital type definitions. If you organize your Haskell modules similarly, you need to mimic these includes: this can be done with import. :

{#import Foobar.Internal.Common #}

In particular, this sets up the pointer mappings from the other module, as well as generating the usual import statement.

Context (optional). Context has two purported purposes. The first is to specify what library the FFI declarations in the file should be linked against; however, in Cabal, this doesn’t actually do anything—so you need to still add the library to Extra-libraries. The second is to save you keystrokes by adding an implicit prefix to every C identifier you reference, in the case that the original C code was namespaced gtk_ or similarly. I personally like not needing to have to qualify my imports to the lower level API and like the visual distinction of C prefixes, so I tend to omit this. Some directives let you change the prefix locally, in particular enum.

Next time. Marshalling with get and set.

The secret to successful autogenerated docs

I’ve had a rather successful tenure with autogenerated documentation, both as a writer and a reader. So when Jacob Kaplan Moss’s articles on writing “great documentation” resurfaced on Reddit and had some harsh words about auto-generated documentation, I sat back a moment and thought about why autogenerated documentation leave developers with a bad taste in their mouths.

I interpreted Moss’s specific objections (besides asserting that they were “worthless” as the following:

1. They usually didn’t contain the information you were looking for (“At best it’s a slightly improved version of simply browsing through the source”),
2. They were verbose (“good for is filling printed pages when contracts dictate delivery of a certain number of pages of documentation”),
3. The writers skipped the “writing” part (“There’s no substitute for documentation written…”),
4. The writers skipped the “organizing” part (“…organized…”),
5. The writers skipped the “editing” part (“…and edited by hand.”),
6. It gave the illusion of having documentation (“…it lets maintainers fool themselves into thinking they have documentation”).

Thus, the secret to successful autogenerated docs is:

No doubt, gentle reader, you are lifting your eyebrow at me, thinking to yourself, “Of course you should remember your audience; that’s what they always teach you in any writing class. You haven’t told us anything useful!” So, let me elaborate.

Why do developers forget to “remember their audience”? A defining characteristic of autogenerated documentation is that the medium it is derived from is source code: lines of programming language and docblocks interleaved together. This has certain benefits: for one, keeping the comments close to the code they describe helps ward off documentation rot as code changes, additionally, source divers have easy access to documentation pertinent to a source file they are reading. But documentation is frequently oriented towards people who are not interested in reading the codebase, and thus the act of writing code and documentation at the same time puts the writer into the wrong mindset. Compare this with sitting down to a tutorial, the text flowing into an empty document unprejudiced by such petty concerns as code.

This is a shame, because in the case of end-user developer documentation (really the only appropriate time for autodocs), the person who originally wrote the code is most likely to have the relevant knowledge to share about the interface being documented.

What does it mean to “remember my audience”? Concisely, it’s putting yourself into your end-user’s shoes and asking yourself, “If I wanted to find out information about this API, what would I be looking for?” This can be hard (and unfortunately, there are no secrets here), but the first step is to be thinking about it at all.

How can I remember to consider the audience when writing docblocks? While it would be nice if I could just snap my fingers and say, “I’m going to write docblocks with my audience in mind,” I know that I’m going to forget and write a snippy docblock because I was in a rush one day or omit the docblock entirely and forget about it. Writing documentation immediately after the code is written can be frustrating if five minutes later you decide that function did the wrong thing and needs to be axed.

Thus, I’ve set up these two rules for myself:

1. It’s OK not to write documentation immediately after writing code (better not yet than poorly).

   Like many people working in high-level languages, I like using code to prototype API designs. I’ll write something, try to use it, change it to fit my use-case, write some more, and eventually I’ll have both working code and a working design. If I don’t write documentation while I’m prototyping, that’s fine, but when that’s all over I need to write the documentation (hopefully before the code slips out of my active mindspace.) The act of writing the documentation at the end helps finalize the API, and can suggest finishing touches. I also use my toolchain to tell me when I’ve left code undocumented (with Sphinx, this is using the coverage plugin).

2. When writing documentation, constantly look at the output the end-user will see.

   You probably have a write/preview authoring cycle when you edit any sort of text that contains formatting. This cycle should carry over to docblocks: you edit your docblock, run your documentation build script, and inspect the results in your browser. It helps if the output you’re producing is beautiful! It also means that your documentation toolchain should be smart about what it needs to recompile when you make changes. The act of inspecting what a live user will see helps put you in the right mindset, and also force you to say, “Yeah, these docs are not actually acceptable.”

My autodocumentor produces verbose and unorganized output! I’ve generally found autogenerated documentation from Python or Haskell to be much more pleasant to read than that from Java or C++. The key difference between these languages is that Python and Haskell organize their modules into files; thus, programmers in those language find it easier to remember the module docblock!

The module docblock is one of great import. If your code is well-written and well-named, a competent source-diver can usually figure out what a particular function does in only a few times longer than it would take for them to read your docblock. The module is the first organizational unit above class and function, precisely where documentation starts becoming the most useful. It is the first form of “high-level documentation” that developers pine for.

So, in Python and Haskell, you write all of the functionality involved in a module in a file, and you can stick a docblock up top that says what the entire file does. Easy! But in Java and C++, each file is a class (frequently a small one), so you don’t get a chance to do that. Java and recent C++ have namespaces, which can play a similar role, but where are you supposed to put the docblock for what in Java is effectively a directory?

There is also substantial verbosity pollution that comes from an autodocumenting tool attempting to generate documentation for classes and functions that were intended to not be used by the end-user. Haddock (Haskell autodocumentor) strongly enforces this by not generating documentation for any function that the module doesn’t export. Sphinx (Python autodocumentor) will ignore by default functions prefixed with an underscore. People documenting Java, which tends to need a lot of classes, should think carefully about which classes they actually want people to use.

Final thoughts. The word “autogenerated documentation” is a misnomer: there is no automatic generation of documentation. Rather, the autodocumentor should be treated as a valuable documentation building tool that lets you get the benefits of cohesive code and comments, as well as formatting, interlinking and more.

Principles of FFI API design

This is part three of a six part tutorial series on c2hs. Today, we take a step back from the nitty gritty details of FFI bindings and talk about more general design principles for your library.

On the one hand, writing an FFI binding can little more than churning out the glue code to let you write “C in Haskell,” the API of your library left at the whimsy of the original library author. On the other hand, you can aspire to make your interface indistinguishable from what someone might have written in pure Haskell, introducing modest innovation of your own to encode informally documented invariants from the C code into the type system.

Overall design. Larger bindings benefit from being divided into two layers: a low-level binding and a higher-level user-friendly binding. There is a large amount of code necessary to make C functions available to call by Haskell, and the obvious thing to do with it is to stow it away in its own namespace, frequently with Internal somewhere in the name.

Low-level design. In the low-level binding, you should organize your foreign imports the way the C header files were organized. Keep the names similar. While it won’t be possible to have identical C function names and Haskell function names—C functions are allowed to begin with capital letters and Haskell functions are not (there is an opposite situation for types and data constructors)—you can still adopt a consistent transformation. c2hs, by default, converts Foo_BarBaz in C to fooBarBaz; that is, words after underscores are capitalized, the first letter is uncapitalized, and underscores are removed.

There is room to improve upon the original API, however. The rule of thumb is, if you can make a non-invasive/local change that improves safety or usability, do it. These include:

• Marshalling vanilla C values (e.g. int, float and even char*, if it’s a null terminated string) into their natural Haskell forms (Int, Float and String). Care should be taken, as the native Haskell types lose precision to their C counterparts, and thus should ensure that the application doesn’t need to squeeze out every last higher bit (e.g. with a bit field). 80% of the time, a lossy transformation is acceptable,
• Converting int into Bool from some naming convention (perhaps boolean values are prefixed with f for flag),
• Putting malloc’d pointers into memory management with foreign pointers. This advice is worth repeating: Haskell has memory management, and it would be criminal not to use it as soon as possible. Plus, you don’t have to write an explicit Haskell deallocation function.
• Converting functions that initialize some memory space (set_struct_default) into pure versions (structDefault) with unsafePerformIO, alloca and peek. Note that you should do this in conjunction with the appropriate Storable instance to marshal the C struct into a persistent Haskell datatype.
• Marshalling more complex C values (arrays, mostly) into Haskell lists, assuming that bounds information is consistent and available locally.

We will discuss these techniques in more detail in the coming posts, since this is precisely where c2hs is used the most.

It’s useful to know when not to simplify: certain types of libraries may have highly efficient in-memory representations for large structures; marshalling them to and from Haskell is wasteful. Poorly written C code may also you hand you arrays for which you cannot find easily their length; deferring their marshalling to the higher-level interface may be a better idea. Decide which structures you are explicitly not going to marshal and enforce this across the board. My preference is to marshal flat structs that contain no pointers, and leave everything else be.

High-level design. While there are certainly exotic computational strcutures like arrows, applicative functors and comonads which can be useful in certain domains, we will restrict our discussion to the go-to tools of the Haskell programmer: pure functions and monads.

• Pure functions. Transforming the mutable substrate that C is built upon into the more precious stuff of pure functions and persistent data structures is a tricky task, rife with unsafePerformIO. In particular, just because a C function published purpose seems to not involve mutating anything, it may perform some shared state change or rebalance the input datastructure or printf on failure, and you have to account for it. Unless your documentation is extremely good, you will need to do source diving to manually verify invariants.

  There will be some number of functions that are referentially transparent; these are a precious commodity and can be transformed into pure functions with ease. From there, you will need to make decisions about how a library can and cannot be used. A set of internal state transformation functions may not be amenable to a pure treatment, but perhaps a function that orchestrates them together leaks no shared state. Data structures that were intended to be mutated can be translated into immutable Haskell versions, or frozen by your API, which exposes no method of mutating them (well, no method not prefixed by unsafe) to the end user.

• Monads. First an obvious choice: are you going to chuck all of your functions into the IO monad, or give the user a more restrictive monad stack that performs IO under the hood, but only lets the user perform operations relevant to your library. (This is not difficult to do: you newtype your monad stack and simply fail to export the constructor and omit the MonadIO instance.) :

  newtype MyMonad a = MyMonad { unMyMonad :: ReaderT Env IO a }
    deriving (MonadReader Env, Monad, Functor)
  

  You will be frequently passing around hidden state in the form of pointers. These should be wrapped in newtypes and not exposed to the end-user. Sometimes, these will be pointers to pointers, in the case of libraries that have parameters **ppFoo, which take your pointer and rewrite it to point somewhere else, subsuming the original object. :

  newtype OpaqueStruct = OpaqueStruct { unOpaqueStruct :: ForeignPtr (Ptr CStruct) }
  

  Shared state means that thread safety also comes into play. Haskell is an incredibly thread friendly language, and it’s easy, as a user of a library, to assume that any given library is thread-safe. This is an admirable goal for any library writer to strive for, but one made much harder by your dependence on a C-based library. Fortunately, Haskell provides primitives that make thread safety much easier, in particular MVar, TVar and TMVar; simply store your pointer in this shared variable and don’t let anyone else have the pointer. Extra care is necessary for complex pointer graphs; be sure that if you have an MVar representing a lock for some shared state, there isn’t a pointer squirreled away elsewhere that some other C code will just use. And of course, if you have persistent structures, maintaining consistency is trivial. :

  withMVar (unOpaqueStruct o) $ \o_ ->
    withForeignPtr o_ $ \p ->
      -- peek ’n poke the piggy, erm, pointer
  

  One particularly good technique for preventing end-users from smuggling pointers out of your beautiful thread-safe sections is the application of a rank-2 type, akin to the ST monad. The basic premise is that you write a function with the type (forall s. m s a) -> a. The forall constraint on the argument to this function requires the result a to not contain s anywhere in its type (for the more technically inclined, the forall is a statement that I should be able to place any s in the statement and have it be valid. If some specific s' was in a, it would be only valid if I set my s to that s', and no other s). Thus, you simply add a phantom type variable s to any datatype you don’t want smuggled out of your monad, and the type system will do the rest. Monadic regions builds on this basic concept, giving it composability (region polymorphism). :

  newtype LockedMonad i a = LockedMonad { unLockedMonad :: ReaderT Env IO a }
    deriving (MonadReader Env, Monad, Functor)
  runLockedMonad :: (forall i. LockedMonad i a) -> IO a
  runLockedMonad m = runReaderT (unLockedMonad m) =<< newEnv
  data LockedData i a = LockedData a
  

We will not be discussing these ideas as part of c2hs; use of the preprocessor is mostly distinct from this part of the design process. However, it is quite an interesting topic in its own right!

Next time. First steps in c2hs