prose :: and :: conz

Well Mr. Static Typing… this really sucks

Yeah, I’ve been singing the praises of static typing for a few weeks now, but it’s time for me to admit some of the faults I’ve run into even in a well thought out static typing system like in Scala. While I’ve been wholly converted to the static typing camp, these problems I’ve found bother me. The one I’m highlighting here emerged while working on SNMP4S.

One of the key features of SNMP4S is type-safety derived from compiling the MIBs which define the SNMP interface. Think of it as a WSDL for an SNMP interface. Each object that you can manipulate via SNMP has a MAX-ACCESS property (Read-Only, Read-Write, Not-Accessible, etc) and a type (typically OCTET-STRING, INTEGER32, and enumerations), both of which I have encoded as SNMP4S types. Then with each object, you can get or set the current state. Since these operations are I/O bound on the network, you want to batch all of the gets or sets into a single packet. Hence, you need get() and set() to accept a variable number of arguments. Initially you may think that you can implement this by having get/set accept a sequence of objects such as this:

def get[A <: Readable, T]
  (objs:Seq[DataObject[A, T]])
  (implicit m:Manifest[T]):
  Either[SnmpError,Seq[Either[SnmpError,T]]] = ...

Unfortunately, this won’t work for all cases. It restricts all of the arguments to have the same access A and type T. You won’t be able to mix a read-only object of type String with a read-write object of type Int. To do that, you must have two explicit arguments with the type annotations for get():

def get[A1 <: Readable, T1, A2 <: Readable, T2]
  (obj1:DataObject[A1, T1], obj2:DataObject[A2, T2])
  (implicit m1:Manifest[T1], m2:Manifest[T2]):
  Either[SnmpError,(Either[SnmpError,T1], Either[SnmpError,T2])] = ...

That does work for the case of getting two objects, but what about three? Four? N? To support that, we must have a function for each number of arguments we want to support. Hence, it’s not possible to support an arbitrary number of objects. We have to accept an upper bound. Picking the upper bound is easy, because it’s 22. So I have to write 22 functions for each operation. Wow, that sucks. Limited functionality AND copy/pasting.

The good news is there is sbt-boilerplate to code this up for me. You just define the base case and it generates all 22 cases from it. This is what my get() declaration looks like in the boilerplate’s template language:

[#def get[[#A1 <: Readable, T1#]]
  ([#obj1:DataObject[A1, T1]#])
  (implicit [#m1:Manifest[T1]#]):
  Either[SnmpError,([#Either[SnmpError,T1]#])] = ... #]

And it works beautifully! Each snippet surrounded by [##] gets repeated 22 times, generating my functions in a scala source file. Write some tests… Everything is working well. Generate the documentation and, uh-oh… 22 functions named get which all do basically the same thing. The scaladoc page is bloated with nasty entries like this:

get[A1 <: Readable, T1, A2 <: Readable, T2, A3 <: Readable, T3, A4 <: Readable, T4](objs: (DataObject[A1, T1], DataObject[A2, T2], DataObject[A3, T3], DataObject[A4, T4]))(implicit m1: Manifest[T1], m2: Manifest[T2], m3: Manifest[T3], m4: Manifest[T4]): Either[SnmpError, (Either[SnmpError, T1], Either[SnmpError, T2], Either[SnmpError, T3], Either[SnmpError, T4])]

That’s just the case where n = 4! You can imagine how ugly the 22-arg one is. Add to that the fact that there are 22 of these. Good luck finding that now-obscured function walk(). Hopefully someone will tell me how to exclude these from scaladoc generation. Otherwise, this critical documentation will be very painful for potential users of my DSL. For now, I’ve decided to not worry about the documentation clutter. Hopefully my published examples will keep folks on track.

I used boilerplate to produce my get() functions as described above, as well as my set() methods. Then I decided it could be handy if get() could also accept a tuple of objects. (I won’t get into why I thought that would be handy.) Well then I hit another problem. This time, it’s a bit more problematic than unusable documentation. Check out how long it took to compile and run my tests:

[info] Passed: : Total 21, Failed 0, Errors 0, Passed 21, Skipped 0
[success] Total time: 904 s, completed Jun 6, 2013 1:41:40 PM

Yes, you see that correctly. 904 seconds, or 15 minutes and 4 seconds.

Prior to adding the get(TupleN), the time needed to compile my test suite was getting slow, but not THAT slow. After removing the tuples:

[info] Passed: : Total 20, Failed 0, Errors 0, Passed 20, Skipped 0
[success] Total time: 44 s, completed Jun 6, 2013 2:19:37 PM

Not only does the compiler take an unacceptable amount of time to run, my eclipse IDE hangs up for long periods of time while trying to save a file. I greatly appreciate the extra work that the Scala compiler does for me, say compared to javac, but this is a sure way to have an unusable library. Needless to say, I threw that tuple version of get() out.

Cluttered, useless documentation. Slow compilation. Frozen IDE. Stupid static type checking.

Now, it’s possible that shapeless is the solution. I was able to get some toy code working with this library. Unfortunately, I found myself wrapped around the axle of Scala higher kinded types. After some discussion, I had decided for now I’d just use sbt-boilerplate. Too bad that led me into these problems. I’ll probably revert back to shapeless eventually in hopes of making all of this better. For now, I want to hammer out some real value in this DSL before I make it pretty.

So there ya go. The good and the bad. Prose and conz… Yes Mom, I’ll get around to writing about lambdas before I drop this static typing discussion…

Olde Comments
  1. Aleh says:

    hm i’m curious why did you choose this way instead of just composing Eithers (or better yet, Validations if you want to accumulate failures)
    e.g. for { a <- get(A).right; b <- get(B).right }

    in my experience overloading is a sign of a missing abstraction

    and btw methods in scala do not have 22 parameters limitation (only functions)

  2. Aleh says:

    yeah batching presents an api challenge, you may want to look at scala-machines for a way of decoupling of query definition from actual query execution

  3. barnesjd says:

    Thanks for the replies, Aleh. I’ll take a look at Validations. Perhaps that will make the return type less crowded. I’ll try to explain what I hope to accomplish by showing a code example: (Realistically, I expect my users to utilize type inference, but just to be clear)

    val res:Either[SnmpError,(Either[SnmpError, Int], Either[SnmpError, String]])] = get(IfIndex(1), IfDescr(1))

    It’s possible that the entire request fails, hence the res being an Either. It’s also possible that one object fails, but another doesn’t, hence the Either for each object.

    Without using the 1-22 overloads (btw, thanks for the clarification on that. Unfortunately for me, I’m still limited because I’m using tuples in the result), I don’t see a good way for IfIndex(1) which is typed with [Int] and IfDescr(1) which is typed with [String] to cause the result to be typed correctly.

    I’ll take a look at scala-machines. I had played with it previously for some other project, but didn’t know it had an application regarding queries.

  4. Batching a large number of requests of different types and receiving a batched response made up of the corresponding types means you need a big type to hold all that type information. But using recursion that can be quite manageable.

    If you create a composition function (Request[A], Request[B]) => Request[(A, B)], and a corresponding Response[(A, B)] => (Response[A], Response[B]), you will get batchable and type safe requests and responses. You would apply these many times to create batches larger than two. Nested Tuple2s are easier to automate and compose than TupleNs. You could also go in the direction of using the shapeless HList, which is similarly nested.

    Sorry if this is a little sketchy. I can flesh it out a bit further if you’d like,

    • barnesjd says:

      That sounds very promising. I’ll poke around with it tonight. I would greatly appreciate you fleshing it out a little more or pointing me at some references that can get me on track. Thanks for reading and taking the time to comment.

    • barnesjd says:

      I’m reading this blog series. Perhaps this is what I’m missing in my understanding of the type system.

      • Funny, that I’m actually enjoying this blog-series. Because a while ago I used shapeless to a have composable/chaining interceptors in Play2; and I used it as an esoteric lib ^^.
        My post is still here (quite outdated but the underlying ideas remain ;))

        I’m interesting with a follow up of your use case using TLP!

  5. Yes the above link seems quite relevant.

    I’ll started writing something more precise, will try to send it soon.

  6. Here’s what I can up with:

    I dropped Response altogether. The request is parameterized by the type contained in the response. It illustrates type safe batches of requests. Each contained request has its return type, which could allow for errors, and the request batch can also generate its errors.

    I’m not yet very familiar with SNMP, (although I may well be working with it in the not too distant future) so I’ve simplified a bunch of things. It is likely my suggestion would need significant adjustments to make it applicable to SNMP4S.

  7. I am doing some NMS work at moment. I can’t say much about it. I’ll definitely keep an eye on SNMP4S developments!

    • barnesjd says:

      Awesome! I suppose we’re just missing one another. Today was my last day in telecom for now, developing an NMS at ADTRAN.

  8. […] also experience frustration with both. Recently I posted about how sometimes static typing does in fact suck (blogger’s note: after some very helpful comments, I was able to learn that it was the […]

  9. […] week I lamented some flaws I had run into with Scala’s static typing while working on SNMP4S. Thanks to the venerable […]

Tagged with: scala (41), functional-programming (31), static-typing (16), snmp4s (3)