Matt Roberts

Automatic type class derivation with shapeless - Part Three

Thu, 20 Apr 2017 07:00:00 +0000

This post is part of a series. You might want to read Part One and Part Two first if you haven’t already.

In Part One I explained that I wanted to fall back to case class defaults where possible. For example, given the following case class:

case class SimpleArguments(alpha: String = "alpha", beta: Int, charlie: Boolean)

And run-time arguments:

my-app --beta 1 --charlie

I would like the parser to return:

SimpleArguments(alpha = "alpha", beta = 1, charlie = true)

If you read Part Two then you might be thinking: maybe there is a LabelledGenericWithDefaults that we can use. Unfortunately, there isn’t. That makes sense to me because they feel like separate concerns. There is however a Default. The ‘trick’ we’re going to use here is to have an underlying parser that accepts two arguments (one being a HList of defaults represented as Options) and then convert it to a parser that requires one argument at the last moment.

Let’s start with the UnderlyingParser (I explain how it works below):

trait UnderlyingParser[A, B] {
  def parse(args: List[String], defaults: B): A
}

object UnderlyingParser {
  import shapeless.LabelledGeneric
  import shapeless.{HList, HNil, ::}
  import shapeless.Lazy
  import shapeless.Witness
  import shapeless.labelled.FieldType
  import shapeless.labelled.field
  import shapeless.Default

  def create[A, B](thunk: (List[String], B) => A): UnderlyingParser[A, B] = {
    new UnderlyingParser[A, B] {
      def parse(args: List[String], defaults: B): A = thunk(args, defaults)
    }
  }

  implicit def hlistParser[K <: Symbol, H, T <: HList, TD <: HList](
    implicit
    witness: Witness.Aux[K],
    hParser: Lazy[UnderlyingParser[FieldType[K, H], Option[H]]],
    tParser: UnderlyingParser[T, TD]
  ): UnderlyingParser[FieldType[K, H] :: T, Option[H] :: TD] = {
    create { (args, defaults) =>
      val hv = hParser.value.parse(args, defaults.head)
      val tv = tParser.parse(args, defaults.tail)
      hv :: tv
    }
  }

  implicit def stringParser[K <: Symbol](
    implicit
    witness: Witness.Aux[K]
  ): UnderlyingParser[FieldType[K, String], Option[String]] = {
    val name = witness.value.name
    create { (args, defaultArg) =>
      val providedArg = getArgFor(args, name)
      val arg = providedArg.getOrElse(
        defaultArg.getOrElse(
          // TODO: Use Either instead of throwing an exception.
          throw new IllegalArgumentException(s"Missing argument $name")
        )
      )
      field[K](arg)
    }
  }

  implicit def intParser[K <: Symbol](
    implicit
    witness: Witness.Aux[K]
  ): UnderlyingParser[FieldType[K, Int], Option[Int]] = {
    val name = witness.value.name
    create { (args, defaultArg) =>
      val providedArg = getArgFor(args, name).map(_.toInt)
      val arg = providedArg.getOrElse(
        defaultArg.getOrElse(
          // TODO: Use Either instead of throwing an exception.
          throw new IllegalArgumentException(s"Missing argument $name")
        )
      )
      field[K](arg)
    }
  }

  implicit def booleanParser[K <: Symbol](
    implicit
    witness: Witness.Aux[K]
  ): UnderlyingParser[FieldType[K, Boolean], Option[Boolean]] = {
    val name = witness.value.name
    create { (args, default) =>
      val arg = args.find(a => a == s"--$name").isDefined
      field[K](arg)
    }
  }

  implicit val hnilParser: UnderlyingParser[HNil, HNil] = {
    create { (_, _) => HNil }
  }

  private def getArgFor(args: List[String], name: String): Option[String] = {
    val indexOfName = args.indexOf(s"--$name")
    val indexAfterName = indexOfName + 1
    if (indexOfName > -1 && args.isDefinedAt(indexAfterName)) {
      Some(args(indexAfterName))
    } else {
      None
    }
  }
}

You’ll notice there is no apply or genericParser method. This is because I decided that client code should not be able to instantiate an UnderlyingParser. It also means that there is only one particularly difficult method to wrap our heads around:

implicit def hlistParser[K <: Symbol, H, T <: HList, TD <: HList](
  implicit
  witness: Witness.Aux[K],
  hParser: Lazy[UnderlyingParser[FieldType[K, H], Option[H]]],
  tParser: UnderlyingParser[T, TD]
): UnderlyingParser[FieldType[K, H] :: T, Option[H] :: TD] = {
  create { (args, defaults) =>
    val hv = hParser.value.parse(args, defaults.head)
    val tv = tParser.parse(args, defaults.tail)
    hv :: tv
  }
}

Whilst it looks a little daunting the above hlistParser has only one extra type parameter when compared to the method in Part Two: TD. TD is used to represent the tail of the HList that contains default parameters. The UnderlyingParser[FieldType[K, H], Option[H]] type is also more complicated, but again there is just one extra parameter when compared to Part Two: Option[H]. This is the head of the HList that contains default parameters. You can see this again in the method return type: UnderlyingParser[FieldType[K, H] :: T, Option[H] :: TD]. If I were to read this allowed I would say:

The hlistParser method returns a generic UnderlyingParser where both type parameters are HLists. The head of the first HList is a generic FieldType and the tail is some T that is also a HList. The head of the second HList is a generic Option and the tail is some TD that is also a HList.

We, as humans, implicitly know that the tail of each HList will have to contain field types and options. If we wanted to we could make use of shapeless’ HList constraints to codify this in the method:

import shapeless.UnaryTCConstraint._
import shapeless.LUBConstraint._

implicit def hlistParser[
  K <: Symbol,
  H,
  T <: HList : <<:[FieldType[_, _]]#λ,
  TD <: HList : *->*[Option]#λ
](
  implicit
  witness: Witness.Aux[K],
  hParser: Lazy[UnderlyingParser[FieldType[K, H], Option[H]]],
  tParser: UnderlyingParser[T, TD]
): UnderlyingParser[FieldType[K, H] :: T, Option[H] :: TD] = {
  create { (args, defaults) =>
    val hv = hParser.value.parse(args, defaults.head)
    val tv = tParser.parse(args, defaults.tail)
    hv :: tv
  }
}

The above explicitly states that T must be a HList containing FieldTypes and TD must be a HList of Option. This is good because it explicitly states what we know to be implicitly the case. The downside is we have to write more code and a lot of people find it hard to read. When I wrote Claper I didn’t know about HList constraints and didn’t use them, but hopefully they’ll be useful to someone.

On an aside, shapeless also has a KeyConstraint that, from the documentation, sounds like it might have been useful:

Type class witnessing that every element of L is of the form FieldType[K, V] where K is an element of M.

I’m afraid you’ll have to have a dig around the shapeless code (there are tests!) if you want to find out more abut that though.

Back to parsing, so we have our UnderlyingParser and we’re reasonably confident that the Scala compiler will be able to complete it’s derivation, so let’s make use of it:

trait DefaultedParser[A] {
  def parse(args: List[String]): A
}

object DefaultedParser {
  import shapeless.LabelledGeneric
  import shapeless.{HList, HNil, ::}
  import shapeless.Lazy
  import shapeless.Witness
  import shapeless.labelled.FieldType
  import shapeless.labelled.field
  import shapeless.Default

  def apply[A](
    implicit
    st: Lazy[DefaultedParser[A]]
  ): DefaultedParser[A] = st.value

  def create[A](thunk: List[String] => A): DefaultedParser[A] = {
    new DefaultedParser[A] {
      def parse(args: List[String]): A = thunk(args)
    }
  }

  implicit def genericParser[A, R <: HList, D <: HList](
    implicit
    defaults: Default.AsOptions.Aux[A, D],
    generic: LabelledGeneric.Aux[A, R],
    parser: Lazy[UnderlyingParser[R, D]]
  ): DefaultedParser[A] = {
    create { args => generic.from(parser.value.parse(args, defaults())) }
  }
}

The interesting method here is genericParser it states. Default.AsOptions.Aux[A, D] is provided by the shapeless.Default import (and a macro in shapeless). LabelledGeneric.Aux[A, R] is provided by shapeless.LabelledGeneric import (and another macro). Lazy[UnderlyingParser[R, D]] is provided hlistParser in the UnderlyingParser object (and some hard work by the compiler). The args and the applied defaults are then passed to the UnderlyingParser’s parse method which returns a HList from which we can construct an A using generic.

Here’s a test showing that it all works:

import org.scalatest.{MustMatchers, FlatSpec}

case class SimpleArguments(alpha: String = "alpha", beta: Int, charlie: Boolean)

class DefaultedParserSpec extends FlatSpec with MustMatchers {
  "DefaultedParser::apply" must "derive a parser for SimpleArguments" in {
    val args = List("--beta", "1", "--charlie")
    val parsed = DefaultedParser[SimpleArguments].parse(args)
    parsed must be (SimpleArguments("alpha", 1, true))
  }
}

If you find yourself suffering from type blindness (I sometimes do) then I recommend you read Part One and Part Two and / or diff the code locally in your favourite text editor.

The code for Part Three is available on Github.

Automatic type class derivation with shapeless - Part Two

Tue, 18 Apr 2017 19:00:00 +0000

This post is part of a series. You might want to read Part One first if you haven’t already. And when you’re done here you might want to read Part Three.

In Part One I explained that I wanted a library that would parse *nix style options:

my-app --alpha foo --beta 1 --charlie

Into this:

SimpleArguments(alpha = "foo", beta = 1, charlie = true)

I also stated that I wanted to fall-back to case class defaults, but let’s ignore that requirement for now. Don’t panic! I will solve that in Part Three. As before, here’s my build.sbt file with shapeless and scalatest:

scalaVersion := "2.12.1"

libraryDependencies ++= Seq(
  "com.chuusai" %% "shapeless" % "2.3.2",
  "org.scalatest" %% "scalatest" % "3.0.1" % "test"
)

And here’s my second attempt at a parser (I explain how it works below):

trait LabelledParser[A] {
  def parse(args: List[String]): A
}

object LabelledParser {
  import shapeless.LabelledGeneric
  import shapeless.{HList, HNil, ::}
  import shapeless.Lazy
  import shapeless.Witness
  import shapeless.labelled.FieldType
  import shapeless.labelled.field

  private def create[A](thunk: List[String] => A): LabelledParser[A] = {
    new LabelledParser[A] {
      def parse(args: List[String]): A = thunk(args)
    }
  }

  def apply[A](
    implicit
    st: Lazy[LabelledParser[A]]
  ): LabelledParser[A] = st.value

  implicit def genericParser[A, R <: HList](
    implicit
    generic: LabelledGeneric.Aux[A, R],
    parser: Lazy[LabelledParser[R]]
  ): LabelledParser[A] = {
    create(args => generic.from(parser.value.parse(args)))
  }

  implicit def hlistParser[K <: Symbol, H, T <: HList](
    implicit
    hParser: Lazy[LabelledParser[FieldType[K, H]]],
    tParser: LabelledParser[T]
  ): LabelledParser[FieldType[K, H] :: T] = {
    create { args =>
      val hv = hParser.value.parse(args)
      val tv = tParser.parse(args)
      hv :: tv
    }
  }

  implicit def stringParser[K <: Symbol](
    implicit
    witness: Witness.Aux[K]
  ): LabelledParser[FieldType[K, String]] = {
    val name = witness.value.name
    create { args =>
      val arg = args.dropWhile(a => a != s"--$name").tail.head
      field[K](arg)
    }
  }

  implicit def intParser[K <: Symbol](
    implicit
    witness: Witness.Aux[K]
  ): LabelledParser[FieldType[K, Int]] = {
    val name = witness.value.name
    create { args =>
      val arg = args.dropWhile(a => a != s"--$name").tail.head.toInt
      field[K](arg)
    }
  }

  implicit def booleanParser[K <: Symbol](
    implicit
    witness: Witness.Aux[K]
  ): LabelledParser[FieldType[K, Boolean]] = {
    val name = witness.value.name
    create { args =>
      val arg = args.find(a => a == s"--$name").isDefined
      field[K](arg)
    }
  }

  implicit val hnilParser: LabelledParser[HNil] = {
    create(args => HNil)
  }
}

And a test that demonstrates the automatic type class derivation:

import org.scalatest.{MustMatchers, FlatSpec}

case class SimpleArguments(alpha: String, beta: Int, charlie: Boolean)

class LabelledParserSpec extends FlatSpec with MustMatchers {
  "LabelledParser::apply" must "derive a parser for SimpleArguments" in {
    val args = List("--alpha", "a", "--beta", "1", "--charlie")
    val parsed = LabelledParser[SimpleArguments].parse(args)
    parsed must be (SimpleArguments("a", 1, true))
  }
}

Let’s start with the same methods as before, namely apply, genericParser and hlistParser:

def apply[A](
  implicit
  st: Lazy[LabelledParser[A]]
): LabelledParser[A] = st.value

implicit def genericParser[A, R <: HList](
  implicit
  generic: LabelledGeneric.Aux[A, R],
  parser: Lazy[LabelledParser[R]]
): LabelledParser[A] = {
  create(args => generic.from(parser.value.parse(args)))
}

implicit def hlistParser[K <: Symbol, H, T <: HList](
  implicit
  hParser: Lazy[LabelledParser[FieldType[K, H]]],
  tParser: LabelledParser[T]
): LabelledParser[FieldType[K, H] :: T] = {
  create(args => hParser.value.parse(args) :: tParser.parse(args))
}

The apply method here looks very similar to the apply method defined in Part One except we’re asking the compiler to look for a LabelledParser for some A now. Things start to look a little different when we examine the implicit parameters for genericParser. In particular, the compiler is asked to find a LabelledGeneric.Aux[A, R] instead of a Generic.Aux[A, R]. Other than that, however, the method declaration and definition look reasonably similar. What is the difference between Generic and LabelledGeneric? From the documentation:

LabelledGeneric is similar to Generic, but includes information about field names or class names in addition to the raw structure.

From that we can guess that the HList R in the genericParser method here is not the same as in Part One. This is because it must include information about field names in addition to the raw structure. We can check this intuition by looking at the return type of hlistParser which, as expected, states that the HList is made up of FieldTypes.

Let’s compare the above definition of hlistParser with the one from Part One. First, the type parameters H and T are the same as in Part One (except the latter will be a HList of FieldTypes). K is new and used to state that the field name of the FieldType must be a Symbol. Second, the Scala compiler now needs to find a head parser for FieldType[K, H] and a tail parser for T. This tells us that the definitions for primitive types e.g. String, Int, Boolean must also be different form their Part One counterparts (and if you read the full code you will see they are). Third, and finally, the tail parser now receives the same args as the head parser (as opposed to just the tail). This makes sense because the arguments could now be in any order.

Lets look at one of the new primitive type definitions. Here is the String parser definition:

implicit def stringParser[K <: Symbol](
  implicit
  witness: Witness.Aux[K]
): LabelledParser[FieldType[K, String]] = {
  val name = witness.value.name
  create { args =>
    val arg = args.dropWhile(a => a != s"--$name").tail.head
    field[K](arg)
  }
}

The Witness is used to obtain the field name. The documentation for LabelledGeneric explains that a Witness is required, but I’m still not sure why Miles decided to separate them out (I’m sure there is a good reason):

Note that the representation does not include the labels! The labels are actually encoded in the generic type representation using Witness types.

In comparison, the field method is relatively straight forward and lets us return a FieldType[K, String] instead of a String. This pattern can and is (in the code above) repeated for each primitive type allowing the compiler to complete it’s derivation.

If you find yourself struggling with the type definitions then I recommend you read Part One and / or diff the code locally in your favourite text editor. In Part Three I show how to use Default to retrieve case class default values at compile time.

The code for Part Two is available on Github.

Automatic type class derivation with shapeless - Part One

Sun, 16 Apr 2017 10:00:00 +0000

This post is part of a series. You might want to read Part Two and Part Three when you’re done here.

I recently found myself needing to parse command line arguments in Scala. I discovered scopt and scallop, but felt they required quite a lot of boiler plate. What I wanted was a library that would take a case class and automatically derive a parser for me at compile time. Ideally that library would parse *nix style options e.g. my-app --foo bar, fall-back to defaults defined by the case class for missing options and return (not throw) an error when that failed. I made claper to do just this and decided to write about how I did it here.

I started by trying to solve for a simpler problem because I thought solving all of the above with my first try was a little ambitious to start out with. So Instead I tried to convert this:

my-app foo 1 true

Into this:

SimpleArguments(alpha = "foo", beta = 1, charlie = true)

Which is much easier because the parameters are in order and I don’t need to know the field names of the case class. I also ignored the issue of defaults. Don’t panic! I did solve for my actual use case and I’ll cover how in parts two and three.

I decided to use shapeless because I find the Scala macros api clumsy and it is still considered experimental. The former is really a matter of personal taste, but the latter means as a library maintainer your code can be broken by new versions of Scala. This makes supporting old, current and future versions of Scala difficult! I speak from the first-hand experience of maintaining slf4s - where I maintain a branch per Scala release.

Here’s what my build.sbt file looks like with shapeless and scalatest:

scalaVersion := "2.12.1"

libraryDependencies ++= Seq(
  "com.chuusai" %% "shapeless" % "2.3.2",
  "org.scalatest" %% "scalatest" % "3.0.1" % "test"
)

And here’s my first attempt at a parser (I explain how it works below):

trait Parser[A] {
  def parse(args: List[String]): A
}

object Parser {
  import shapeless.Generic
  import shapeless.{HList, HNil, ::}
  import shapeless.Lazy

  private def create[A](thunk: List[String] => A): Parser[A] = {
    new Parser[A] {
      def parse(args: List[String]): A = thunk(args)
    }
  }

  def apply[A](
    implicit
    st: Lazy[Parser[A]]
  ): Parser[A] = st.value

  implicit def genericParser[A, R <: HList](
    implicit
    generic: Generic.Aux[A, R],
    parser: Lazy[Parser[R]]
  ): Parser[A] = {
    create(args => generic.from(parser.value.parse(args)))
  }

  implicit def hlistParser[H, T <: HList](
    implicit
    hParser: Lazy[Parser[H]],
    tParser: Parser[T]
  ): Parser[H :: T] = {
    create(args => hParser.value.parse(args) :: tParser.parse(args.tail))
  }

  implicit val stringParser: Parser[String] = {
    create(args => args.head)
  }

  implicit val intParser: Parser[Int] = {
    create(args => args.head.toInt)
  }

  implicit val boolParser: Parser[Boolean] = {
    create(args => args.head.toBoolean)
  }

  implicit val hnilParser: Parser[HNil] = {
    create(args => HNil)
  }
}

And a test that demonstrates the automatic type class derivation:

import org.scalatest.{MustMatchers, FlatSpec}

case class SimpleArguments(alpha: String, beta: Int, charlie: Boolean)

class ParserSpec extends FlatSpec with MustMatchers {
  "Parser::apply" must "derive a parser for SimpleArguments" in {
    val args = List("a", "1", "true")
    val parsed = Parser[SimpleArguments].parse(args) // Magic.
    parsed must be (SimpleArguments("a", 1, true))
  }
}

Now, in my opinion, the real “magic” is here (I explain the trick below):

def apply[A](
  implicit
  st: Lazy[Parser[A]]
): Parser[A] = st.value

implicit def genericParser[A, R <: HList](
  implicit
  generic: Generic.Aux[A, R],
  parser: Lazy[Parser[R]]
): Parser[A] = {
  create(args => generic.from(parser.value.parse(args)))
}

implicit def hlistParser[H, T <: HList](
  implicit
  hParser: Lazy[Parser[H]],
  tParser: Parser[T]
): Parser[H :: T] = {
  create(args => hParser.value.parse(args) :: tParser.parse(args.tail))
}

The apply method causes the Scala compiler to search for an implicit Parser[A]. It finds the genericParser method and that in turn causes it to look for an implicit Generic.Aux[A, R] and Parser[R]. Where R is a HList that Generic can create an A from. It finds a Generic.Aux[A, R] thanks to the shapeless.Generic import (and a macro in the shapeless library). The Parser[R] requirement, as you may have guessed, is satisfied by the hlistParser method. The implicit Parser[H] is satisfied by the, rather mundane, implicit values that handle primitive types and the Parser[T] is handled recursively until the terminal HNil case is reached.

For the SimpleArguments example the compiler automatically derives the following for us:

import org.scalatest.{MustMatchers, FlatSpec}

class ParserSpec extends FlatSpec with MustMatchers {
  "Parser::apply" must "let us derive a parser for SimpleArguments" in {
    import Parser._
    import shapeless.Generic
    import shapeless.Lazy
    val args = List("a", "1", "true")
    val parser = Parser[SimpleArguments](
      genericParser(
        Generic[SimpleArguments],
        Lazy(hlistParser(
          Lazy(stringParser),
          hlistParser(
            Lazy(intParser),
            hlistParser(
              Lazy(boolParser),
              hnilParser
            )
          )
        ))
      )
    )
    val parsed = parser.parse(args)
    parsed must be (SimpleArguments("a", 1, true))
  }
}

You might be wondering why I haven’t mentioned Lazy or why I use it. First, the easy answer: I haven’t mentioned it because I think it hinders understanding and readability. As for why I use it, in short: it stops the compiler giving up it’s search for implicits too early. In more detail: the Scala compiler uses heuristics to make sure that it doesn’t get stuck in an infinite recursion when resolving implicits. For more complex data types, these heuristics tend to be too aggressive. The use of Lazy lets us workaround this issue.

The code for Part One is available on Github.

In Part Two I show how to use LabelledGeneric to retrieve case class field names at compile time and in Part Three I show how to use Default to, you guessed it, retrieve case class default values at compile time.

Genetic Programming in Scala

Fri, 09 Sep 2016 00:00:00 +0000

Introduction

I hope that one day software engineers will obsolete ourselves. That is, we will write an application capable of writing all other applications for us. Genetic programming (GP) is about just that: creating programs that solve problems based on expected outputs for given inputs. As a colleague of mine put it, you write the unit tests and the computer does the rest. Of course, both my colleague and I know we’re not quite there yet!

Most, if not all, of the algorithms implemented here are based on the excellent descriptions in A Field Guide to Genetic Programming. I’ve concentrated on readability and correctness as opposed to speed of execution or even reliability, so please excuse the lacklustre performance and the fact that tail recursion is not used in places. Also note that the algorithms implemented here are, to quote the authors, “the simplest thing that could possibly work”.

The snippets (those without file names) below should work in the SBT console of the project (or an IntelliJ worksheet). The project is available in my GitHub repository: mattroberts297/genetic-programming-in-scala.

Representing programs

There are three main steps in GP (2 and 3 are repeated):

Generate a population of programs;
Evaluate the fitness of those programs; and
Evolve a new, hopefully fitter, population of programs.

In order to do any of those things, however, we need to be able to represent programs in some way. This is normally done by means of an abstract syntax tree (AST). The AST considered here is handcrafted and only capable of representing simple polynomial expressions. There are, however, many tools capable of automatically generating ASTs from grammar files (my personal favourite being ANTLR). Here is the handcrafted AST:

model/Exp.scala

package model

sealed trait Exp

case class Con(value: Float) extends Exp

case class Var(name: Symbol) extends Exp

sealed trait BinOp extends Exp {
  def lhs: Exp
  def rhs: Exp
}

case class Add(lhs: Exp, rhs: Exp) extends BinOp

case class Sub(lhs: Exp, rhs: Exp) extends BinOp

case class Mul(lhs: Exp, rhs: Exp) extends BinOp

case class Div(lhs: Exp, rhs: Exp) extends BinOp

object Exp {
  import scala.util.Try
  def eval(exp: Exp, st: ST): Float = exp match {
    case Con(value) => value
    case Var(name) => st(name)
    case Add(lhs, rhs) => eval(lhs, st) + eval(rhs, st)
    case Sub(lhs, rhs) => eval(lhs, st) - eval(rhs, st)
    case Mul(lhs, rhs) => eval(lhs, st) * eval(rhs, st)
    case Div(lhs, rhs) => Try(eval(lhs, st) / eval(rhs, st)).getOrElse(1f)
  }
}

And a type alias from ST to Map[Symbol, Float] for the symbol table:

model/package.scala

import scala.collection.immutable.Map

package object model {
  type ST = Map[Symbol, Float]
}

There are a three design decisions in the above code worth discussing.

First, constants and variables are always of type Float. There are two main reasons for this: type safety and memory consumption. Creating a generic AST and accompanying symbol table(s) would be reasonably straight forward, but writing GP algorithms that operate on them in a type safe fashion would not be so simple.

Second, the eval method never throws java.lang.ArithmeticException: / by zero when performing division returning 1 instead. This ensures both type and evaluation safety and, again, makes it easier to write GP algorithms that operate on ASTs without encountering runtime exceptions.

Third, and finally, both Exp and BinOp are sealed traits. This makes pattern matching easier because the compiler will warn us about incomplete matches instead of leaving us at the mercy of runtime exceptions.

Example ASTs

In order to better understand what we are trying to achieve automatically it is useful to manually create some ASTs and plot the results of evaluating them for various values of x.

Second degree polynomial

Here is an AST for the second degree polynomial f(x) = x^2 - x - 2:

import model._

Sub(Sub(Mul(Var('x), Var('x)), Var('x)), Con(2))

It can be evaluated for various values of x like so:

import model._

val exp = Sub(Sub(Mul(Var('x), Var('x)), Var('x)), Con(2))
Exp.eval(exp, Map('x -> -3)) // res0: Float = 10
Exp.eval(exp, Map('x -> 0)) // res0: Float = -2
Exp.eval(exp, Map('x -> 3)) // res0: Float = 4

This plots shows the result of the expression for values of x between -3 and 3:

Third degree polynomial

Here is another, slightly more complex, AST for the third degree polynomial f(x) = x^3 / 4 + 3x^2 / 4 - 3x / 2 - 2:

import model._

Sub(Sub(Add(Div(Mul(Var('x),Mul(Var('x),Var('x))),Con(4)),Div(Mul(Con(3),Mul(Var('x),Var('x))),Con(4))),Div(Mul(Con(3),Var('x)),Con(2))),Con(2))

As before, the result for various values of x can be determined like so:

import model._

val exp = Sub(Sub(Add(Div(Mul(Var('x),Mul(Var('x),Var('x))),Con(4)),Div(Mul(Con(3),Mul(Var('x),Var('x))),Con(4))),Div(Mul(Con(3),Var('x)),Con(2))),Con(2))
Exp.eval(exp, Map('x -> -2)) // res1: Float = 2.0
Exp.eval(exp, Map('x -> 0)) // res2: Float = -2.0
Exp.eval(exp, Map('x -> 2)) // res3: Float = 0.0

And here is the plot:

From looking at both the AST and the chart you can hopefully see that the third degree polynomial will be more difficult to generate automatically than the second degree polynomial. These two expressions, one simpler and one more complex, will make good test cases for our GP solution.

Generating programs

In GP, the leafs of an AST, whether they be constants or variables, are referred to as the terminal set. The branches whether they be unary, binary or some other arity function are referred to as the function set. Using the AST defined above the following terminal and functional sets can be created:

import scala.collection.immutable._

val terminalSet = IndexedSeq(Var('x)) ++ 1f.to(5f, 1f).map(Con)
val functionSet = IndexedSeq(Add, Sub, Div, Mul)

With these sets it’s possible to generate an AST of some arbitrary depth depth. The simplest way to do this is to create a full tree. The algorithm is reasonably simple. If the depth has been reached then return a random terminal from the terminal set. Otherwise return a random function from the function set and use the result of the next recursion as its arguments. Here is a function that does that:

GP.scala

def full(
    depth: Int,
    functions: IndexedSeq[(Exp, Exp) => Exp],
    terminals: IndexedSeq[Exp]): Exp = {
  def loop(i: Int): Exp = {
    if (i == depth) {
      random(terminals)
    } else {
      random(functions)(loop(i + 1), loop(i + 1))
    }
  }
  loop(0)
}

def random[T](elements: IndexedSeq[T]): T = {
  elements(Random.nextInt(elements.length))
}

The full function resides in an object named GP. I omit the following “boiler plate” when discussing methods in that object:

GP.scala

import scala.annotation.tailrec
import scala.collection.immutable._
import scala.util.Random
import org.slf4s.Logging
import model._
object GP with Logging {
  // Functions go here.
}

Here are some example ASTs with depths 1, 2 and 3 generated with the full method:

The full method is simple and definitely works, but all the trees for a given depth have the exactly same number of leafs. This leads to a surprisingly uniform population. The grow method resolves this issue by randomly stopping the recursion:

GP.scala

def grow(
    depth: Int,
    functions: IndexedSeq[(Exp, Exp) => Exp],
    terminals: IndexedSeq[Exp]): Exp = {
  def randomStop: Boolean = {
    val tl = terminals.length.toFloat
    val fl = functions.length.toFloat
    random() < tl / (tl + fl)
  }
  def loop(i: Int): Exp = {
    if (i == depth || randomStop) {
      random(terminals)
    } else {
      random(functions)(loop(i + 1), loop(i + 1))
    }
  }
  loop(0)
}

def random(): Float = Random.nextFloat()

Here are three example ASTs all of depth 3 generated with grow method:

These two methods can be composed together to create the method rampHalfHalf. The idea behind this is to use grow for one half of the population and full for the other. Further, the method starts at depth 1 and ramps up to some maxDepth (as opposed to all trees being the same depth):

GP.scala

def rampHalfHalf(
    count: Int,
    maxDepth: Int,
    functions: IndexedSeq[(Exp, Exp) => Exp],
    terminals: IndexedSeq[Exp]): Set[Exp] = {
  @tailrec
  def loop(acc: Set[Exp], i: Int, depth: Int): Set[Exp] = {
    if(i == count) {
      acc
    } else {
      val tree = if (i % 2 == 0) {
        full(depth, functions, terminals)
      } else {
        grow(depth, functions, terminals)
      }
      val nextDepth = if (depth == maxDepth) 1 else depth + 1
      if (acc.contains(tree)) {
        loop(acc, i, nextDepth)
      } else {
        loop(acc + tree, i + 1, nextDepth)
      }
    }
  }
  loop(Set.empty, 0, 1)
}

The method above is a little finicky for several reasons. First, instead of comparing acc.size to count the value i is used. This is because the size method on set is inefficient. Second, if the acc set already contains a tree then i is not incremented. Third, and contrary to i, depth is always incremented unless the maxDepth has been reached and in this case it is set to 1. This is because, especially for large values of count, it is possible that all permutations of trees at a certain depth have been generated. If depth were not incremented then the recursion could be infinite.

In fact, even the above method would not terminate if count were high and maxDepth low, so be careful if you use it. If termination is important to you, and it probably is, then it’s best to accept duplicates and return a Seq instead.

With the above, it is now possible to create an initial population:

val count = 10000
val maxDepth = 5
val terminalSet = IndexedSeq(Var('x)) ++ 1f.to(5f, 1f).map(Con)
val functionSet = IndexedSeq(Add, Sub, Div, Mul)
val initial = rampHalfHalf(count, maxDepth, functionSet, terminalSet).toVector

Evaluating fitness

Evaluating fitness is easier than you might think. Given some input there is an expected output. Then evaluating fitness is just a matter of measuring the difference between the expected and actual outputs produced by evaluating a function. For anyone who has done unit testing with multiple test cases this should feel familiar. The next step, namely reducing those differences into one measure for all test cases, may feel decidely unfamiliar. For anyone with a machine learning background this will feel quite natural though.

In our example the inputs are stored in the symbol table. The symbol table therefore is the only input. We also know how to calculate the result of the polynomial expression, so we can calculate the expected values for a given set of inputs:

val cases = (-1f).to(1f, 0.05f).map(x => (Map('x -> x), pow(x, 2) - x - 2)).toMap

The type of cases is Map[ST, Float]. In other words, it maps symbol tables (inputs) to expected results. Now we need to use these cases to evaluate the fitness of a given tree. To do this, first we have to evaluate each case against the tree and store the absolute difference between the actual and expected results. Then we have to sum these differences to give one final figure for all cases:

GP.scala

def fitness(cases: Map[ST, Float])(tree: Exp): Float = {
  cases map { case (symbols, expected) =>
    val actual = Exp.eval(tree, symbols)
    Math.abs(expected - actual)
  } reduce { (a, b) =>
    a + b
  }
}

I have intentional written this using map and reduce. I could have also used foldLeft or foldRight and an accumulator. The map-reduce nature of the algorithm should feel natural to anyone from a machine learning background and hints at the fact that this can be trivially parallelised.

You may have already guessed this, but as the correctness of the generated trees improves the value returned by the fitness function will approach 0. When the tree gives the correct result for each case then the fitness function will return 0:

import scala.collection.immutable._
import model._
import GP._

def pow(a: Float, b: Float): Float = Math.pow(a, b).toFloat
val exp = Sub(Sub(Mul(Var('x), Var('x)), Var('x)), Con(2))
val cases = (-1f).to(1f, 0.05f).map(x => (Map('x -> x), pow(x, 2) - x - 2)).toMap
val f = fitness(cases)(exp)
f // 0.0

Evolving programs

The next step is to evolve the initial population into a new population. There are many algorithms to evolve programs (some haven’t even been discovered yet). I’ll look at three:

Mutation
Crossover
Replication

Mutation

Mutation will introduce a degree of randomness and brute force into the overall approach. Most GP enthusiasts would recommend keeping it’s use to a minimum (and I agree). That said, it is important because it can help bump a GP run out of a local optimum. The idea is to take a tree, pick a random subtree within it’s branches and replace it with another tree grown from the function and terminal sets:

GP.scala

  def mutate(
      functionSet: IndexedSeq[(Exp, Exp) => Exp],
      terminalSet: IndexedSeq[Exp],
      maxDepth: Int)(
      exp: Exp): Exp = {
    val target = random(exp)
    val replacement = grow(maxDepth, functionSet, terminalSet)
    replace(exp, target, replacement)
  }

Note the use of two new functions random and replace and one old function grow. The first function invokes collectAll to convert the Exp into an IndexedSeq[Exp] containing the root of the tree and all its subtrees using the generic collect function and a, not so partial, anonymous function. The result is then passsed to the previously defined random function that takes an IndexedSeq[T]:

GP.scala

  def random(tree: Exp): Exp = {
    random(collectAll(tree))
  }

  def collectAll(tree: Exp): IndexedSeq[Exp] = {
    collect(tree) { case e => e }
  }

  def collect[T](tree: Exp)(pf: PartialFunction[Exp, T]): IndexedSeq[T] = {
    def loop(subtree: Exp, acc: IndexedSeq[T]): IndexedSeq[T] = {
      val result = if (pf.isDefinedAt(subtree)) acc :+ pf(subtree) else acc
      subtree match {
        case v: Var => result
        case c: Con => result
        case o: BinOp => result ++ loop(o.lhs, acc) ++ loop(o.rhs, acc)
      }
    }
    loop(tree, IndexedSeq.empty[T])
  }

The second function is responsible for replacing some node target in the original expression exp with a replacement. The method uses reference equality instead of value equality because there may be many subtrees with value equality. This works because the mutate method picks an element from the original tree. The method performs a recursive descent through the tree. The first case in the match statement checks for reference equality and if it is found then returns the replacement. All other cases perform a copy. If the target does not exist then a copy of the original expression is returned:

GP.scala

  def replace(exp: Exp, target: Exp, replacement: Exp): Exp = {
    def repl(exp: Exp) = replace(exp, target, replacement)
    exp match {
      case exp: Exp if (exp.eq(target)) => replacement
      case Con(value) => Con(value)
      case Var(symbol) => Var(symbol)
      case Add(lhs, rhs) => Add(repl(lhs), repl(rhs))
      case Sub(lhs, rhs) => Sub(repl(lhs), repl(rhs))
      case Mul(lhs, rhs) => Mul(repl(lhs), repl(rhs))
      case Div(lhs, rhs) => Div(repl(lhs), repl(rhs))
    }
  }

Crossover and Tournament

Crossover

Crossover is responsible for breeding new trees. It does this by taking two parents (left and right) and selecting random subtree from each (replacement and target). The random selection is actually quite heavily biased in favour of operators to avoid pruning trees to aggressively. A copy of the right tree is made with the target replaced with the replacement:

GP.scala

  def crossover(left: Exp, right: Exp): Exp = {
    val replacement = biasedRandom(left)
    val target = biasedRandom(right)
    replace(right, target, replacement)
  }

  def biasedRandom(tree: Exp): Exp = {
    random(biasedCollect(tree))
  }

  def biasedCollect(tree: Exp): IndexedSeq[Exp] = {
    val ops = collectOps(tree)
    if (random() > 0.9 || ops.isEmpty) {
      collectTerminals(tree)
    } else {
      ops
    }
  }

  def collectOps(tree: Exp): IndexedSeq[Exp] = {
    collect(tree) { case o: BinOp => o }
  }

  def collectTerminals(tree: Exp): IndexedSeq[Exp] = {
    collect(tree) {
      case v: Var => v
      case c: Con => c
    }
  }

Tournament

Tournament is the process by which two parents are selected for crossover. The method takes two trees and their respective fitness values and returns the fittest tree:

GP.scala

def tournament(a: (Exp, Float), b: (Exp, Float)): Exp = {
  val (aExp, aFit) = a
  val (bExp, bFit) = b
  if (aFit < bFit) aExp else bExp
}

Replication

The last algorithm is very simple. Given a population, take a percentage of the fittest individuals and carry them across to the next population. The method to implement replication will have to take a population of trees as it’s input. Note that up until now each algorithm has taken one or two trees as inputs. With that in mind, here is the replicas function:

GP.scala

def replicas(
    treesAndFitness: IndexedSeq[(Exp, Float)],
    percent: Float = 0.19f,
    acc: Set[Exp] = Set.empty[Exp]): Set[Exp] = {
  val length = acc.size + (treesAndFitness.length.toFloat * percent).toInt
  val sortedTrees = treesAndFitness.
    sortBy { case (_, fitness) => fitness }.
    map { case (exp, _) => exp }
  @tailrec
  def loop(acc: Set[Exp], remaining: IndexedSeq[Exp]): Set[Exp] = {
    if (acc.size == length) {
      acc
    } else {
      loop(acc + remaining.head, remaining.tail)
    }
  }
  loop(acc, sortedTrees)
}

Crossover and mutation again

Before putting everything together, let’s create methods that wrap mutate and crossover, but take populations of trees as their arguments. Starting with crossovers:

GP.scala

def crossovers(
    treesAndFitness: IndexedSeq[(Exp, Float)],
    percent: Float = 0.8f,
    acc: Set[Exp]= Set.empty[Exp]): Set[Exp] = {
  val length = acc.size + (treesAndFitness.length.toFloat * percent).toInt
  @tailrec
  def loop(acc: Set[Exp]): Set[Exp] = {
    if (acc.size == length) {
      acc
    } else {
      loop(acc + crossover(
        tournament(random(treesAndFitness), random(treesAndFitness)),
        tournament(random(treesAndFitness), random(treesAndFitness))))
    }
  }
  loop(acc)
}

And finally the mutants:

GP.scala

def mutants(
    functionSet: IndexedSeq[(Exp, Exp) => Exp],
    terminalSet: IndexedSeq[Exp],
    maxDepth: Int)(
    trees: IndexedSeq[Exp],
    percent: Float = 0.01f,
    acc: Set[Exp]= Set.empty[Exp]): Set[Exp] = {
  val length = acc.size + (trees.length.toFloat * percent).toInt
  def mutate = GP.mutate(functionSet, terminalSet, maxDepth)_
  @tailrec
  def loop(acc: Set[Exp]): Set[Exp] = {
    if (acc.size == length) {
      acc
    } else {
      loop(acc + mutate(random(trees)))
    }
  }
  loop(acc)
}

One thing to note is that each method takes an accumulator. This avoids the situation whereby the population is compromised of trees that are exactly the same.

Finishing touches

The run method uses a tight loop to repeatedly evolve the initial population until some user-defined fitness criteria is reached:

GP.scala

def run(
    functionSet: IndexedSeq[(Exp, Exp) => Exp],
    terminalSet: IndexedSeq[Exp],
    cases: Map[ST, Float],
    fitness: Map[ST, Float] => Exp => Float,
    criteria: Float => Boolean,
    maxRuns: Int = 1000,
    maxDepth: Int = 6,
    populationSize: Int = 100000): Exp = {
  val initial = rampHalfHalf(populationSize, maxDepth, functionSet, terminalSet).toVector

  @tailrec
  def loop(run: Int, current: IndexedSeq[Exp]): Exp = {
    val treesAndFitness = current.map { tree => tree -> fitness(cases)(tree) }
    val (topTree, minFitness) = fittest(treesAndFitness)
    val mutants = GP.mutants(functionSet, terminalSet, maxDepth)_
    log.debug(s"run=${run}, minFitness=${minFitness}")
    if (criteria(minFitness)) {
      topTree
    } else {
      loop(
        run + 1,
        crossovers(
          treesAndFitness,
          0.8f,
          mutants(
            current,
            0.01f,
            replicas(
              treesAndFitness,
              0.19f,
              Set.empty))).toIndexedSeq)
    }
  }
  loop(1, initial)
}

def fittest(treesAndFitness: IndexedSeq[(Exp, Float)]): (Exp, Float) = {
  treesAndFitness.minBy { case (_, fitness) => fitness }
}

The Main application contains key parameters including:

populationSize The number of trees to generate.
maxDepth The maximum depth of the initial tree population.
terminalSet The terminals to use when constructing trees.
functionSet The functions to use when constructing trees.
cases The cases to use when evaluating the fitness of trees.
criteria The criteria used to decide when to stop evolving trees.

Main.scala

import scala.collection.immutable._
import org.slf4s.Logging
import model._

object Main extends App with Logging {
  import GP._
  val maxDepth = 6
  val terminalSet = IndexedSeq(Var('x)) ++ 1f.to(5f, 1f).map(Con)
  val functionSet = IndexedSeq(Add, Sub, Div, Mul)
  def pow(a: Float, b: Float): Float = Math.pow(a, b).toFloat
  val cases = (-3f).to(3f, 0.5f).map(x => (Map('x -> x), pow(x, 2) - x - 2)).toMap
  def criteria(fitness: Float): Boolean = fitness < 0.01f
  val fitTree = run(functionSet, terminalSet, cases, fitness, criteria, populationSize = 1000)
  log.info(s"Fittest tree: ${fitTree}")
  log.info("expected\t\tactual")
  cases.foreach { case (symbols, expected) =>
    log.info(s"${expected}\t${Exp.eval(fitTree, symbols)}")
  }
}

If you’re interested to see what a single evolution looks like then you can do that from the console or a worksheet:

import scala.collection.immutable._
import model._
import GP._

val count = 10
val maxDepth = 6
val terminalSet = IndexedSeq(Var('x)) ++ 1f.to(5f, 1f).map(Con)
val functionSet = IndexedSeq(Add, Sub, Div, Mul)
val initial = rampHalfHalf(count, maxDepth, functionSet, terminalSet).toVector
def pow(a: Float, b: Float): Float = Math.pow(a, b).toFloat
val cases = (-1f).to(1f, 0.05f).map(x => (Map('x -> x), pow(x, 2) - x - 2)).toMap
val treesAndFitness = initial.map { tree => tree -> fitness(cases)(tree) }
val mutants = GP.mutants(functionSet, terminalSet, maxDepth)_
val next = crossovers(
  treesAndFitness,
  0.8f,
  mutants(
    initial,
    0.01f,
    replicas(
      treesAndFitness,
      0.19f,
      Set.empty))).toIndexedSeq

Convergence

Second degree polynomial

The Main object and worksheet above contain cases for the second degree polynomial introduced way back at the beginning of this post. As the number of runs increase the aim is to converge on the actual solution. If we chart the fittest tree of each iteration then we can see this happening run by run:

Note that the actual line is obscured by the expected line i.e. it is a perfect match. Also, I labelled the charts as runs 1 through 5, but they are actually just runs where there was an improvement i.e. more runs were needed. That said, convergence seemed to happen pretty quickly all things considered i.e. sometimes sub second and no more than 2 seconds or 20 runs.

Third degree polynomial

To try and generate a third degree polynomial I just had to change my cases:

val cases = (-3f).to(3f, 0.5f).map(x => (Map('x -> x), pow(x,3) / 4 + 3 * pow(x, 2) / 4 - 3 * x / 2 - 2)).toMap

I suggested earlier that a third degree polynomial would be more difficult to generate automatically. In fact, what I found is that I would be lucky if the program converged at all. As time passed and the number of runs increased beyond a 100 then things started to slow down. Further, the fitness decrements became smaller and more infrequent. The give away for me was the runs slowing down. I took an educated guess and assumed the trees were getting too big and then the program became stuck in a local, but large optima (I could have proven this by printing out some trees).

Pruning

The solution, as with the other algorithms, is reasonably straight forward: prune the trees after each evolution. I’m sure this is probably in the book, but I either skimmed over it or forgot it. Here is the method I used:

def prune(
      terminalSet: IndexedSeq[Exp],
      maxDepth: Int)(
      tree: Exp): Exp = {
    def loop(subtree: Exp, depth: Int): Exp = {
      subtree match {
        case op: BinOp if (depth >= maxDepth) => random(terminalSet)
        case Con(value) => Con(value)
        case Var(symbol) => Var(symbol)
        case Add(lhs, rhs) => Add(loop(lhs, depth + 1), loop(rhs, depth + 1))
        case Sub(lhs, rhs) => Sub(loop(lhs, depth + 1), loop(rhs, depth + 1))
        case Mul(lhs, rhs) => Mul(loop(lhs, depth + 1), loop(rhs, depth + 1))
        case Div(lhs, rhs) => Div(loop(lhs, depth + 1), loop(rhs, depth + 1))
      }
    }
    loop(tree, 0)
  }

And here is how I tacked it onto the end of the loop (note that I was too lazy to write a version that operates on a population of trees):

loop(
  run + 1,
  crossovers(
    treesAndFitness,
    0.8f,
    mutants(
      current,
      0.01f,
      replicas(
        treesAndFitness,
        0.19f,
        Set.empty))).toIndexedSeq.map(prune), minFitness)

Doing this constrains the search space and means that each execution is quick i.e. no slow down over time. With pruning I found the number of runs varied from 50-200, but that the time to execute floated around the 2 second mark on average. Here are some more charts:

Final thoughts

Genetic programming seems like a brilliant way to search a constrained space for an optimum or approximate solution to some problem. Don’t forget the full source is available on GitHub at mattroberts297/genetic-programming-in-scala. If there is interest then I’ll do a follow up post showing how to run this on a Spark cluster.

ANTLR 4 Setup

Thu, 21 Apr 2016 08:35:00 +0000

I use macports so the following directories exist already:

/opt/local/lib
/opt/local/bin

The latter is on my PATH. You can check yours with echo $PATH.

sudo su -

mkdir /opt/local/lib/antlr4

curl --output /opt/local/lib/antlr4/antlr-4.5.3-complete.jar http://www.antlr.org/download/antlr-4.5.3-complete.jar

cat > /opt/local/bin/antlr << EOL
#!/bin/sh
java -cp "/opt/local/lib/antlr4/antlr-4.5.3-complete.jar:$CLASSPATH" org.antlr.v4.Tool $*
EOL
chmod a+x /opt/local/bin/antlr4

cat > /opt/local/bin/grun << EOL
#!/bin/sh
java -cp "/opt/local/lib/antlr4/antlr-4.5.3-complete.jar:$CLASSPATH" org.antlr.v4.gui.TestRig $*
EOL
chmod a+x /opt/local/bin/grun

Git Comprehension

Wed, 23 Mar 2016 08:35:00 +0000

Some commands to help navigate a large and / or fast moving repository:

# Show first line of commit message only
git log --oneline
# Show the files touched by a commit
git log --status
# Do not show merges (particularly useful when rebasing is frowned upon)
git log --no-merges
# Show details of the last commit (HEAD)
git show
# Show details of the penultimate commit
git show HEAD~1
# Show details of a particular commit e.g. c6a105e
git show c6a105e
# Show the commits that touched a file and where
git blame README
# Show changed files across commits
git diff-tree --no-commit-id --name-only -r HEAD..HEAD~3
# Show commits that match some regex e.g. foo
git log --grep foo
# Show commits by an author (use email too if more than one MR)
git log --author "Matt Roberts"

Minimal Play 2.3 Project

Tue, 22 Mar 2016 08:20:00 +0000

Get started with the Play Framework 2.3 with just 4 files and no Typesafe Activator:

.
├── app
│   └── controllers
│       └── Application.scala
├── build.sbt
├── conf
│   └── routes
├── project
│   └── plugins.sbt
└── test

Start with plugins.sbt:

addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.3.9")

Then build.sbt:

name := "play-app"

version := "1.0.0-SNAPSHOT"

lazy val root = (project in file(".")).enablePlugins(PlayScala)

Then Application.scala:

package controllers

import play.api.mvc._

object Application extends Controller {
  def index = Action {
    Ok("It works!")
  }
}

And finally the routes file:

GET  /  controllers.Application.index

At version 2.3.x, the PlayScala plugin still includes quite a few dependencies that you might not actually want. Including, but not limited to, specs2 (I prefer ScalaTest).

Oh and whilst completely optional, here is my .gitignore file:

logs
project/project
project/target
target
tmp
dist
.cache

Python version management

Mon, 21 Mar 2016 18:32:00 +0000

I stumbled upon pyenv today. It lets you:

change the global python version per user
change the python version per project
override the python version with an environment variable

It doesn’t depend on python (unlike virtualenv).

There’s also a pyenv plugin to manage virtualenv.

It’s a fork of rbenv and works in the same way i.e. shims and shell functions.

Gmail and Mac OS X Mail

Mon, 21 Mar 2016 08:00:00 +0000

I wanted to be able to read and write emails on the train. I do tether my laptop to my phone, but the signal drops out pretty regularly. That makes using Gmail in a browser difficult. When not on the train I usually use Gmail in a browser because it has better search than Mail.

In Gmail -> Settings -> Forwarding and POP/IMAP:

Check ‘Enable IMAP’
Check ‘Auto-expunge-off’
Check ‘Archive the message (default)’
Check ‘Do not limit the number of messages in an IMAP folder (default)’

The second setting is required to enable the third setting.

In Mail -> Preferences -> Accounts -> Google -> Mailbox Behaviors:

Uncheck ‘Store draft messages on the server’
Uncheck ‘Store sent messages on the server’
Uncheck ‘Store deleted messages on the server’

The first setting stops lots of drafts appearing in Gmail. The second and third settings prevent duplicates (and probably reduces traffic).

Move . to foo excluding foo

Wed, 16 Mar 2016 18:32:02 +0000

I don’t think you can do this with the mv command alone, so I use find.

Test first:

find  . -depth 1 ! -name foo -exec echo {} \;

Do the move:

find  . -depth 1 ! -name foo -exec mv {} foo \;

Stack excludes as required:

find  . -depth 1 ! -name foo ! -name bar -exec mv {} foo \;