Practical type mining in Scala

Practical type mining in Scala
the fastest way from A to Z.tpe
Rose Toomey, Novus Partners
11 June 2013 @ Scala Days

Where Salat started
https://github.com/novus/salat
Salat began because we wanted a
simple, seamless way to serialize and
deserialize our data model without external
mappings.
Pickled Scala signatures (SID #10) allowed us
to mine type information without resorting to
runtime reflection.

Scala reflection before 2.10
Why did Salat resort to pickled Scala signatures?
• Scala before 2.10 used reflection from Java
– Reflection didn’t know about Scala features like
implicits, path-dependent types, etc.
– Type erasure: parameterized types were
unrecoverable at runtime without Manifest
workaround
Why should we settle for having less information than the
compiler does?
Workaround: raid the compiler’s secret stash.

Benefits of Scala 2.10 reflection
• Choose between runtime and compile time
reflection
• Significant parts of the compiler API are now exposed
• Reify Scala expressions into abstract syntax trees
• Vastly better documentation!

Navigating the universe
A universe is an environment with
access to trees, symbols and their
types.
• scala.reflect.runtime.universe
links symbols and types to the
underlying classes and runtime
values of the JVM
• scala.reflect.macros.Universe
is the compiler universe

Macros and the compiler
The compiler universe has one mirror
scala.tools.nsc.Global#rootMirror
Macros access the compiler universe and
mirror via an instance of
scala.reflect.macros.Context
To get started with a simple example, see
Eugene Burmako’s printf macro:
http://docs.scala-
lang.org/overviews/macros/overview.html

Mirror, mirror
Mirrors provide access to the symbol
table within a universe.
The compiler has one universe and one
mirror, which loads symbols from pickled
Scala signatures using ClassFileParser.
At runtime there is only one universe, but
it has a mirror for each classloader. The
classloader mirror creates invoker
mirrors, which are used for
instances, classes, methods, fields –
everything.

Which universe?
Play with the compiler’s universe using the Scala REPL :power mode.
At runtime, get a mirror for your classloader and then use
reflect, reflectClass and reflectModule to get more specific invoker
mirrors.
scala.reflect.runtime.currentMirror
For macros, your macro implementation takes a Context c and then import
the macro universe.
The macro universe exposes the compiler universe and provides mutability
for reflection artifacts so your macros can create or transform ASTs.
import c.universe._

Symbols and Types
Symbols exist in a hierarchy that
provides all available information
about the declaration of entities and
members.
Types represent information about
the type of a symbol: its
members, base
types, erasure, modifiers, etc.

What can I do with a Type?
• Comparisons: check equality, subtyping
• Mine type information about the members and inner types
– declarations gets all the members declared on the type
– members gets all the members of this type, either declared or
inherited
– Use declaration or member to find a type by symbol
Get the type’s own termSymbol or typeSymbol
Type instances represent information about the type of a
corresponding symbol – so to understand types we need to
examine which types of symbols are interesting and why.

Great! Now I want a type…
Import a universe and use typeOf:
scala> import scala.reflect.runtime.universe._
import scala.reflect.runtime.universe._
scala> case class Foo(x: Int)
defined class Foo
scala> val fooTpe = typeOf[Foo]
fooTpe: reflect.runtime.universe.Type = Foo
scala.reflect.internal.Definitions defines value class
types (Unit, primitives) and trivial types
(Any, AnyVal, AnyRef).

Comparing types
Don’t compare types using == to check for
equality because under certain conditions it does
not work. Type aliases are one example but due
to some internal implementation details, == could
fail even for the same types if they were loaded
differently.
Use these handy emoji instead:
=:= Is this type equal to that type?
<:< Is this type a subtype of that type?
* Don’t confuse these type comparisons with deprecated Manifest operations like <:< and >:>
* Tip of the hat to @softprops for the original emoji usage in his presentation on sbt

Inspecting types in detail
The REPL :power mode is full of undocumented treats like :type –v
scala> :type -v case class Foo[T](t: T)
// Type signature
[T]AnyRef
with Product
with Serializable {
val t: T
private[this] val t: T
def <init>(t: T): Foo[T]
def copy[T](t: T): Foo[T]
...
}
// Internal Type structure
PolyType(
typeParams = List(TypeParam(T))
resultType = ClassInfoType(
...
)
)

Symbols in more depth
Start here:
scala.reflect.internal.Symbols
TypeSymbol represents types, classes, traits and type parameters.
It provides information about covariance and contravariance.
TermSymbol covers a lot of ground: var, val, def, object
declarations.
SymbolApi provides is methods to check whether a Symbol
instance can be cast to a more specific type of symbol, as well as
as methods to actually cast, e.g. isTerm and asTerm.

Interesting type symbols
ClassSymbol provides access to all the information
contained in a class or trait.
• baseClasses in linear order from most to least specific
• isAbstractClass, isTrait, isCaseClass
• isNumeric, isPrimitive, isPrimitiveValueClass
• Find companion objects

The world of term symbols
Term symbols represent val, var, def, and object
declarations as well as packages and value parameters.
Accordingly you can find interesting methods on them like:
• isVal, isVar
• isGetter, isSetter, isAccessor, isParamAccessor
• isParamWithDefault (note there is not any easy way to get
the value of the default argument yet)
• isByNameParam (big improvement!)
• isLazy

Term symbols: methods
Use MethodSymbol to get all the details of methods:
• is it a constructor? the primary constructor?
• use paramss to get all the parameter lists of the methods
(ss = list of lists of symbols)
• return type
• type params (empty for non parameterized methods)
• does the method support variable length argument lists?
When members or member(ru.Name) returns a Symbol, you can
convert it to a MethodSymbol using asMethod

Term symbols: modules
Use ModuleSymbol to navigate object declarations:
• Find companion objects (See this StackOverflow
discussion)
• Find nested objects (See this StackOverflow
discussion)
Given a ClassSymbol, use companionSymbol.asModule to get a
ModuleSymbol which you can turn into a companion
object instance using the mirror
reflectModule(moduleSymbol).instance

Getting symbols out of types
Have a Type?
- typeSymbol returns either NoSymbol or a Symbol which can
be cast using asType
- similarly, termSymbol
Use the members method to get a MemberScope, which has an
iterator of symbols:
scala> typeOf[Foo].members
res61: reflect.runtime.universe.MemberScope =
Scopes(constructor Foo, value x, ...

Ask for it by name
If you know exactly what you want, use newTermName and
newTypeName. If it doesn’t work out, you’ll get back NoSymbol.
scala> case class Foo(x: Int)
defined class Foo
scala> typeOf[Foo].member(ru.newTermName("x"))
res64: reflect.runtime.universe.Symbol = value x
scala> typeOf[Foo].member(ru.newTypeName("x"))
res65: reflect.runtime.universe.Symbol = <none>

Find the constructor
scala.reflect.api.StandardNames provides standard term
names as nme, available from your universe.
scala> typeOf[Foo].member(nme.CONSTRUCTOR)
res66: reflect.runtime.universe.Symbol =
constructor Foo
scala> res66.asMethod.isPrimaryConstructor
res68: Boolean = true

Trees
Trees (ASTs) are the foundation of
Scala’s abstract type syntax for
representing code.
The parser creates an untyped tree
structure that is immutable except for
Position, Symbol and Type. A later
stage of the compiler then fills in this
information.

From tree to Scala signature
$ scalac -Xshow-phases
phase name id description
---------- -- -----------
parser 1 parse source into ASTs, perform simple
desugaring
namer 2 resolve names, attach symbols to named trees
typer 4 the meat and potatoes: type the trees
pickler 8 serialize symbol tables
• The parser creates trees
• The namer fills in tree symbols, creates completers (symbol.info)
• The typer computes types for trees
• The pickler serializes symbols along with types into ScalaSignature
annotation

Make it so
reify takes a Scala expression and converts
into into a tree.
When you use reify to create a tree, it is
hygienic: once the identifiers in the tree are
bound, the meaning cannot later change.
The return type of reify is Expr, which wraps
a typed tree with its TypeTag and some
methods like splice for transforming trees.

Creating a tree
scala> reify{ object MyOps { def add(a: Int, b: Int)
= a + b } }.tree
res15: reflect.runtime.universe.Tree =
{
object MyOps extends AnyRef {
def <init>() = {
super.<init>();
()
};
def add(a: Int, b: Int) = a.$plus(b)
};
()
}

Inspecting the raw tree
Once you’ve reified an expression using the macro
universe, you can use showRaw to show the raw tree, which you
can use in a macro:
scala> showRaw(reify{ object MyOps { def add(a: Int, b: Int) = a
+ b } })
res16: String =
Expr(Block(List(ModuleDef(Modifiers(), newTermName("MyOps"), Tem
plate(List(Ident(newTypeName("AnyRef"))), emptyValDef, List(DefD
ef(Modifiers(), nme.CONSTRUCTOR, List(), List(List()), TypeTree(
), Block(List(Apply(Select(Super(This(tpnme.EMPTY), tpnme.EMPTY)
, nme.CONSTRUCTOR), List())), Literal(Constant(())))), DefDef(Mo
difiers(), newTermName("add"), List(), List(List(ValDef(Modifier
s(PARAM), newTermName("a"), Ident(scala.Int), EmptyTree), ValDef
(Modifiers(PARAM), newTermName("b"), Ident(scala.Int), EmptyTree
))), TypeTree(), Apply(Select(Ident(newTermName("a")), newTermNa
me("$plus")), List(Ident(newTermName("b"))))))))), Literal(Const
ant(()))))

Scala ToolBox: compile at runtime
Runtime classloader mirrors can create
a compilation toolbox whose symbol
table is populated by that mirror.
Want a tree? Use ToolBox#parse to
turn a string of code representing an
expression into an AST.
Have a tree? Use Toolbox#eval to spawn
the compiler, compiler in memory, and
launch the code.
See scala.tools.reflect.ToolBox for
more, as well as this StackOverflow
discussion.

Type erasure: fighting the good fight
$ scalac -Xshow-phases
phase name id description
---------- -- -----------
erasure 16 erase types, add interfaces for traits
When you inspect types at runtime, you will be missing some of
the type information that was available to the compiler during
stages before the JVM bytecode was generated.
If you want to mine types out of options, collections and
parameterized classes, you need to ask the compiler to stash the
type information where you'll be able to get to it at runtime.

Across the river
What ferries compiler type information to
runtime?
Before 2.10: Manifest[T]
After 2.10: TypeTag[T]
Request the compiler generate this information
using:
- using an implicit parameter of type Manifest or
TypeTag
- context bound of a type parameter on a
method or a class
- via the methods manifest[T] or typeTag[T]

Before Scala 2.10: manifests
The manifest is a shim where the compiler stores type
information, which is used to later provide runtime access
to the erased type as a Class instance.
scala> case class A[T : Manifest](t: T) { def m =
manifest[T] }
defined class A
scala> A("test").m
res26: Manifest[java.lang.String] = java.lang.String
scala> A(1).m
res27: Manifest[Int] = Int

Scala 2.10: type tag
Mirabile visu: instead of getting back a manifest, we get
back an actual type.
scala> case class A[T : TypeTag](t: T) { def tpe =
typeOf[T] }
defined class A
scala> A("test").tpe
res19: reflect.runtime.universe.Type = String
scala> A(1).tpe
res20: reflect.runtime.universe.Type = Int

Type arguments: before Scala 2.10
Using manifests:
scala> A(Map.empty[String, A[Int]]).m.erasure
res5: java.lang.Class[_] = interface
scala.collection.immutable.Map
scala> A(Map.empty[String, A[Int]]).m.typeArguments
res6: List[scala.reflect.Manifest[_]] =
List(java.lang.String, A[Int])

Type arguments: Scala 2.10
The parameterized types are now a list of types:
scala> A(Map.empty[String,A[Int]]).tpe.erasure
res17: reflect.runtime.universe.Type =
scala.collection.immutable.Map[_, Any]
scala> res10 match { case TypeRef(_, _, args)
=> args }
res18: List[reflect.runtime.universe.Type] =
List(String, A[Int])

Sadly…
The runtime reflection API is
not currently thread safe.
Keep an eye on this issue for
developments.
https://issues.scala-
lang.org/browse/SI-6240
Cheer up! The reflection used
in macros is not affected.

Reflection tools
The Scala REPL has a magnificent :power mode which is
not well explained. Examine its underpinnings here:
scala.tools.nsc.interpreter.Power
Get more details by using scalac to compile small test
files – start by playing around with the –Xprint:
compiler options:
scala.tools.nsc.settings.ScalaSettings

sbt project
To use Scala 2.10 reflection:
libraryDependencies <+= (scalaVersion)("org.scala-lang" %
"scala-compiler" % _)
To use pickled Scala signatures
libraryDependencies <+= scalaVersion("org.scala-lang" %
"scalap" % _)

Macros in the wild
• Spire – a numeric library for Scala (examples of
macros and
specializationhttp://github.com/non/spire
• Sherpa – a serialization toolkit and ‘reflection-less’
case class mapper for Scala
http://github.com/aloiscochard/sherpa
• sqlτyped – a macro which infers Scala types by
analysing SQL statements
https://github.com/jonifreeman/sqltyped

Things to read, things to watch
• Martin Odersky's Lang-NEXT 2012 keynote, Reflection
and compilers
• Paul Phillips ScalaDays 2012 presentation, Inside the
Sausage Factory: scalac internals
• Eugene Burmako’s Metaprogramming in Scala
• Daniel Sobral’s blog posts on JSON serialization with
reflection in Scala (Part I / Part II)
• StackOverflow posts tagged with Scala 2.10 reflection
• Scala issue tracker reflection tickets contain detailed
discussion and useful links

Thanks to…
• Eugene Burmako (@xeno_by) not only for many
helpful StackOverflow posts, but also his comments
on these slides
Follow me on Twitter for more interesting
presentations - @prasinous

Practical type mining in Scala

More Related Content

What's hot (17)

Viewers also liked (9)

Similar to Practical type mining in Scala (20)

Recently uploaded (20)

Practical type mining in Scala