Scala eXchange opening

Scala
What’s New, What’s Next

Scala eXchange opening address
June 15th, 2011

Martin Odersky
Typesafe and EPFL

Trajectory
10 years ago
–  The plan: Unify functional and object-oriented programming in a
practical language.
8 years ago
–  First Scala release (experimental, students as subjects)
5 years ago
–  Open source system forms around Scala
3 years ago
–  Industrial adoption starts, spreads rapidly through internet companies
and financial sector.
1 year ago
–  First Scala Days conference demonstrates strength of community,
demand for industrial support.

#1 Concern: Concurrency and Parallelism

Two motivations:

•  Make use of multicore/GPU processing power (scale-up)
•  Distribute computations over many nodes (scala-out)

Two approaches:

•  Explicit: Programmer defines and manages processes
•  Implicit: Control of parallelism given to system

  Implicit is simpler,
  But explicit is often needed when nodes and connections can
fail.

3

Scala’s Toolbox

4

Different Tools for Different Purposes

Implicit:

Parallel Collections
Collections
Distributed Collections

Parallel DSLs

Explicit:

Actors
Software transactional memory Akka
Futures

5

Scala Collections:
Building Interesting Things in Space
•  De-emphasize destructive scala> val ys = List(1, 2, 3)
ys: List[Int] = List(1, 2, 3)
updates
scala> val xs: Seq[Int] = ys
•  Focus on transformers that xs: Seq[Int] = List(1, 2, 3)
map collections to collections scala> xs map (_ + 1)
res0: Seq[Int] = List(2, 3, 4)
•  Have a complete range of
scala> ys map (_ + 1)
persistent collections res1: List[Int] = List(2, 3, 4)

6

Some General Scala Collections

Constructed automatically using decodify.

7

An Example
•  Task: Phone keys have mnemonics assigned to them.

val
mnemonics
=
Map(

'2'
-‐>
"ABC",
'3'
-‐>
"DEF",
'4'
-‐>
"GHI",
'5'
-‐>
"JKL",

'6'
-‐>
"MNO",
'7'
-‐>
"PQRS",
'8'
-‐>
"TUV",
'9'
-‐>
"WXYZ")

•  Assume you are given a dictionary dict as a list of words. Design a class
Coder with a method translate such that

new
Coder(dict).translate(phoneNumber)

produces all phrases of words in dict that can serve as mnemonics for the
phone number.

•  Example: The phone number “7225276257” should have the mnemonic
Scala
rocks

as one element of the list of solution phrases.

2-8

Program Example: Phone Mnemonics

•  This example was taken from:
Lutz Prechelt: An Empirical Comparison of Seven Programming
Languages. IEEE Computer 33(10): 23-29 (2000)

•  Tested with Tcl, Python, Perl, Rexx, Java, C++, C

•  Code size medians:
–  100 loc for scripting languages
–  200-300 loc for the others

2-9

Outline of Class Coder

class
Coder(words:
List[String])
{

private
val
mnemonics
=
Map(

'2'
-‐>
"ABC",
'3'
-‐>
"DEF",
'4'
-‐>
"GHI",
'5'
-‐>
"JKL",

'6'
-‐>
"MNO",
'7'
-‐>
"PQRS",
'8'
-‐>
"TUV",
'9'
-‐>
"WXYZ")

/**
Invert
the
mnemonics
map
to
give
a
map
from
chars
'A'
...
'Z'
to
'2'
...
'9'
*/

private
val
charCode:
Map[Char,
Char]
=
??

/**
Maps
a
word
to
the
digit
string
it
can
represent,
e.g.
“Java”
-‐>
“5282”
*/

private
def
wordCode(word:
String):
String
=
??

/**
A
map
from
digit
strings
to
the
words
that
represent
them,

*
e,g.
“5282”
-‐>
Set(“Java”,
“Kata”,
“Lava”,
...)
*/

private
val
wordsForNum:
Map[String,
Set[String]]
=
??

/**
Return
all
ways
to
encode
a
number
as
a
list
of
words
*/

def
encode(number:
String):
Set[List[String]]
=
??

/**
Maps
a
number
to
a
list
of
all
word
phrases
that
can
represent
it
*/

def
translate(number:
String):
Set[String]
=
encode(number)
map
(_
mkString
"
")

}

10

Class Coder
(1)

class
Coder(words:
List[String])
{

private
val
mnemonics
=
Map(

'2'
-‐>
"ABC",
'3'
-‐>
"DEF",
'4'
-‐>
"GHI",
'5'
-‐>
"JKL",

'6'
-‐>
"MNO",
'7'
-‐>
"PQRS",
'8'
-‐>
"TUV",
'9'
-‐>
"WXYZ")

/**
Invert
the
mnemonics
map
to
give
a
map
from
chars
'A'
...
'Z'
to
'2'
...
'9'
*/

private
val
charCode:
Map[Char,
Char]
=

/**
Maps
a
word
to
the
digit
string
it
can
represent
*/

private
def
wordCode(word:
String):
String
=
??

/**
A
map
from
digit
strings
to
the
words
that
represent
them
*/

private
val
wordsForNum:
Map[String,
List[String]]
=
??

/**
Return
all
ways
to
encode
a
number
as
a
list
of
words
*/

def
encode(number:
String):
Set[List[String]]
=
??

/**
Maps
a
number
to
a
list
of
all
word
phrases
that
can
represent
it
*/

def
translate(number:
String):
Set[String]
=
encode(number)
map
(_
mkString
"
")

}

11

Class Coder
(1)

class
Coder(words:
List[String])
{

private
val
mnemonics
=
Map(

'2'
-‐>
"ABC",
'3'
-‐>
"DEF",
'4'
-‐>
"GHI",
'5'
-‐>
"JKL",

'6'
-‐>
"MNO",
'7'
-‐>
"PQRS",
'8'
-‐>
"TUV",
'9'
-‐>
"WXYZ")

/**
Invert
the
mnemonics
map
to
give
a
map
from
chars
'A'
...
'Z'
to
'2'
...
'9'
*/

private
val
charCode:
Map[Char,
Char]
=

for
((digit,
str)
<-‐
mnemonics;
ltr
<-‐
str)
yield
(ltr
-‐>
digit)

/**
Maps
a
word
to
the
digit
string
it
can
represent
*/

private
def
wordCode(word:
String):
String
=
??

/**
A
map
from
digit
strings
to
the
words
that
represent
them
*/

private
val
wordsForNum:
Map[String,
List[String]]
=
??

/**
Return
all
ways
to
encode
a
number
as
a
list
of
words
*/

def
encode(number:
String):
Set[List[String]]
=
??

/**
Maps
a
number
to
a
list
of
all
word
phrases
that
can
represent
it
*/

def
translate(number:
String):
Set[String]
=
encode(number)
map
(_
mkString
"
")

}

12

Class Coder
(2)

class
Coder(words:
List[String])
{

private
val
mnemonics
=
Map(

'2'
-‐>
"ABC",
'3'
-‐>
"DEF",
'4'
-‐>
"GHI",
'5'
-‐>
"JKL",

'6'
-‐>
"MNO",
'7'
-‐>
"PQRS",
'8'
-‐>
"TUV",
'9'
-‐>
"WXYZ")

/**
Invert
the
mnemonics
map
to
give
a
map
from
chars
'A'
...
'Z'
to
'2'
...
'9'
*/

private
val
charCode:
Map[Char,
Char]
=

for
((digit,
str)
<-‐
m;
letter
<-‐
str)
yield
(letter
-‐>
digit)

/**
Maps
a
word
to
the
digit
string
it
can
represent,
e.g.
“Java”
-‐>
“5282”
*/

private
def
wordCode(word:
String):
String
=

/**
A
map
from
digit
strings
to
the
words
that
represent
them
*/

private
val
wordsForNum:
Map[String,
Set[String]]
=
??

/**
Return
all
ways
to
encode
a
number
as
a
list
of
words
*/

def
encode(number:
String):
Set[List[String]]
=
??

/**
Maps
a
number
to
a
list
of
all
word
phrases
that
can
represent
it
*/

def
translate(number:
String):
Set[String]
=
encode(number)
map
(_
mkString
"
")

}

13

Class Coder
(2)

class
Coder(words:
List[String])
{

private
val
mnemonics
=
Map(

'2'
-‐>
"ABC",
'3'
-‐>
"DEF",
'4'
-‐>
"GHI",
'5'
-‐>
"JKL",

'6'
-‐>
"MNO",
'7'
-‐>
"PQRS",
'8'
-‐>
"TUV",
'9'
-‐>
"WXYZ")

/**
Invert
the
mnemonics
map
to
give
a
map
from
chars
'A'
...
'Z'
to
'2'
...
'9'
*/

private
val
charCode:
Map[Char,
Char]
=

for
((digit,
str)
<-‐
m;
letter
<-‐
str)
yield
(letter
-‐>
digit)

/**
Maps
a
word
to
the
digit
string
it
can
represent,
e.g.
“Java”
-‐>
“5282”
*/

private
def
wordCode(word:
String):
String
=

word.toUpperCase
map
charCode

/**
A
map
from
digit
strings
to
the
words
that
represent
them
*/

private
val
wordsForNum:
Map[String,
Set[String]]
=
??

/**
Return
all
ways
to
encode
a
number
as
a
list
of
words
*/

def
encode(number:
String):
Set[List[String]]
=
??

/**
Maps
a
number
to
a
list
of
all
word
phrases
that
can
represent
it
*/

def
translate(number:
String):
Set[String]
=
encode(number)
map
(_
mkString
"
")

}

14

Class Coder
(3)

class
Coder(words:
List[String])
{

private
val
mnemonics
=
Map(

'2'
-‐>
"ABC",
'3'
-‐>
"DEF",
'4'
-‐>
"GHI",
'5'
-‐>
"JKL",

'6'
-‐>
"MNO",
'7'
-‐>
"PQRS",
'8'
-‐>
"TUV",
'9'
-‐>
"WXYZ")

/**
Invert
the
mnemonics
map
to
give
a
map
from
chars
'A'
...
'Z'
to
'2'
...
'9'
*/

private
val
charCode:
Map[Char,
Char]
=

for
((digit,
str)
<-‐
m;
letter
<-‐
str)
yield
(letter
-‐>
digit)

/**
Maps
a
word
to
the
digit
string
it
can
represent
*/

private
def
wordCode(word:
String):
String
=

word.toUpperCase
map
charCode

/**
A
map
from
digit
strings
to
the
words
that
represent
them,

*
e,g.
“5282”
-‐>
Set(“Java”,
“Kata”,
“Lava”,
...)
*/

private
val
wordsForNum:
Map[String,
List[String]]
=

/**
Return
all
ways
to
encode
a
number
as
a
list
of
words
*/

def
encode(number:
String):
Set[List[String]]
=
??

/**
Maps
a
number
to
a
list
of
all
word
phrases
that
can
represent
it
*/

def
translate(number:
String):
Set[String]
=
encode(number)
map
(_
mkString
"
")

}

15

Class Coder
(3)

class
Coder(words:
List[String])
{

private
val
mnemonics
=
Map(

'2'
-‐>
"ABC",
'3'
-‐>
"DEF",
'4'
-‐>
"GHI",
'5'
-‐>
"JKL",

'6'
-‐>
"MNO",
'7'
-‐>
"PQRS",
'8'
-‐>
"TUV",
'9'
-‐>
"WXYZ")

/**
Invert
the
mnemonics
map
to
give
a
map
from
chars
'A'
...
'Z'
to
'2'
...
'9'
*/

private
val
charCode:
Map[Char,
Char]
=

for
((digit,
str)
<-‐
m;
letter
<-‐
str)
yield
(letter
-‐>
digit)

/**
Maps
a
word
to
the
digit
string
it
can
represent
*/

private
def
wordCode(word:
String):
String
=

word.toUpperCase
map
charCode

/**
A
map
from
digit
strings
to
the
words
that
represent
them,

*
e,g.
“5282”
-‐>
Set(“Java”,
“Kata”,
“Lava”,
...)
*/

private
val
wordsForNum:
Map[String,
List[String]]
=

(words
groupBy
wordCode)
withDefaultValue
List()

/**
Return
all
ways
to
encode
a
number
as
a
list
of
words
*/

def
encode(number:
String):
Set[List[String]]
=
??

/**
Maps
a
number
to
a
list
of
all
word
phrases
that
can
represent
it
*/

def
translate(number:
String):
Set[String]
=
encode(number)
map
(_
mkString
"
")

}

16

Class Coder
(4)

class
Coder(words:
List[String])
{
...

/**
Return
all
ways
to
encode
a
number
as
a
list
of
words
*/

def
encode(number:
String):
Set[List[String]]
=

/**
Maps
a
number
to
a
list
of
all
word
phrases
that
can
represent
it
*/

def
translate(number:
String):
Set[String]
=
encode(number)
map
(_
mkString
"
")

}

17

Class Coder
(4)

class
Coder(words:
List[String])
{
...

/**
Return
all
ways
to
encode
a
number
as
a
list
of
words
*/

def
encode(number:
String):
Set[List[String]]
=

if
(number.isEmpty)

Set(List())

/**
Maps
a
number
to
a
list
of
all
word
phrases
that
can
represent
it
*/

def
translate(number:
String):
Set[String]
=
encode(number)
map
(_
mkString
"
")

}

18

Class Coder
(4)

class
Coder(words:
List[String])
{
...

/**
Return
all
ways
to
encode
a
number
as
a
list
of
words
*/

def
encode(number:
String):
Set[List[String]]
=

if
(number.isEmpty)

Set(List())

else
{

for
{

splitPoint
<-‐
1
to
number.length

/**
Maps
a
number
to
a
list
of
all
word
phrases
that
can
represent
it
*/

def
translate(number:
String):
Set[String]
=
encode(number)
map
(_
mkString
"
")

}

19

Class Coder
(4)

class
Coder(words:
List[String])
{
...

/**
Return
all
ways
to
encode
a
number
as
a
list
of
words
*/

def
encode(number:
String):
Set[List[String]]
=

if
(number.isEmpty)

Set(List())

else
{

for
{

splitPoint
<-‐
1
to
number.length

word
<-‐
wordsForNum(number
take
splitPoint)

/**
Maps
a
number
to
a
list
of
all
word
phrases
that
can
represent
it
*/

def
translate(number:
String):
Set[String]
=
encode(number)
map
(_
mkString
"
")

}

20

Class Coder
(4)

class
Coder(words:
List[String])
{
...

/**
Return
all
ways
to
encode
a
number
as
a
list
of
words
*/

def
encode(number:
String):
Set[List[String]]
=

if
(number.isEmpty)

Set(List())

else
{

for
{

splitPoint
<-‐
1
to
number.length

word
<-‐
wordsForNum(number
take
splitPoint)

rest
<-‐
encode(number
drop
splitPoint)

}
yield
word
::
rest

}.toSet

/**
Maps
a
number
to
a
list
of
all
word
phrases
that
can
represent
it
*/

def
translate(number:
String):
Set[String]
=
encode(number)
map
(_
mkString
"
")

}

21

In Summary
•  In the original experiment:
–  Scripting language programs were shorter and often even faster than
Java/C/C++ because their developers tended to use standard
collections.

•  In Scala’s solution:
–  Obtained a further 5x reduction in size because of the systematic use of
functional collections.

•  Scala’s collection’s are also easily upgradable to parallel.

Think collection transformers, not CRUD!

22

A Solution in Java
By Josh Bloch, author of Java collections

23

Going Parallel

•  In Scala 2.9, collections support parallel operations.
•  Two new methods:
c.par
returns parallel version of collection c

c.seq
returns sequential version of collection c

•  The right tool for addressing the PPP (popular parallel
programming) challenge.
•  I expect this to be the cornerstone for making use of
multicores for the rest of us.

24

a parallel collection
Parallel Coder

class
Coder(words:
ParVector[String])
{

private
val
mnemonics
=
Map(

'2'
-‐>
"ABC",
'3'
-‐>
"DEF",
'4'
-‐>
"GHI",
'5'
-‐>
"JKL",

'6'
-‐>
"MNO",
'7'
-‐>
"PQRS",
'8'
-‐>
"TUV",
'9'
-‐>
"WXYZ")

/**
Invert
the
mnemonics
map
to
give
a
map
from
chars
'A'
...
'Z'
to
'2'
...
'9'
*/

private
val
charCode:
Map[Char,
Char]
=

for
((digit,
str)
<-‐
m;
letter
<-‐
str)
yield
(letter
-‐>
digit)

/**
Maps
a
word
to
the
digit
string
it
can
represent
*/

private
def
wordCode(word:
String):
String
=
word.toUpperCase
map
charCode

/**
A
map
from
digit
strings
to
the
words
that
represent
them
*/

private
val
wordsForNum:
Map[String,
List[String]]
=
words
groupBy
wordCode

/**
Return
all
ways
to
encode
a
number
as
a
list
of
words
*/

def
encode(number:
String):
Set[List[String]]
=

if
(number.isEmpty)
Set(List())

else
{

mapping sequential

for
{
collection to parallel

splitPoint
<-‐
(1
to
number.length).par

word
<-‐
wordsForNum(number
take
splitPoint)

rest
<-‐
encode(number
drop
splitPoint)

}
yield
word
::
rest

}.toSet

25

Parallel Collections
•  Split work by number of Processors
•  Each Thread has a work queue that is split
exponentially. Largest on end of queue
•  Granularity balance against scheduling overhead
•  On completion threads “work steals” from end of other
thread queues

26

General Collection Hierarchy

GenTraversable

GenIterable
Traversable
GenSeq
Iterable ParIterable

Seq ParSeq

27

Going Distributed
•  Can we get the power of parallel collections to work on
10’000s of computers?
•  Hot technologies: MapReduce (Google’s and Hadoop)
•  But not everything is easy to fit into that mold
•  Sometimes 100’s of map-reduce steps are needed.
•  Distributed collections retain most operations, provide a
powerful frontend for MapReduce computations.
•  Scala’s uniform collection model is designed to also
accommodate parallel and distributed.

28

The Future
Scala’s persistent collections are
•  easy to use: few steps to do the job
•  concise: one word replaces a whole loop
•  safe: type checker is really good at catching errors
•  fast: collection ops are tuned, can be parallelized
•  scalable: one vocabulary to work on all kinds of
collections: sequential, parallel, or distributed.

I see them play a rapidly increasing role in software development.

29

Akka: Event-driven Middleware in Scala
•  Unified programming model for:
–  scaling up to many cores
–  scaling out to many nodes
–  fault-tolerance
•  Actors:
–  lightweight (run millions on a single node)
–  send and receive messages async
–  location transparent
•  Transparent and adaptive:
–  load-balancing and routing
–  replication and fail-over
–  elasticity: grow/shrink dynamically
•  Decouples app logic from low-level
mechanisms

30

Akka vs scala.actors

Akka:
+ Best performance
+ Scalable to clusters and clouds
+ Best safety through encapsulation

Scala Actors:
+ Simple to get started
+ Good integration with threads
+ Blocking as well as non-blocking receives
− Erlang-inherited mailbox model can lead to inefficient code

Over the next releases we plan to merge the two libraries.
31

Other New Developments
•  Scala-based reflection
•  Type Dynamic
•  Effects in an optional pluggable type system (?)

•  Libraries:
–  I/O, including event-based
–  STM
–  Reactive programming, GUI

•  Most effort will go in tools
–  Eclipse IDE
–  REPL
–  SBT (simple build tool)
–  Type debugger

32

Going further

But how do we keep a bunch of Fermi’s happy?
–  How to find and deal with 10000+ threads in an
application?
–  Parallel collections and actors are necessary but not
sufficient for this.
Our bet for the mid term future: parallel embedded DSLs.
–  Find parallelism in domains: physics simulation, machine
learning, statistics, ...
Joint work with Kunle Olukuton, Pat Hanrahan @ Stanford.
EPFL side funded by ERC.

33

Scientific Virtual Personal Data
Applications
Engineering Worlds Robotics informatics

Domain
Physics Probabilistic Machine
Specific Rendering Scripting Learning
Languages (Liszt) (RandomT) (OptiML)

Domain Embedding Language (Scala)
Polymorphic Embedding Staging Static Domain Specific Opt.
DSL
Infrastructure
Parallel Runtime (Delite, Sequoia, GRAMPS)
Dynamic Domain Spec. Opt. Task & Data Parallelism Locality Aware Scheduling

Hardware Architecture
Heterogeneous
OOO Cores SIMD Cores Threaded Cores Specialized Cores
Hardware
Programmable Scalable Isolation & On-chip Pervasive
Hierarchies Coherence Atomicity Networks Monitoring
34

Example: Liszt - A DSL for Physics
Simulation

Combustion

Turbulence

Fuel injection
Transition Thermal
•  Mesh-based
•  Numeric Simulation
•  Huge domains Turbulence

–  millions of cells
•  Example: Unstructured Reynolds-averaged Navier
Stokes (RANS) solver

35

Liszt as Virtualized Scala
val // calculating scalar convection (Liszt)

val Flux = new Field[Cell,Float] AST
val Phi = new Field[Cell,Float]
val cell_volume = new Field[Cell,Float]
val deltat = .001
...
untilconverged {
for(f <- interior_faces) {
val flux = calc_flux(f)
Flux(inside(f)) -= flux
Flux(outside(f)) += flux
}
for(f <- inlet_faces) { Optimisers Generators
Flux(outside(f)) += calc_boundary_flux(f)
} …
for(c <- cells(mesh)) {
Phi(c) += deltat * Flux(c) /cell_volume Schedulers
(c)
} …
for(f <- faces(mesh))
Flux(f) = 0.f Hardware
}

DSL Library GPU, Multi-Core, etc

36

Other Developments
DelayedInit
Dynamic Scala 2.9 also introduces two new “magic” traits:
Tools
DelayedInit and
Reflection
Dynamic!
STM
@specialized
A lot work has been done on the tools side:
Slick
Scala IDE for Eclipse 2.0,
scala.reflect
SBT 0.10,
scala.react
enhanced REPL,
Scala.NET
Migration Manager Diagnostics (MiMaLib)
Scala.IO (?)
Effects (?)
Javascript (?)

37

Planned Work
DelayedInit
Dynamic
Over the next releases, we plan to introduce:
Tools
•  Scala STM, a standard STM API,
Reflection
•  Enhanced support for @specialized,
STM
•  Slick, A common framework for connecting with
@specialized
databases and distributed collections.
Slick
•  Scala.reflect, a Scala-centric reflection library.
scala.reflect
•  Scala.react, a reactive programming framework.
scala.react
•  An updated Scala.NET
Scala.NET
We are also exploring:
Scala.IO (?)
•  Adding Scala.IO to the standard distibution
Effects (?)
•  An effect system for Scala
Javascript (?)
•  A Scala to Javascript translator

38

DelayedInit and App
DelayedInit
Dynamic Trait Application is convenient, but has hidden problems:
Tools
object First extends Application {
Reflection println("hi there!)
STM }

@specialized
Slick Body is executed as part of the object initializer.
scala.reflect main method inherited from Application is empty.

scala.react Body is neither optimized nor threading-friendly.
Scala.NET
Scala.IO (?) Solution: Replace Application with App.
Effects (?)
Javascript (?)

39

DelayedInit and App
DelayedInit
Dynamic Trait App solves these problems:
Tools
object First extends App {
Reflection println("hi there!)
STM }

@specialized
Slick Body is now stored in a buffer, executed by main method.
scala.reflect Here’s an outline of this trait.
scala.react
Scala.NET trait App extends DelayedInit {
Scala.IO (?) def main(args: Array[String]) =
for (code <- inits) code()
Effects (?) }
Javascript (?)

40

Conclusion
Scala has the right “genes” to scale into parallel +
distributed computing:
•  Strong, practical foundation: JVM
•  Functional programming reduces liabilities
•  Library-centric approach makes for building tailored
solutions.
Makes it possible to write:
•  High-level, easy to use libraries (e.g., collections)
•  Safer concurrency concepts (e.g., actors)
•  Parallel DSLs (e.g., EPFL/Stanford project)

41

Thank You

scala-lang.org

typesafe.com

42

Scala eXchange opening

More Related Content

What's hot (17)

Similar to Scala eXchange opening (20)

More from Martin Odersky (11)

Recently uploaded (20)

Scala eXchange opening