A Scala Corrections Library

A Scala Corrections Library
Paul Phillips

paulp@improving.org

Source: xkcd, of course.

“When I'm working on a problem, I never think
about beauty. I think only how to solve the problem.”
!

“But when I have ﬁnished, if the solution is not
beautiful, I know it is wrong.”
– R. Buckminster Fuller
(syntax highlighting donated by paulp)

“When I'm working on a problem, I never
think about beauty. I think only how to
solve the problem.”

!

“But when I have ﬁnished, if the solution
is not beautiful, I know it is wrong.”

– R. Buckminster Fuller

trait ParSeqViewLike[
+T,
+Coll <: Parallel,
+CollSeq,
+This <: ParSeqView[T, Coll, CollSeq]
with ParSeqViewLike[T, Coll, CollSeq, This, ThisSeq],
+ThisSeq <: SeqView[T, CollSeq]
with SeqViewLike[T, CollSeq, ThisSeq]
] extends GenSeqView[T, Coll]
with GenSeqViewLike[T, Coll, This]
with ParIterableView[T, Coll, CollSeq]
with ParIterableViewLike[T, Coll, CollSeq, This, ThisSeq]
with ParSeq[T]
with ParSeqLike[T, This, ThisSeq]

The Winding Stairway
• Five years on scala

• Rooting for scala/typesafe

• But I quit a dream job...

• ...because I lost faith

Should you care?
•

I offer my credentials only to bear witness to my
credibility

•

I suspect I have written more scala code than
anyone else, ever.

•

What’s visible in compiler/library represents only
a small fraction of it

Caveats
•

I ran out of time. Slides are rushed. Forgive me.

•

Error messages and repl transcripts have been
heavily trimmed for clarity on a slide

•

This works counter to message when the point
involves complexity or incomprehensibility

•

So verbosify all compiler messages by a factor
of three for a more accurate feel

My axe is dull
•

I have been pulling my punches

•

This has left some thinking that I quit over
technical esoterica: java compatibility, jvm
limitations, intractable compiler challenges

•

This is not accurate

Subtext, people
•

Prevailing programmer culture frowns upon
criticism of named individuals

•

In this case that doesn’t leave much room for
additional speciﬁcity

•

All the relevant facts are available in the googles

Is Scala too complex?
•

I’ll ﬁeld this one: YES

•

Is anyone fooled by specious comparisons
of language grammar size? Who cares?

•

Half the time when someone hits a bug they
can’t tell whether it is a bug in scala or the
expected behavior

•

That deﬁnitely includes me

Perceived Problem

C

•

A meme is going around that scala is too complex

•

Option A: Own it

•

Option B: Address it

•

Option C: Obscure it

p
O

i
t

n
o

Thus is born the “use case”
// A fictional idealized version of the genuine method
def map[B](f: (A)
B): Map[B]
!

// The laughably labeled "full" signature
def map[B, That](f: ((A, B))
B)
(implicit bf: CanBuildFrom[Map[A, B], B, That]): That

neither has any basis in reality!

the true name of map
// markers to distinguish Map's class type parameters
scala> class K ; class V
defined class K, V
!
scala> val host = typeOf[Map[K, V]]
host: Type = Map[K,V]
!
scala> val method = host member TermName("map")
method: Symbol = method map
!
// Correct signature for map has FOUR distinct identifiers
scala> method defStringSeenAs (host memberType method)
res0: String =
def map[B, That](f: ((K, V)) => B)
(implicit bf: CBF[Map[K,V],B,That]): That

•

Now you’re thinking “use case thing is a bug, big deal,
bugs get ﬁxed.” Do they?

•

Surely as soon as it is known the documentation spins
these fabrications, it will be addressed? If not ﬁxed, at
least it’ll be marked as inaccurate? Something?

•

Nope! To this day it’s the same. Your time is worthless.

Slightly Caricatured
map
def map[B](f: A => B): F[B]

Signature

Elegance

Advantages

Spokespicture

“map”
def map[B, That](f: A => B)
(implicit bf:
CanBuildFrom[Repr, B,
That]): That

Among the purest and
most reusable
<—- Not this.
abstractions known to
computing science
Can reason abstractly
about code

Can map a BitSet to a
BitSet without typing
“toBitSet”

The Bitset Gimmick
// Fancy, we get a Bitset back!
scala> BitSet(1, 2, 3) map (_.toString.toInt)
res0: BitSet = BitSet(1, 2, 3)

!
// Except…
scala> BitSet(1, 2, 3) map (_.toString) map (_.toInt)
res1: SortedSet[Int] = TreeSet(1, 2, 3)

!
// Um…
scala> (BitSet(1, 2, 3) map identity)(1)
<console>:21: error: type mismatch;
found
: Int(1)
required:
scala.collection.generic.CanBuildFrom[scala.collection.imm
utable.BitSet,Int,?]
(BitSet(1, 2, 3) map identity)(1)
^

similarly
scala> def f[T](x: T) = (x, new Object)
f: [T](x: T)(T, Object)
!
scala> SortedSet(1 to 10: _*)
res0: SortedSet[Int] = TreeSet(1, 2, 3,
!
scala> SortedSet(1 to 10: _*) map (x =>
res1: SortedSet[Int] = TreeSet(1, 2, 3,
!
scala> SortedSet(1 to 10: _*) map f map
res2: Set[Int] = Set(5, 10, 1, 6, 9, 2,

4, 5, 6, 7, 8, 9, 10)
f(x)._1)
4, 5, 6, 7, 8, 9, 10)
(_._1)
7, 3, 8, 4)

and in a similar vein
scala> val f: Int => Int = _ % 3
f: Int => Int = <function1>
!
scala> val g: Int => Int = _ => System.nanoTime % 1000000 toInt
g: Int => Int = <function1>
!
scala> Set(3, 6, 9) map f map g
res0: Set[Int] = Set(633000)
!
scala> Set(3, 6, 9) map (f andThen g)
res1: Set[Int] = Set(305000, 307000, 308000)

Java Interop: the cruelest joke
•

It’s impossible to call scala’s map from java!

•

See all the grotesque details at SI-4389

IX
F
T
ON

“I played with it until it got too tedious. I think the signatures work ﬁne.
What does not work is that the variances of CanBuildFrom cannot be
modelled in Java, so types do not match. And it seems Java does not
even let me override with a cast. So short answer: You can't call these
things from Java because instead of declaration side variance you
have only a broken wildcard system.”
!
— Martin Odersky

W

Lightning Round
•

My time is running out and I can hear you saying…

•

“Just give us a laundry list of collections issues”

•

Okay, you asked for it (in my mind)

•

Implementation details infest everything

•

And every detail is implementation-deﬁned

•

Capabilities should be designed around the laws
of variance; instead variance checks are
suppressed and key method contains is untyped

•

Speciﬁcity rules render contravariance useless

•

Implicit selection and type inference inextricably
bound - so type inference is largely frozen
because any change will break existing code

•

Extreme pollution of base objects - all collections
have “size: Int”, all Seqs have “apply”, etc.

•

Bundling of concerns (e.g. invariant Set)

•

Inheritance of implementation is the hammer for
every nail…

•

…yet “ﬁnal” and “private”, critical for a hope of
correctness under inheritance, are almost
unknown

•

Semantics discovered instead of designed

assume the worst
In Set(x) ++ Set(x), which x wins?
!

Can xs filter (_ => true) return xs?
!

Are defaults preserved across
operations? Which operations? Is
sortedness? Will views and Streams
retain laziness when zipped?

xs map identity
scala> val m = Map(1 -> 2) withDefaultValue 10
m: Map[Int,Int] = Map(1 -> 2)
!
scala> m(1000)
res0: Int = 10
!
scala> (m map identity)(1000)
found
: Int(1000)
required: CanBuildFrom[Map[Int,Int],(Int, Int),?]
(m map identity)(1000)
^
!
scala> m map identity apply 1000
java.util.NoSuchElementException: key not found: 1000
at MapLike$class.default(MapLike.scala:228)

types are for suckers
% find collection -name ‘*.scala’ |
xargs egrep asInstanceOf | wc -l

556

How could 556 casts ever go wrong
scala> val xs: Set[Int] =
(1 to 3).view.map(x => x)(breakOut)
!

java.lang.ClassCastException: SeqViewLike$$anon$3
cannot be cast to immutable.Set

get and apply
trivially fall into disagreement
!
scala> Map[Int,Int]() withDefaultValue 123
res0: Map[Int,Int] = Map()
!
scala> res0 contains 55
res1: Boolean = false
!
scala> res0 get 55
res2: Option[Int] = None
!
scala> res0 apply 55
res3: Int = 123

Why is covariance such an object of worship?
Types exist so we don’t have to live like this!
// WHY infer this utterly useless type?
scala> List(1, 2) ::: List(3, 4.0)
res0: List[AnyVal] = List(1, 2, 3.0, 4.0)
!

scala> PspList(1, 2) ::: PspList(3, 4.0)
found
: PspList[Int]
required: PspList[Double]

Type Inference
+
Variance
——————————

Abstracting over mutability
•
•
•

An inherited implementation is ALWAYS wrong somewhere!!

•

Half the overrides in collections exist to stave off the
incorrectness which looms above. This is nuts.!

•

Not to mention “Map”, “Set”, etc. in three namespaces

Example: how do you write "drop" so it's reusable?!
In a mutable class, drop MUST NOT share, but in an
immutable class, drop MUST share!

How many ways are there to write ‘slice’ ?
% ack --no-filename 'def slice(' src/library/

!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

override def slice(from: Int, until: Int): Iterator[A] =
def slice(from: Int, until: Int): Iterator[A] = {
def slice(from: Int, until: Int): Repr =
def slice(from: Int, until: Int): Repr = {
def slice(from: Int, until: Int): Repr = {
def slice(start: Int): PagedSeq[T] = slice(start, UndeterminedEnd)
def slice(unc_from: Int, unc_until: Int): Repr
override /*IterableLike*/ def slice(from: Int, until: Int): Vector[A] =
override /*TraversableLike*/ def slice(from: Int, until: Int): Repr = {
override def slice(_start: Int, _end: Int): PagedSeq[T] = {
override def slice(from1: Int, until1: Int): IterableSplitter[T] =
override def slice(from1: Int, until1: Int): SeqSplitter[T] =
override def slice(from: Int, until: Int) = {
override def slice(from: Int, until: Int) = {
override def slice(from: Int, until: Int): List[A] = {
override def slice(from: Int, until: Int): Repr = self.slice(from, until)
override def slice(from: Int, until: Int): Repr = {
override def slice(from: Int, until: Int): Stream[A] = {
override def slice(from: Int, until: Int): String = {
override def slice(from: Int, until: Int): This =
override def slice(from: Int, until: Int): This =
override def slice(from: Int, until: Int): Traversable[A]
override def slice(from: Int, until: Int): WrappedString = {
override def slice(unc_from: Int, unc_until: Int): Repr = {

scala.conﬂation
•

Every collection must have size

•

Every sequence must have apply

•

Every call to map includes a "builder factory"

•

Every set must be invariant

•

Everything must suffer universal equality

predictability
One of these expressions returns 2 and one returns
never. Feeling lucky?

!

scala> (Stream from 1) zip (Stream from 1)
map { case (x, y) => x + y } head
!

scala> (Stream from 1, Stream from 1).zipped
map (_ + _) head

sets
Two complementary ways to define Set[A].
Complementary - and NOT the same thing!
Intensional

Extensional

Speciﬁcation

Membership test

Members

Variance

Set[-A]

Set[+A]

Deﬁning Signature

A => Boolean

Iterable[A]

Size

Unknowable

Known

Duplicates(*)

Meaningless

Disallowed

What's going on here?
scala> class xs[A] extends Set[A]
error: class xs has 4 unimplemented members.
!

// Intensional/extensional, conflated.
// Any possibility of variance eliminated.
def iterator: Iterator[A]
def contains(elem: A): Boolean
// What are these doing in the interface?
// Why can I define a Seq without them?
def -(elem: A): Set[A]
def +(elem: A): Set[A]

todo: also add all other methods
% git grep 'todo: also add' 607cb4250d
SynchronizedMap.scala: // !!! todo: also add all other methods

!
% git grep 'todo: also add' origin/master
SynchronizedMap.scala: // !!! todo: also add all other methods

!
commit 607cb4250d
Author: Martin Odersky <odersky@gmail.com>
Date:
Mon May 25 15:18:48 2009 (4 years, 8 months ago)

!
added SynchronizedMap; changed Set.put to Set.add, implemented
LinkedHashMap/Set more efficiently.

tyranny of the interface
•

Mandating "def size: Int" for all collections is the fast
track to Glacialville!

•

Countless times have I ﬁxed xs.size != 0

•

Collections are both worlds: all performance/
termination trap, no exploiting of size information!

•

A universal size method must be SAFE and CHEAP

Psp Collections
•

So here is a little of what I would do differently

•

I realized since agreeing to this talk that I may
have to go cold turkey to escape scala’s orbit.
It’s just too frustrating to use.

•

Which means this may never go anywhere

•

But you can have whatever gets done

Conceptual Integrity
trait Collections {
type CC[+X]
type Min[+X]
type Opt[+X]
type CCPair[+X]
type ~>[-V1, +V2]

!

!

}

type
type
type
type
type
type
type
type

//
//
//
//
//

Iso[A]
Map[-A, +B]
FlatMap[-A, +B]
Grouped[A, DD[X]]
Fold[-A, +R]
Flatten[A]
Build[A]
Pure[A]

the overarching container type (in scala: any covariant collection, e.g. List, Vector)
least type constructor which can be reconstituted to CC[X] (scala: GenTraversableOnce)
the container type for optional results (in scala: Option)
some representation of a divided CC[A] (at simplest, (CC[A], CC[A]))
some means of composing operations (at simplest, Function1)
=
=
=
=
=
=
=
=

CC[A] ~> CC[A]
CC[A] ~> CC[B]
CC[A] ~> Min[B]
CC[A] ~> CC[DD[A]]
CC[A] ~> R
CC[Min[A]] ~> CC[A]
Min[A] ~> CC[A]
A ~> CC[A]

trait Relations[A] {
type MapTo[+B] = Map[A, B]
type FoldTo[+R] = Fold[A, R]
type This
= CC[A]
type Twosome
= CCPair[A]
type Self
= Iso[A]
type Select
= FoldTo[A]
type Find
= FoldTo[Opt[A]]
type Split
= FoldTo[Twosome]
}

//
//
//
//
//
//
//
//

//
//
//
//
//
//
//
//

e.g. filter, take, drop, reverse, etc.
e.g. map, collect
e.g. flatMap
e.g. sliding
e.g. fold, but also subsumes all operations on CC[A]
e.g. flatten
for use in e.g. sliding, flatMap
we may not need

an alias incorporating the known A
another one
the CC[A] under consideration
a (CC[A], CC[A]) representation
a.k.a. CC[A] => CC[A], e.g. tail, filter, reverse
a.k.a. CC[A] => A, e.g. head, reduce, max
a.k.a. CC[A] => Opt[A], e.g. find
a.k.a. CC[A] => (CC[A], CC[A]), e.g. partition, span

“Do not multiply entities
unnecessarily”
•

mutable / immutable

•

Seq / Set / Map

•

parallel / sequential

•

view / regular

24 Combinations!

Surface Area Reduced 96%
•

A Set is a Seq without duplicates.

•

A Map is a Set paired with a function K => V.

•

A mutable collection has nothing useful in common with an
immutable collection. Write your own mutable collections.

•

If we can’t get sequential collections right, we have no hope of
parallel collections. Write your own parallel collections.

•

“Views” should be how it always works.

predictability: size matters
scala> def f(xs: Iterable[Int]) = xs.size
f: (xs: Seq[Int])Int
!

// O(1)
scala> f(Set(1))
res0: Int = 1
!

// O(n)
scala> f(List(1))
res1: Int = 1
!

// O(NOES)
scala> f(Stream continually 1)
<ctrl-C>

Asking the right question
SizeInfo
/
Atomic
/
Infinite

Precise

Bounded

Don’t ask unanswerable questions
(Unless you enjoy hearing lies)
scala> val xs = Foreach from BigInt(1)
xs: psp.core.Foreach[BigInt] = unfold(1)(<function1>)

!
scala> xs.size
<console>:22: error: value size is not a member of
psp.core.Foreach[BigInt]
xs.size
^

!
scala> xs.sizeInfo
res0: psp.core.SizeInfo = <inf>

the joy of the invariant leaf
scala> List(1, 2, 3) contains "1"
res0: Boolean = false
!

scala> PspList(1, 2, 3) contains "1"
found
: String("1")
required: Int
PspList(1, 2, 3) contains "1"
^

HEY! MAP NEED NOT BE RUINED!
scala> "abc" map (_.toInt.toChar)
res1: String = abc
!
scala> "abc" map (_.toInt) map (_.toChar)
res2: IndexedSeq[Char] = Vector(a, b, c)
!
// psp to the rescue
scala> "abc".m map (_.toInt) map (_.toChar)
res3: psp.core.View[String,Char] = view of abc
!
scala> res3.force
res4: String = abc

A Scala Corrections Library

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to A Scala Corrections Library (20)

Recently uploaded (20)

A Scala Corrections Library