a million bots can't be wrong

a million bots can't be wrong
@remeniuk, Viaden Media
#ScalaSBP, 18-05-2012

In Viaden our aim is to make the best poker ever

we know that
performance tests should be
the first-class citizens

and kill 2 birds with one stone,
using bots for testing

#1 we can emulate 50k
players using just one
medium EC2 instance

#2 bots are interactive,
so client teams can use
them in development,
and QA for testing

why Scala and Akka is a perfect
choice for making bots?

actors are !(interactive)
straightforward remoting
simple scalability/clustering
~30 minutes to make a DSL and CLI
with Scala and SBT

..., but, I have to warn you ...

4 dead simple tips
for keeping your sanity
when you do asynch with Akka 2.0

typical approach in Akka 1.x

lobby

desk
login
l ink
bot

tourney

Akka 1.x: actors have a long lifecycle

lobby

desk
l ink nk
bot li
un
l ink
play game
bot
tourney

Akka 1.x: actors have a long lifecycle

lobby

desk
bot

k
lin
bot

un
tourney
play tournament link
bot

in Akka 2.0 the world has changed

props
paths
new
supervision
actors

now, you're forced to do "the right thing" (c)

lobby
desk
tournament

DeskBot
desk

DeskBot

IdleBot

all actors are supervised

lobby

login
desk

easy come

lobby

desk
play game
IdleBot

easy go (when supervisor to be changed)

lobby

desk
IdleBot dies
DeskBot

borns

class Lobby extends Actor {

case Login(username, password) =>
context.actorOf(Props(new IdlePokerBot(...)))

case JoinAnyDesk(props) => findDesk ! JoinDesk(props)

}

class Desk extends Actor {

case JoinDesk(botProps)=>
context.actorOf(Props(new DeskPokerBot(botProps)))

}

class IdlePokerBot(props: BotProps) extends Actor {

case PlayNow =>
context.parent ! JoinAnyDesk(props); context.stop(self)

}

Props Pattern - "the soul" of an actor

IdleBot

BotProps
Props remains alive
between actor
"reincarnations"

DeskBot

BotProps

case class BotProperties(id: Long,
login: String,
script: Seq[BotEvent],
aggressiveness: Int,
sessionId: Protocol.SessionId)

class IdlePokerBot(val botProperties: BotProperties)
extends Bot[Poker]

class DeskPokerBot(val botProperties: BotProperties)
extends Bot[Poker]

when you know, who's supervising, life's
simple

akka://gpdev/user/lobby/player1234

akka://gpdev/user/lobby/desk1/player1234

akka://gpdev/user/lobby/tournament1/desk1/
player1234

Bad news
ActorRegistry, actor UUID
were removed from Akka

but what should I do, now,
when I don't know, where
to look for my bot?

you can make your own registry
(using Extensions, backed with a
distributed data structure)...

or, use the full power of location transparency

Projection
lobby Manager

Projection
desk var location:
ActorPath

DeskBot

/projection/player123
IdleBot /lobby/desk123/player123

class Projection(var container: ActorPath) extends Actor {

def receive = {
case newContainer: ActorPath => container = newContainer
case msg =>
context.actorFor(container.child(self.path.name)) !
msg
}

}

class ProjectionManager extends Actor {

def receive = {
case Add(actor) => context.actorOf(Props(new
Projection(actor.path.parent)), actor.path.name)
case Forward(msg, path) => context.actorFor(path) ! msg
}

}

projectionManager ! Add(actorRef)
projectionManager ! Forward("ping", "actor1")

class Projection(var container: ActorPath) extends Actor {

def receive = {
case newContainer: ActorPath => container = newContainer
case msg =>
context.actorFor(container.child(self.path.name)) ! msg
}

}


def receive = {
case Forward(msg, path) => context.actorFor(path) ! msg
}

}

system.actorFor(projectionManager.path.child("actor" + i)) !
"ping"


def receive = {
case Forward(msg, path) =>
context.actorSelection("../*/" + path) ! msg
}

}

val projectionManager = system.actorOf(Props
[ProjectionManagerRoutee]
.withRouter(RoundRobinRouter(resizer = Some
(DefaultResizer(lowerBound = 10, upperBound = 20)))),
"projection")

projectionManager ! Forward("ping", "actor1")

case class CustomRouter(n: Int, routerDispatcher: String =
DefaultDispatcherId, supervisorStrategy: SupervisorStrategy =
defaultStrategy) extends RouterConfig {

def createRoute(props: Props, provider: RouteeProvider) = {

provider.registerRoutees((1 to n).map(i =>
provider.context.actorOf(Props[ProjectionManager], i.
toString)))

def destination(sender: ActorRef, path: String) =
List(Destination(sender,
provider.routees(abs(path.hashCode) %
n)))

{
case m@(sender, Add(actor)) ⇒
destination(sender, actor.path.name)
case m@(sender, Forward(_, name)) ⇒
destination(sender, name)
}
}

}

tip #3: don't do anything stupid

you've tried all the options, system load is fine,
only 1/10 of the heap is used, but you still can
start not more than 1k bots!?

your actor is lacking of throughput?

wait before adding pools
share responsibility!
one fine-grained actor is enough in 99% of the cases

100-300 threads are serving 300 bots!?

spawn futures, backed with standalone
[bounded] pools, for blocking operations

class ThirdPartyWrapper extends Actor {

case F(x) => sender ! thirdPartyService.f(x)
// call to a function that takes a lot of time to
// complete

}

class ThirdPartyWrapper extends Actor {

case F(x) => val _sender = sender
Future(thirdPartyService.f(x)).map(_sender ! _)
// ugly, but safe, and perfectly right
}

use separate dispatchers
lobby-dispatcher projection-manager-dispatcher
PinnedDispatcher BalancingDispatcher

lobby Projection
Manager
desk
Projection

DeskBot

projection-dispatcher
container-dispatcher desk-bot-dispatcher Dispatcher
Dispatcher Dispatcher

GOTCHA: Akka successfully bootstraps, even if your
dispatcher is not configured, or the config is wrong

Always check the logs to make sure that dispatchers are used!

[WARN][gpdev-akka.actor.default-dispatcher-1] [Dispatchers]
Dispatcher [bot-system.container-dispatcher] not
configured, using default-dispatcher
[WARN][gpdev-bot-system.container-dispatcher-1]
[PinnedDispatcherConfigurator] PinnedDispatcher [bot-
system.lobby-dispatcher] not configured to use
ThreadPoolExecutor, falling back to default config.

[DEBUG][gpdev-akka.actor.default-dispatcher-24] [akka:
//gpdev/user/poker/lobby] logged in
[DEBUG][gpdev-akka.actor.default-dispatcher-14] [akka:
//gpdev/user/poker/projeciton/$g/player20013] starting
projection...

how to measure?

Metrics - pushes various collected metrics to Graphite
Carbon and Graphite - gather metrics, and expose them via web
interface

object BotMetrics {

1. val loggedInCount = new Counter(Metrics.newCounter(classOf[Lobby
[_]],
"logged-in-count"))

3. GraphiteReporter.enable(1, TimeUnit.MINUTES, "localhost", 2003)

}

class Lobby extends Actor {

2. case Login(username, pass) => BotMetrics.loggedInCount += 1

}

1. add logged-in user counter 4.
2. update it
3. enable reporting to
Graphite
4. build a graph in Grtaphite

what to measure?

- mailbox size1
- throughput
- time, before the message is processed (both in
actor and future)2
- time to process a message
- count of threads
- actor pool size
- heap size

1
requires implementation of a custom mailbox that can expose mailbox size
2
every message should be stuffed with a timestamp

how to tune dispatchers?

VisualVM - thread timeline shows, if thread polls behind dispatchers
are used effectively

don't neglect old good logging

[ERROR][05/06/2012 12:55:43.826] [gpdev-bot-system.
desk-bot-dispatcher-7]
[akka://gpdev/user/
poker/lobby/tournament5382577/desk109129/player2012
1]
unprocessed game event: GameEvent(CHAT,None,None)

we're hiring!
viaden.com/careers/vacancies.html

a million bots can't be wrong

More Related Content

What's hot (20)

Viewers also liked (10)

Similar to a million bots can't be wrong (20)

More from Vasil Remeniuk (20)

Recently uploaded (20)

a million bots can't be wrong