Perl for visualization

Perl for
Visualization
YAPCASIA:2013 9/21
113年9月21日土曜日

print Dumper $me
{
twitter => “@muddydixon”,
organization => “NIFTY”,
job => “low spec full stack engineer”,
skill => [
“data collecting”,
“data cleansing”,
“visualization”,
]
}

Today’s Topic

Key of Visualization
STORIES
are main concept of visualization
are buried in data
enhance your business

Visualization
Two purposes:
1.Explanatory visualization
2.Exploratory visualization

Explanatory visualization
communicating information
clearly and effectively

Can you find
STORIES?

加齢による
技術向上
The more use, the higher skill

ひとりで
できない
10年選手は
いなかったEverybody who touch perl 10 years use perl :)

7-9年目に
がんばろう
7-10 years users practice more!

ここから
読み取る
のはムリ
We cannot find STORIES from text log

This is the
POWER of
Explanatory
visualization 2013年9月21日土曜日

I / You can
inform STORIES in data
to my / your colleague
to my / your boss
to my / your audiences

Exploratory visualization
Visualization allow you to find
STORIES from data
Data Mining

via（道具としての）データサイエンティストのつかい方

Problems
Data Mining Engineer
a few domain specific knowledge
Domain Specialist
a few mining skill

GOAL
Business
Success
Domain Specific Knowledge
Data Mining Skill

How to apply
to
Data Mining Processes

What is Glue between two?
What is Ladder to Goal?

GOAL
Business
Success
Data Mining Skill

Ans. Visualization

GOAL
Business
Success
Data Mining Skill
Trends
five-number
summary
Co-occurence
Mosaic map
Flow chart
Parallel chart

Domain Specific Expert
Data Mining Engineer

It looks like
communication
to your data and
your collaborators

Exploratory Visualization
for Exploratory Data Analysis
do before model processing or
fitting, testing
with Domain specific knowledge

How to communicate ?
R
S
S-Plus
SYSTAT
SPSS
Pands

We have Perl !!

Perl Data Language

Perl Data Language
standard Perl the ability to compactly store and
speedily manipulate the large N-dimensional
data arrays which are the bread and butter of
scientific computing.
PDL turns Perl into a free, array-oriented,
numerical language similar to (but, we believe,
better than) such commercial packages as IDL
and MatLab. One can write simple perl

Perl Data Language

Perl Data Language
We want
Hash Object

#!/usr/bin/env perl
use strict;
use warnings;
use Data::Dumper;
use PDL;

my $obj;
$obj = pdl([[1,2,3],[4,5,6]]);
print $obj;

# [
# [1 2 3]
# [4 5 6]
# ]

$obj = pdl([{a => 1, b => 2, c => 3}, {a => 4, b => 5, c => 6}]);
# Hash given as a pdl - but not {PDL} key! at Basic/Core/
Core.pm.PL (i.e. PDL::Core.pm) line 1292.
# 工エェ(´Д`)ェエ工

Data::Cube
is an array of multi-dimensional data
has OLAP methods: dice, slice, etc.

Slice Rotate
Dice
Drill-down

え・・・ 4813年9月21日土曜日

Data::Cube released!
processing Array of Hash Object
add / remove rollup measure
add / remove dimension
add / remove hierarchy

Data::Cube 1. Data
Date Country SalesPerson Product Units Unit_Cost Total
3/15/2005 US Sorvino Pencil 56 2.99 167.44
3/7/2006 US Sorvino Binder 7 19.99 139.93
8/24/2006 US Sorvino Desk 3 275.00 825.00
9/27/2006 US Sorvino Pen 76 1.99 151.24
5/22/2005 US Thompson Pencil 32 1.99 63.68
10/14/2006 US Thompson Binder 57 19.99 1139.43
4/18/2005 US Andrews Pencil 75 1.99 149.25

Data::Cube 2. Usage
my $file = shift;
my $data = Text::CSV::Slurp->load(file => $file);
my $cube;

say "============================================================";
say "raw data size: ".(scalar @$data)."n";

say "n============================================================";
$cube = new Data::Cube("experience");
$cube->put($data);
say Dumper $cube->rollup(noValues => 1);

say "n============================================================";
$cube->add_dimension("skill");

Data::Cube 2. Usage
my $file = shift;
my $data = Text::CSV::Slurp->load(file => $file);
my $cube;

say "============================================================";
say "raw data size: ".(scalar @$data)."n";

say "n============================================================";
$cube = new Data::Cube("experience");
$cube->put($data);

say "n============================================================";
$cube->add_dimension("skill");
たったのこれだけ

Data::Cube 3. Results
$VAR1 = [
{ 'count' => 150, 'dim' => 10年以上 },
{ 'count' => 76, 'dim' => 1~3年 },
{ 'count' => 32, 'dim' => 1年未満 },
{ 'count' => 93, 'dim' => 4~6年 },
{ 'count' => 43, 'dim' => 7~9年 }
];

Data::Cube 4. Results$VAR1 = [
{
'dim' => 10年以上, 'values' => [
{ 'count' => 79, 'dim' => 上級者（一通り書こうと思えば書ける。わかんなかったらとりあえずソースコード読んじゃう） },
{ 'count' => 71, 'dim' => 中級者（だいたいやりたいことはできるが、本やサイトを頼りにして確認したりする） }]},
{
'dim' => 1~3年, 'values' => [
{ 'count' => 56, 'dim' => 中級者（だいたいやりたいことはできるが、本やサイトを頼りにして確認したりする） },
{ 'count' => 12, 'dim' => 初級者（人に聞いたりしないとなかなか書けない） }]},
{
'dim' => 1年未満, 'values' => [
{
'dim' => 4~6年, 'values' => [
{
'dim' => 7~9年, 'values' => [
{ 'count' => 1, 'dim' => 初級者（人に聞いたりしないとなかなか書けない） }]}
];

Data::Cube 5. Measure
my $cube = new Data::Cube("Country");
$cube->put($data);
$cube->add_measure("sum", sub {
my $sum = 0;
foreach my $d (@_){ $sum += $d->{Units};}
$sum;});
$cube->add_measure("mean", sub {
my $sum = 0;
$sum / (scalar @_);});
print Dumper $cube->rollup(noValues => 1);

$cube->add_dimension("Product");

my $cube = new Data::Cube("Country");
$cube->put($data);
$cube->add_measure("sum", sub {
my $sum = 0;
$sum;});
$cube->add_measure("mean", sub {
my $sum = 0;
$sum / (scalar @_);});

$cube->add_dimension("Product");
たったのこれだけ

Data::Cube 6. Measure$VAR1 = [
{
'count' => 10,
'dim' => 'CA',
'sum' => 687,
'mean' => '68.7'
},
{
'count' => 11,
'dim' => 'UK',
'sum' => 764,
'mean' => '69.4545454545455'
},
{
'count' => 22,
'dim' => 'US',
'sum' => 1103,
'mean' => '50.1363636363636'
}
];

$VAR1 = [
{
'values' => [
{ 'count' => 5, 'dim' => 'Binder', 'sum' => 288, 'mean' => '57.6'},
{ 'count' => 1, 'dim' => 'Pen', 'sum' => 51, 'mean' => 51},
{ 'count' => 1, 'dim' => 'PenSet', 'sum' => 61, 'mean' => 61},
{ 'count' => 3, 'dim' => 'Pencil', 'sum' => 287, 'mean' => '95.6666666666667'}
],
'dim' => 'CA'
},
{
'values' => [
{ 'count' => 4, 'dim' => 'Binder', 'sum' => 242, 'mean' => '60.5'},
{ 'count' => 1, 'dim' => 'Pen', 'sum' => 12, 'mean' => 12},
{ 'count' => 3, 'dim' => 'PenSet', 'sum' => 205, 'mean' => '68.3333333333333'},
{ 'count' => 3, 'dim' => 'Pencil', 'sum' => 305, 'mean' => '101.666666666667'}
],
'dim' => 'UK'
},

to Visualization
$cube -> HASH ref -> JSON
JSON -> d3.js -> visualization

Summary

Summary
せっかく紹介しまし
たが、僕はこの手の
処理はRでやります

> data = read.csv("./data/perl5census2013.csv")
> summary(data)
address experience
関東地方 :292 1~3年 : 76
近畿地方 : 39 10年以上:150
中部地方 : 23 1年未満 : 32
九州地方・沖縄: 17 4~6年 : 93
北海道地方 : 12 7~9年 : 43
東北地方 : 6
(Other) : 5
skill
初級者（人に聞いたりしないとなかなか書けない） : 38
上級者（一通り書こうと思えば書ける。わかんなかったらとりあえずソースコード読んじゃう）:131
中級者（だいたいやりたいことはできるが、本やサイトを頼りにして確認したりする） :225
frequencyatbusiness frequencyatprivate
Min. : 1.000 Min. : 1.000
1st Qu.: 2.000 1st Qu.: 2.000
Median : 6.000 Median : 5.000
Mean : 5.779 Mean : 5.388
3rd Qu.:10.000 3rd Qu.: 8.000
Max. :10.000 Max. :10.000
versionmanager
システム Perl (rpm, yum, インストール済みのperl等) :136
システム Perl (rpm, yum, インストール済みのperl等), perlbrew : 92
perlbrew : 76
システム Perl (rpm, yum, インストール済みのperl等), perlbrew, plenv: 14
perlbrew, plenv : 12
plenv : 9
(Other) : 55
2行6213年9月21日土曜日

Summary: but
継続的な可視化を
専門家に見せるとき
圧倒的な優位性！

Summary: Example
Time Series
sales, repeat rate, DAU, system info, activities
Effects of trial / campaign
attribution, condition, cost, cash back, etc.

Perl for visualization

More Related Content

What's hot (19)

Viewers also liked (17)

Perl for visualization