HPCC Systems vs SAS: The Final Countdown

Innovation and
Reinvention Driving
Transformation
OCTOBER 9, 2018
2018 HPCC Systems® Community Day
Luke Pezet, Archway Health
HPCC Systems vs SAS: The Final Countdown

“Change is the only constant in life”
HPCC Systems vs SAS: The Final Countdown 2
— Heraclitus

Me, Me and Me...at Archway
• Solution Architect with over 15 years of experience
• Worked for Archway Health Advisors ~ 5 years
• Archway helps care providers manage bundled payment programs.
• Needed to process medical claims 5 years ago and chose HPCC Systems over SAS,
Hadoop*, etc.
• New employees brought other technologies, including SAS
3HPCC Systems vs SAS: The Final Countdown

Introduction
HPCC Systems
• Open-source data-intensive computing system platform developed by
LexisNexis Risk Solutions.
• Development started before 2000.
• Scalable Data refinery called Thor and scalable rapid data delivery engine
called ROXIE.
SAS (“Statistical Analysis System”)
• Proprietary software suite developed by SAS Institute that provides advanced
analytics.
• Development started in 1966.

Use Case
• Based on Regression With SAS Chapter 1 - Simple And Multiple Regression web book
from Institute for Digital Research and Education at UCLA.
• It's about data analysis and demonstrates how to use software for regression
analysis. This is not about the statistical basis of multiple regression or which
criterion is best to choose models, etc.
• Data was created by randomly sampling 400 elementary schools from the California
Department of Education's API 2000 dataset.
• Contains a measure of school academic performance as well as other attributes such
as class size, enrollment, poverty, etc.

Helper
SASsy ECL bundle
ecl-bundle install https://github.com/lpezet/SASsy.git
Usage:
IMPORT SASsy;
// OR
IMPORT SASsy.PROC;

Loading data
SAS
DATA scores;
INFILE datalines dsd;
INPUT Name : $9. Score1-Score3 Team ~ $25.
Div $;
DATALINES;
Smith,12,22,46,"Green Hornets, Atlanta",AAA
Mitchel,23,19,25,"High Volts, Portland",AAA
Jones,09,17,54,"Vulcans, Las Vegas",AA
;
ECL
layout := { STRING Name; UNSIGNED Score1;
UNSIGNED Score2; UNSIGNED Score3; STRING
Team; STRING Div; };
scores := DATASET( [ { ‘Smith’,12,22,46,’Green
Hornets, Atlanta’, ‘AAA’ }, { ‘Mitchel’,
23,19,25,’High Volts, Portland’, ‘AAA’ }, { ‘Jones’,
09, 17, 54, ‘Vulcans, Las Vegas’, ‘AA’ } ], layout );

Looking at the data (SAS)
PROC PRINT data=”elemapi” (obs=5);
run;

Looking at the data (ECL)
IMPORT SASsy.PROC;
PROC.PRINT( ElemAPIDS, 5 );
// CHOOSEN( ElemAPIDS, 5 );

PROC CONTENTS data=”elemapi”;
run;

IMPORT SASsy.PROC;
PROC.CONTENTS( ElemAPIDS );

PROC MEANS data=”elemapi”;
var api00 acs_k3 meals full;
run;

IMPORT SASsy.PROC;
PROC.MEANS( oMeans, ElemAPIDS,
'api00,acs_k3,meals,full' );
OUTPUT( oMeans, NAMED('MEANS'));

IMPORT DataPatterns;
DataPatterns.Profile( ElemAPIDS,
features :=
‘fill_rate,best_ecl_types,cardinali
ty,lengths,min_max,mean,std_dev,qua
rtiles,correlations’ );

PROC UNIVARIATE data=”elemapi”;
var acs_k3;
run;

IMPORT SASsy.PROC;
PROC.UNIVARIATE( ElemAPIDS,
'acs_k3' );
Extreme - Lowest Extreme - Highest
Missing Values
Basics

PROC FREQ data=”elemapi”;
tables acs_k3;
run;

IMPORT SASsy.PROC;
PROC.FREQ( ACSK3Freq, ElemAPIDS,
'acs_k3' );
OUTPUT( ACSK3Freq, NAMED(‘Frequency’));

PROC UNIVARIATE data=”elemapi”;
var acs_k3;
histogram / cfill=gray;
run;

IMPORT Visualizer;
PlotData := TABLE( SORT( ElemAPIDS,
acs_k3 ), { STRING label := acs_k3;
COUNT(GROUP); }, acs_k3 );
OUTPUT(oPlotData,
NAMED('PlotData'));
Visualizer.MultiD.Column('myChart',,
'PlotData');

MACROs
SAS
%MACRO MISSINGCHECK(VAR, TYPE);
PROC SQL;
CREATE TABLE &VAR._&TYPE. AS
SELECT DISTINCT CLM_TYPE_1, COUNT(SYSKEY) AS
&VAR._MISSING
FROM OUTPUT.&TYPE.
WHERE &VAR. IS MISSING
GROUP BY CLM_TYPE_1
ORDER BY CLM_TYPE_1;
QUIT;
%MEND MISSINGCHECK;
%MISSINGCHECK(MEMBER_ID, &EPI.GENERAL);
%MISSINGCHECK(CLAIM_ID, &EPI.GENERAL);
%MISSINGCHECK(MS_DRG, &EPI.GENERAL);
%MISSINGCHECK(ADM_DGNS, &EPI.GENERAL);
ECL
MissingCheck( pDS, pField, pMissingValue, pByField ) :=
FUNCTIONMACRO
#UNIQUENAME(tabled)
%tabled% := TABLE( pDS( pField = pMissingValue ), {
pByField; COUNT(GROUP); }, pByField );
#UNIQUENAME(sorted)
%sorted% := SORT( %tabled%, pByField);
RETURN %sorted%;
ENDMACRO;
MissingCheck( ElemAPIDS, meals, ‘’, dnum );
MissingCheck( ElemAPIDS, acs_k3, ‘’, dnum );
MissingCheck( ElemAPIDS, api00, ‘’, dnum );

Multiple Regression (SAS)
PROC REG data="c:sasregelemapi"
model api00 = acs_k3 meals full;
run;

Multiple Regression (ECL)
IMPORT ML_Core;
IMPORT LinearRegression;
IMPORT SASsy;
IndVars := 'acs_k3,meals,full';
DepVars := 'api00';
/* … */
ML_Core.ToField( inddata,
inddataNF, __id__ );
ML_Core.ToField( depdata,
depdataNF, __id__ );
MyOLS := LinearRegression.OLS(
inddataNF, depdataNF );
MyModel := MyOLS.GetModel;
SASsy.Utils.reg_report_on_all(
MyOLS, MyModel, inddataNF );

More
ECL Machine Learning Library
• Statistics (e.g. Means, Std Deviation, Modes, Medians, NTiles, etc.)
• Regression
• Clustering (e.g. K-Means)
• Classification (e.g. Logistic Regression, Decision Trees, Perceptron, etc.)
• Unstructured Data (Tokenize, Transform, CoLocation)
• Association (e.g. AprioriN)
• Matrix Manipulation

Today
HPCC Systems used to process data at scale and on a more frequent basis
• Process Medical Claims using Thor and deliver results using Roxie
• Run ETL/ELT processes to load, clean, prepare data
• Run more advanced processing to generate outputs (Bundle Engine)
• Clusters of 8+ nodes
SAS used to run research, exploratory data analysis and modeling.
• Uses HPCC outputs as input
• Single instance
• Restricted on CPU/RAM

Tomorrow
HPCC Systems
• Still run ETL/ELT processes to load, clean, prepare data
• Run processes that need to happen more frequently
• Porting more Advanced Data Analysis And Modeling features to ECL
• Make it easier to create clusters to make experimentation effortless
SAS
• 1 server
• R&D for now
• Validate/compare results with HPCC Systems

HPCC Systems vs SAS: The Final Countdown

More Related Content

What's hot (12)

Similar to HPCC Systems vs SAS: The Final Countdown (20)

More from HPCC Systems (20)

Recently uploaded (20)

HPCC Systems vs SAS: The Final Countdown