MongoDB World 2014 - BillRun, Billing on top of MongoDB

MongoDB World 2014
by S.D.O.C. Ltd.
Billing on top of MongoDB
Ofer Cohen

Who Am I ?
● Open source evangelist
● Former Board member of OpenSourceMatters
(Non-profit organization behind Joomla)
● S.D.O.C. Ltd. Co-Founder

FUD
● FUD: Fear, Uncertainty and Doubt
● Tactic used in sales, marketing, public
relations, politics and propaganda.
Wikipedia

Preferred name for the presentation

S.D.O.C. Ltd. vision
We increase business success through great
open source technologies and services
○ Open & Transparent
○ Don't reinvent the wheel
○ High Quality
○ Lean & Effective

How did we get to billing (history)
● Client: Golan Telecom (Israel)

● Client: Golan Telecom
○ New & lean player in the market
○ 0=>~1M subscribers in few years
○ Limited resources & loves open source

● Client: Golan Telecom
○ Start up environment from Day 1
○ Short and aggressive time to market
○ Unlimited plan for ~25$
○ Customer can do almost everything using the website
■ Obviously requires 24/7 uptime

● Start with Anti-Fraud solution
● 2 Different data structure, 2 separated tables
○ Outgoing calls (MOC)
○ incoming calls (MTC)
○ SMS (duration=0 means SMS)

Anti-Fraud in RDBMS
How did it look like? Example code...
$base_query = "SELECT imsi FROM moc WHERE callEventStartTimeStamp >=" . $start
. " UNION SELECT imsi FROM mtc WHERE callEventStartTimeStamp >=" . $start
$base_query = "SELECT imsi FROM (" . $base_query . ") AS qry ";
if (isset($args['imsi']))
$base_query .= "WHERE imsi = '" . $this->_connection->real_escape_string($args['imsi']) . "'";
$base_query .= "GROUP BY imsi ";
$mtc_join_query = "SELECT 'mtc' AS type, imsi, SUM(callEventDuration) AS duration, SUM(CEILING(callEventDuration/60)*60) AS duration_round "
. ", SUM(chargeAmount) charge "
. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)='972', callEventDuration, IF(CHAR_LENGTH(callingNumber)<=10, callEventDuration, 0))) AS israel_duration
"
. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)='972', CEILING(callEventDuration/60)*60, IF(CHAR_LENGTH(callingNumber)<=10, CEILING
(callEventDuration/60)*60, 0))) AS israel_duration_round "
. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)!='972', IF(CHAR_LENGTH(callingNumber)>10, callEventDuration, 0), 0)) AS non_israel_duration "
. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)!='972', IF(CHAR_LENGTH(callingNumber)>10, CEILING(callEventDuration/60)*60, 0), 0)) AS
non_israel_duration_round "
. ", SUM(IF(callEventDuration = 0, 1, 0)) AS sms_count "

Anti-Fraud in RDBMS
. "FROM mtc "
. "WHERE callEventStartTimeStamp >=" . $start . " "
. "GROUP BY type, imsi";
$moc_join_query = "SELECT 'moc' AS type, imsi, SUM(callEventDuration) AS duration, SUM(CEILING(callEventDuration/60)*60) AS duration_round "
. ", SUM(chargeAmount) charge "
. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)='972', callEventDuration, IF(CHAR_LENGTH(connectedNumber)<=10, callEventDuration, 0))) AS
israel_duration "
. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)='972', CEILING(callEventDuration/60)*60, IF(CHAR_LENGTH(connectedNumber)<=10, CEILING
(callEventDuration/60)*60, 0))) AS israel_duration_round "
. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)!='972', IF(CHAR_LENGTH(connectedNumber)>10, callEventDuration, 0), 0)) AS non_israel_duration "
. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)!='972', IF(CHAR_LENGTH(connectedNumber)>10, CEILING(callEventDuration/60)*60, 0), 0)) AS
non_israel_duration_round "
. ", SUM(IF(callEventDuration = 0, 1, 0)) AS sms_count "
. "FROM moc "
. "WHERE callEventStartTimeStamp >=" . $start . " "
. "GROUP BY type, imsi";

Anti-Fraud in RDBMS
$group_query = "SELECT base.imsi, moc.duration AS moc_duration, moc.charge AS moc_charge, "
. "mtc.duration AS mtc_duration, mtc.charge AS mtc_charge, "
. "mtc.duration_round AS mtc_duration_round, moc.duration_round AS moc_duration_round, "
. "moc.israel_duration AS moc_israel_duration, moc.non_israel_duration AS moc_non_israel_duration, "
. "moc.israel_duration_round AS moc_israel_duration_round, moc.non_israel_duration_round AS
moc_non_israel_duration_round, "
. "mtc.israel_duration AS mtc_israel_duration, mtc.non_israel_duration AS mtc_non_israel_duration, "
. "mtc.israel_duration_round AS mtc_israel_duration_round, mtc.non_israel_duration_round AS
mtc_non_israel_duration_round, "
. "mtc.sms_count AS mtc_sms_count, moc.sms_count AS moc_sms_count "
. "FROM "
. "( " . $base_query . " ) AS base "
. " LEFT JOIN (" . $mtc_join_query . " ) AS mtc ON base.imsi = mtc.imsi "
. " LEFT JOIN (" . $moc_join_query . " ) AS moc ON base.imsi = moc.imsi " ;

Anti-Fraud in RDBMS
How did it feel…?

After moving to Mongo
● One main collection for 2 types (MOC & MTC)
● Aggregation framework is much more simple

$base_match = array(
'$match' => array(
'source' => 'nrtrde',
'unified_record_time' => array('$gte' => new MongoDate($charge_time)),
)
);
$where = array(
'$match' => array(
'record_type' => 'MOC',
'connectedNumber' => array('$regex' => '^972'),
'event_stamp' => array('$exists' => false),
'deposit_stamp' => array('$exists' => false),
'callEventDurationRound' => array('$gt' => 0), // not sms
),
);
$group = array(
'$group' => array(
"_id" => '$imsi',
"moc_israel" => array('$sum' => '$callEventDurationRound'),
'lines_stamps' => array('$addToSet' => '$stamp'),
),
);
$project = array(
'$project' => array(
'imsi' => '$_id',
'_id' => 0,
'moc_israel' => 1,
'lines_stamps' => 1,
),
);

$having = array(
'$match' => array(
'moc_israel' => array('$gte' => Billrun_Factory::config()->getConfigValue('nrtrde.thresholds.moc.israel'))
),
);
$moc_israel = $lines->aggregate($base_match, $where, $group, $project, $having);
//sms out to all numbers
$where['$match']['record_type'] = 'MOC';
$where['$match']['callEventDurationRound'] = 0;
$group['$group']['sms_out'] = $group['$group']['mtc_all'];
unset($group['$group']['mtc_all']);
unset($having['$match']['mtc_all']);
$group['$group']['sms_out'] = array('$sum' => 1);
$having['$match']['sms_out'] = array('$gte' => Billrun_Factory::config()->getConfigValue('nrtrde.thresholds.smsout'));
$project['$project']['sms_out'] = 1;
unset($project['$project']['mtc_all']);
$sms_out = $lines->aggregate($base_match, $where, $group, $project, $having);

What is billing?
● Group of processes of communications
service providers
● responsible to collect consumption data
● calculate charging and billing information
● produce bills to customers
● process their payments and manage debt
collection
Wikipedia

What is billing?
● This is how we see it
● The KISS way

Telecom today - database challenges
● Telecom situation world wide today:
○ Unlimited packages, extend 3g usage, with high
competition
○ High-volume - 4g, LTE
○ Different Events - different data structure

5 *main* data-structures (CDR) from different sources
● NSN - Calls
● SMSC - SMS
● MMSC - MMS
● GGSN - Data
● TAP3 - International usage
Billing and MongoDB - Pros

NSN Record
> db.lines.findOne({"usaget" : "call", aid:XXXXXXX})
{
"_id" : ObjectId("52bafd818f7ac3943a8b96dc"),
"stamp" : "87cea5dec484c8f6a19e44e77a2e15b4",
"record_type" : "01",
"record_number" : "13857000",
"record_status" : 0,
"exchange_id" : "000006270000",
"call_reference" : "700227d9a3",
"urt" : ISODate("2013-12-25T15:09:15Z"),
"charging_start_time" : "20131225170915",
"charging_end_time" : "20131225171517",
"call_reference_time" : "20131225170914",
"duration" : 362,
"calling_number" : "972546918666",
"called_number" : "26366667",
"call_type" : "3",
"chrg_type" : "0",
"called_imsi" : "000030000000000",
"imsi" : "000089000101076",
"in_circuit_group_name" : "",
"in_circuit_group" : "",
"out_circuit_group_name" : "NBZQZA8",
"out_circuit_group" : "0503",
"tariff_class" : "000000",
"called_number_ton" : "6",
"org_dur" : 362,
"type" : "nsn",
"source" : "binary",
"file" : "CF0322.DAT",
"log_stamp" : "0a3ffdb7c9ccc175e38be2fcc00f8c28",
"process_time" : "2013-12-25 17:35:22",
"usaget" : "call",
"usagev" : 362,
"arate" : DBRef("rates", ObjectId("521e07fcd88db0e73f0001c9")),
"aid" : XXXXXXX,
"sid" : YYYYY,
"plan" : "LARGE",
"aprice" : 0,
}

SMS Record
> db.lines.findOne({"usaget" : "sms", aid:XXXXXXX})
{
"_id" : ObjectId("52e52ff1d88db071648b4ad7"),
"stamp" : "9328f3aaa114aaba910910053a11b3e8",
"record_type" : "1",
"calling_number" : "000972546918666",
"calling_imsi" : "000089200000000",
"calling_msc" : "000972000000000",
"billable" : "000000000000000",
"called_number" : "000972547655380",
"called_imsi" : "425089109386379",
"called_msc" : "000972586279101",
"message_submition_time" : "140125192517",
"time_offest" : "02",
"message_delivery_time" : "140125192519",
"time_offest1" : "02",
"cause_of_terminition" : "100",
"call_reference" : "5137864939035049",
"message_length" : "050",
"concatenated" : "1",
"concatenated_from" : "09",
"source" : "separator_field_lines",
"type" : "smsc",
"log_stamp" : "a5950686e364d1400c13dd1857c3340e",
"file" : "140125192403_5735golan.cdr",
"process_time" : "2014-01-26 17:53:21",
"urt" : ISODate("2014-01-25T17:25:17Z"),
"usaget" : "sms",
"usagev" : 1,
"arate" : DBRef("rates", ObjectId("521e07fcd88db0e73f0001db")),
"aid" : XXXXXXX,
"sid" : YYYYY,
"plan" : "LARGE",
"aprice" : 0,
"usagesb" : 4,
"billrun" : "201402"
}

Data Record
> db.lines.findOne({"usaget" : "data", aid:XXXXXXX})
{
"_id" : ObjectId("539076678f7ac34a1d8b9367"),
"stamp" : "8a9f891ec85c5294c974a34653356055",
"imsi" : "400009209100000",
"served_imsi" : "400009209100000",
"ggsn_address" : "XX.XX.144.18",
"charging_id" : "2814645234",
"sgsn_address" : "XX.XX.145.9",
"served_pdp_address" : "XX.XX.237.95",
"urt" : ISODate("2014-06-05T09:34:47Z"),
"record_opening_time" : "20140605123447",
"ms_timezone" : "+03:00",
"node_id" : "GLTNGPT",
"served_msisdn" : "00002546918666",
"fbc_uplink_volume" : 61298,
"fbc_downlink_volume" : 217304,
"rating_group" : 0,
"type" : "ggsn",
"source" : "binary",
"file" : "GLTNGPT_-_0000056580.20140605_-_1251+0300",
"log_stamp" : "45a227ced1098bc76a44774eae04eb67",
"process_time" : "2014-06-05 16:39:40",
"usaget" : "data",
"usagev" : 278602,
"arate" : DBRef("rates", ObjectId("521e07fcd88db0e73f000200")),
"apr" : 0.020264916503503202,
"aid" : XXXXXXX,
"sid" : YYYYY,
"plan" : "LARGE",
"aprice" : 0,
"usagesb" : 478682116,
"billrun" : "201406"
}

TAP3 Record (intl roaming)
> db.lines.findOne({type:"tap3", aid:9073496})
{
"_id" : ObjectId("538d9ac98f7ac3e17d8b4fd6"),
"stamp" : "8f6cdc8662307ee2ed951ce640a585b5",
"basicCallInformation" : {
"GprsChargeableSubscriber" : {
"chargeableSubscriber" : {
"simChargeableSubscriber" : {
"imsi" : "400009209100000"
}
},
"pdpAddress" : "XX.XX.227.158"
},
"GprsDestination" : {
"AccessPointNameNI" : "internet.golantelecom.net.il"
},
"CallEventStartTimeStamp" : {
"localTimeStamp" : "20140529205131",
"TimeOffsetCode" : 0
},
"TotalCallEventDuration" : 163
},
"LocationInformation" : {
"gprsNetworkLocation" : {
"RecEntityCodeList" : {
"RecEntityCode" : [
"",
"u0000"
]
},
"LocationArea" : 00001,
"CellId" : 0001
},
"GeographicalLocation" : {
"ServingNetwork" : "MMMM"
}
},
"ImeiOrEsn" : false,
"GprsServiceUsed" : {
"DataVolumeIncoming" : 195120,
"DataVolumeOutgoing" : 48600,
"ChargeInformationList" : {
"ChargeInformation" : {
"ChargedItem" : "X",
"ExchangeRateCode" : 0,
"ChargeDetailList" : {
"ChargeDetail" : {
"ChargeType" : "00",
"Charge" : 001,
"ChargeableUnits" : 100000,
"ChargedUnits" : 100000,
"ChargeDetailTimeStamp" : {
"localTimeStamp" : "20140529205131",
"TimeOffset" : 0
}
}
}
}
}
},
"OperatorSpecInfoList" : {
"OperatorSpecInformation" : [
"00000000.0000",
"00000000.0000",
"00000000.0000"
]
},
"record_type" : "e",
"urt" : ISODate("2014-05-29T18:51:31Z"),
"tzoffset" : "+0200",
"imsi" : "400009209100000",
"serving_network" : "DEUE2",
"sdr" : 0.0001,
"exchange_rate" : 1.12078,
"type" : "tap3",
"file" : "CDBELHBISRGT02253",
"log_stamp" : "a0ad109c6e795f6c1feeef9ef649d937",
"process_time" : "2014-06-03 12:50:08",
"usaget" : "data",
"usagev" : 243720,
"arate" : DBRef("rates", ObjectId
("521e07fed88db0e73f000219")),
"apr" : 0.46640616,
"aid" : 9073496,
"sid" : 78288,
"plan" : "LARGE",
"out_plan" : 243720,
"aprice" : 0.46640616,
"usagesb" : 39139746,
"billrun" : "201406"
}

Loose coupling compared to traditional billing components
Old
w/o MongoDb New
with MongoDb

● One call can be spread on few CDRs
● BillRun merges records only on presentation
layer or export
● No DB-level aggregation nor accumulation
○ No need for mediation
● Use billing and monitoring CDR in one system
database

Sophisticated rating module
● Rating module depends on CDR structure
● Easy to implement recurring rating

MongoDB advantages with Billing
Invoice JSON2PDF process
● Use json as metadata for the PDF
● Easy to export
● Fast processing

Invoice metadata
> db.billrun.findOne({billrun_key:"201405", aid:9073496})
{
"aid" : NumberLong(9073496),
"subs" : [
{
"sid" : NumberLong(78288),
"subscriber_status" : "open",
"current_plan" : DBRef("plans", ObjectId("51bd8dc9eb2f76d2178dd3dd")),
"next_plan" : DBRef("plans", ObjectId("51bd8dc9eb2f76d2178dd3dd")),
"kosher" : false,
"breakdown" : {
"in_plan" : {
"base" : {
"INTERNET_BILL_BY_VOLUME" : {
"totals" : {
"data" : {
"usagev" : NumberLong(1975547725),
"cost" : NumberLong(0),
"count" : NumberLong(1352)
}
},
"vat" : 0.18
},
"IL_MOBILE" : {
"totals" : {
"sms" : {
},
"call" : {
}
},
"vat" : 0.18
},
"IL_FIX" : {
"totals" : {
"call" : {
}
},
"vat" : 0.18
},
"INTERNAL_VOICE_MAIL_CALL" : {
"totals" : {
"call" : {
}
},
"vat" : 0.18
},
"service" : {
"cost" : 83.898305085,
"vat" : 0.18
}
},
"intl" : {
"KT_USA_NEW" : {
"totals" : {
"call" : {
}
},
"vat" : 0.18
}
}
},

Invoice metadata
"credit" : {
"refund_vatable" : {
"CRM-REFUND_PROMOTION_1024-BILLRUN_201405" : -33.898305084746
}
}
},
"lines" : {
"data" : {
"counters" : {
"20140425" : {
"aprice" : NumberLong(0),
"plan_flag" : "in"
},
…. /* data by date really long, so let’s cut it from this demonstration */
}
}
},
"totals" : {
"vatable" : 50.000000000254005,
"before_vat" : 50.000000000254005,
"after_vat" : 59.00000000029973
},
"costs" : {
"credit" : {
"refund" : {
"vatable" : -33.898305084746
}
},
"flat" : {
"vatable" : 83.898305085
}
}
},
{
"sid" : NumberLong(354961),
"subscriber_status" : "open",
"current_plan" : DBRef("plans", ObjectId("51bd8dc9eb2f76d2178dd3de")),
"next_plan" : DBRef("plans", ObjectId("51bd8dc9eb2f76d2178dd3de")),
"kosher" : false,
"breakdown" : {
"in_plan" : {
"base" : {
"service" : {
"cost" : 8.466101695,
"vat" : 0.18
}
}
}
},
"totals" : {
"vatable" : 8.466101695,
"before_vat" : 8.466101695,
"after_vat" : 9.9900000001
},
"costs" : {
"flat" : {
"vatable" : 8.466101695
}
}
}
],
"vat" : 0.18,
"billrun_key" : "201405",
"totals" : {
"before_vat" : 58.466101695254004,
"after_vat" : 68.99000000039973,
"vatable" : 58.466101695254004
},
"_id" : ObjectId("5382fd3cd88db0c31a8b74cc"),
"invoice_id" : NumberLong(14738102),
"invoice_file" : "201405_009073496_00014738102.xml"
}

Infrastructure
BETA
First Production
Today
Data center 1
Data center 2
Data center 1
Data center 2 Data center 1
Data center 2
Archive

App stack
BillRun application
BillRun core
PHP YAF
framework
Zend and more libraries
MongoDB
Web service Cli/Cron services
Mongodloid https://github.com/BillRun/Mongodloid
https://github.com/mongodb/mongo/
https://github.com/BillRun/system
https://github.com/zendframework/
https://github.com/twbs/bootstrap
https://github.com/laruence/php-yaf

Pay Attention! What to keep in mind
Transactional (tx) processes
● write concern - acknowledged (default in 2.4)
● findAndModify (A.K.A FAM) - document tx
● value=oldValue
● 2 phase commit - app side
● Transaction lock by design

High performance tricks
● With SSD you get x20 more performance
● MongoDB loves RAM
● Follow MongoDB production notes
○ Readahead low as possible
○ THP - transparent hugepages

More tips that happened to all
● Shorten property names
● MongoDB have database lock
○ Separate database for heavy-write collections

TCO - MongoDB based solution
● 3 Months dev
● 3 Months QA
● Few days of maintenance infra and app
● Easy to add features
● Easy to scale

What can BillRun do
● 5,000 events per second
○ Including DB insert and transactional rating
● That means 157,680,000,000 events per year
● Create hundreds of millions invoices

Thank you, Ofer Cohen
ofer@billrun.net
@oc666
Billing on Top of

MongoDB World 2014 - BillRun, Billing on top of MongoDB

More Related Content

What's hot (20)

Viewers also liked (9)

Similar to MongoDB World 2014 - BillRun, Billing on top of MongoDB (20)

More from Ofer Cohen (12)

Recently uploaded (20)

MongoDB World 2014 - BillRun, Billing on top of MongoDB