CDR-Stats : VoIP Analytics Solution for Asterisk and FreeSWITCH with MongoDB
6 likes5,440 views
The document outlines a call data analysis framework using MongoDB for Asterisk and FreeSWITCH, focusing on challenges such as handling millions of call records from multiple sources. It emphasizes the benefits of using NoSQL databases, particularly MongoDB, for fast analytics and real-time fraud detection, along with key technologies involved in the architecture. Additionally, it provides insights into data handling techniques like pre-aggregation, map-reduce, and overall roadmap for future improvements in call monitoring and support for other telecom switches.
![Pre-Aggregate - Daily Collection
Produce data easier to manipulate :
current_y_m_d = datetime.strptime(str(start_uepoch)[:10], "%Y-%m-%d")
CDR_DAILY.update({
'date_y_m_d': current_y_m_d,
'destination_number': destination_number,
'hangup_cause_id': hangup_cause_id,
'accountcode': accountcode,
'switch_id': switch.id,
},{
'$inc':
{'calls': 1,
'duration': int(cdr['variables']['duration']) }
}, upsert=True)
Output db.CDR_DAILY.find() :
{ "_id" : ..., "date_y_m_d" : ISODate("2012-04-30T00:00:00Z"), "accountcode" : "1000", "calls" : 1, "destination_number"
: "0045277522", "duration" : 23, "hangup_cause_id" :9, "switch_id" :1 }
...
- Faster to query pre-aggregate data
- Upsert is your friend / update if exists - insert if not](https://image.slidesharecdn.com/cdr-stats-voipanalyticssolutionmongodbmeetup-120524073436-phpapp02/85/CDR-Stats-VoIP-Analytics-Solution-for-Asterisk-and-FreeSWITCH-with-MongoDB-12-320.jpg)
![Map-Reduce - Reduce Step
Reduce Step is trivial, it simply sums up and counts :
reduce = mark_safe(u'''
function(key,vals) {
var ret = {
calldate__count : 0,
duration__sum: 0,
duration__avg: 0
};
for (var i=0; i < vals.length; i++){
ret.calldate__count += parseInt(vals[i].calldate__count);
ret.duration__sum += parseInt(vals[i].duration__sum);
}
return ret;
}
''')](https://image.slidesharecdn.com/cdr-stats-voipanalyticssolutionmongodbmeetup-120524073436-phpapp02/85/CDR-Stats-VoIP-Analytics-Solution-for-Asterisk-and-FreeSWITCH-with-MongoDB-14-320.jpg)
CDR-Stats : VoIP Analytics Solution for Asterisk and FreeSWITCH with MongoDB
- 1. Call Data Analysis for Asterisk & FreeSWITCH with MongoDB Arezqui Belaid @areskib <info@star2billing.com>
- 2. Problems to solve - Millions of Call records - Multiple sources - Multiple data formats - Replication - Fast Analytics - Multi-Tenant - Realtime - Fraud detection
- 3. Why MongoDB - NoSQL - Schema-Less - Capacity / Sharding - Upserts - Replication : Increase read capacity - Async writes : Millions of entries / acceptable losses - Compared to CouchDB - native drivers
- 4. What does it look like? Dashboard
- 5. Hourly / Daily / Monthly reporting
- 7. World Map
- 8. Realtime
- 9. Under the hood - FreeSWITCH (freeswitch.org) - Asterisk (asterisk.org) - Django (djangoproject.com) - Celery (celeryproject.org) - RabbitMQ (rabbitmq.com) - Socket.IO (socket.io) - MongoDB (mongo.org) - PyMongo (api.mongo.org) - and more...
- 10. Our Data - Call Detail Record (CDR) 1) Call info : 2) BSON : CDR = { 'hangup_cause_q850':'20', ... 'hangup_cause':'NORMAL_CLEARING', 'callflow':{ 'sip_received_ip':'192.168.1.21', 'caller_profile':{ 'sip_from_host':'127.0.0.1', 'tts_voice':'kal',7', 'username':'1000', 'accountcode':'1000', 'destination_number':'5578193435', 'sip_user_agent':'Blink 0.2.8 (Linux)', 'ani':'71737224', 'answerusec':'0', 'caller_id_name':'71737224', 'caller_id':'71737224', ... 'call_uuid':'adee0934-a51b-11e1-a18c- }, 00231470a30c', ... 'answer_stamp':'2012-05-23 15:45:09.856463', }, 'outbound_caller_id_name':'FreeSWITCH', 'variables':{ 'billsec':'66', 'mduration':'12960', 'progress_uepoch':'0', 'effective_caller_id_name':'Extension 1000', 'answermsec':'0', 'sip_via_rport':'60536', 'outbound_caller_id_number':'0000000000', 'uduration':'12959984', 'duration':'3', 'sip_local_sdp_str':'v=0no=FreeSWITCH 'end_stamp':'2012-05-23 15:45:12.856527', 1327491731n' 'answer_uepoch':'1327521953952257', }, 'billmsec':'12960', ... ... 3) Insert Mongo : db.cdr.insert(CDR);
- 11. Pre-Aggregate
- 12. Pre-Aggregate - Daily Collection Produce data easier to manipulate : current_y_m_d = datetime.strptime(str(start_uepoch)[:10], "%Y-%m-%d") CDR_DAILY.update({ 'date_y_m_d': current_y_m_d, 'destination_number': destination_number, 'hangup_cause_id': hangup_cause_id, 'accountcode': accountcode, 'switch_id': switch.id, },{ '$inc': {'calls': 1, 'duration': int(cdr['variables']['duration']) } }, upsert=True) Output db.CDR_DAILY.find() : { "_id" : ..., "date_y_m_d" : ISODate("2012-04-30T00:00:00Z"), "accountcode" : "1000", "calls" : 1, "destination_number" : "0045277522", "duration" : 23, "hangup_cause_id" :9, "switch_id" :1 } ... - Faster to query pre-aggregate data - Upsert is your friend / update if exists - insert if not
- 13. Map-Reduce - Emit Step - MapReduce is a batch processing of data - Applying to previous pre-aggregate collection (Faster / Less data) map = mark_safe(u''' function(){ emit( { a_Year: this.date_y_m_d.getFullYear(), b_Month: this.date_y_m_d.getMonth() + 1, c_Day: this.date_y_m_d.getDate(), f_Switch: this.switch_id }, {calldate__count: 1, duration__sum: this.duration} ) }''')
- 14. Map-Reduce - Reduce Step Reduce Step is trivial, it simply sums up and counts : reduce = mark_safe(u''' function(key,vals) { var ret = { calldate__count : 0, duration__sum: 0, duration__avg: 0 }; for (var i=0; i < vals.length; i++){ ret.calldate__count += parseInt(vals[i].calldate__count); ret.duration__sum += parseInt(vals[i].duration__sum); } return ret; } ''')
- 15. Map-Reduce Query : out = 'aggregate_cdr_daily' calls_in_day = daily_data.map_reduce(map, reduce, out, query=query_var) Output db.aggregate_cdr_daily.find() : { "_id" : { "a_Year" : 2012, "b_Month" : 5, "c_Day" : 13, "f_Switch" :1 }, "value" : { "calldate__count" : 91, "duration__sum" : 5559, "duration__avg" : 0 } } { "_id" : { "a_Year" : 2012, "b_Month" : 5, "c_Day" : 14, "f_Switch" :1 }, "value" : { "calldate__count" : 284, "duration__sum" : 13318, "duration__avg" : 0 } } ...
- 16. Roadmap - Quality monitoring - Audio recording - Add support for other telecoms switches - Improve - refactor (Beta) - Testing - Listen and Learn
- 17. WAT else...? - Website : http://www.cdr-stats.org - Code : github.com/star2billing/cdr-stats - FOSS / Licensed MPLv2 - Get started : Install script Try it, it's easy!!!
- 18. Questions ? Twitter : @areskib Email : areski@gmail.com