Scaling Telecom Billing Solutions for Growth

Understanding the Telecom Billing Ecosystem

Telecom companies rely on various software systems in order to provide their offerings to their customers in a seamless way. There are systems like a CRM system, an end-customer portal, a BRM system (Billing and Revenue Management System), a financial accounting and reporting system, a network provisioning system and more. Broadly, the systems are divided into the BSS (Business Support Systems) and OSS (Operational Support Systems). The systems like the CRM, the BRM, the end customer portal etc form the part of BSS domain.  Check this page to know more about the role and the importance of each of the software systems in the Telecom Billing Ecosystem.

The BRM system (or the Billing & Revenue Management System) is considered the heart of the telecom software ecosystem. This is because this is the system that helps with the revenue posting and collection. It has all the billing, revenue and financial information required by all systems. All the software systems therefore are connected to the BRM system and rely heavily on this system for their functioning to a great extent.

What does more growth bring forth for a Telco?

The telecom world involves a highly competitive landscape which is regulated and monitored quite closely. To the telcos, the business growth comes as good news and as an organisation you and the team are looking forward to the bigger opportunities this business growth would help fuel. However, it is important to ensure that the business growth is smooth and is not mired in system issues, billing problems, backlog issues (trains not running on time!) and regulatory problems. A business growth should go hand in hand with investment of your team's time, and a budget (dollars) to make sure that you are keeping your teams and your systems ready for growth.

As one would expect, growth would bring more volumes, more service issues, more customer complaints and more competition. To resolve these issues, you need to scale your various systems to handle more transactions and more data. The BRM system needs scaling to ensure that it can handle more customers, more services and more usage records (or CDRs) for rating and charging, a larger number of invoices, e-bills, print bills and payments.

What are the various aspects of Scaling a Telecom Billing System?

There are various aspects to scaling a telecom billing solution to meet your growing demands. These aspects would include a strategy aspect, a technical aspect, a testing / certification aspect and benchmarking.

Business Strategy

The business should have a clear picture of the current business scenario. The business team should know where they are going and what would be the expected rate of growth in the near to medium term future. For safety, they should project a range of growth in terms of minimum  and maximum expected revenue and customer growth, and the associated growth in volumes of transactions.

The increase in volumes of transactions could be because of telco life cycle transactions such as new sales, subscribing to newer services, plan changes, cancellations and subscriber data updates etc. The volumes would also increase because of the increasing number of Call Data Records (CDRs) for processing in billing mediation or OCS (Online Charging System). The growth in number of subscribers would mean increasing volumes for invoicing and payment collection processes as well.

The business team needs to work with the system architects in the organisation to carefully estimate the growth in numbers and then provide a plan to the engineering team to prepare the systems for the expected growth. The estimates should be realistic with a factor of safety applied slightly on the higher side.

Planning a Technical Evolution

The system architect along with an engineering team can take a complete review of all systems like the CRM system, Provisioning Systems, Billing & Revenue Management (BRM) system and Accounting system. During this review, the focus is on the areas in these systems (or their integrations) which are known or potential bottlenecks to handle the growing demand. A business manager should always be involved in such an exercise and keep the discussion honest and focused.

In performance scaling, it is observed that about 60 to 70% of focus and tasks will be associated with the BRM system and its integrations with other systems. In the BRM system, the functional modules like billing mediation, OCS (Online Charging System), the billing process and the collection process are the major batch jobs that handle more volumes with growing business.

It requires several back to back meetings potentially over a few weeks period to complete a review exercise and identify all the areas of improvements in the mentioned systems and modules. Once all the potential areas are jotted down, it requires that prioritising takes place to decide on which areas are taken up first. The categorisation can be done into 4 priorities -

• Low investment Higher Returns (a.k.a. the low hanging fruits)

• Medium investment Medium Returns

• High Investment for High Returns (and must do)

• Higher Investment for Low/Medium Returns (but must do)

Once categorized into 4 phases of execution like this, the team is essentially coming up with a technical evolution plan or a roadmap for scaling the solution. The business manager and the solutions architect can decide priorities of the project work based on the categorization. The business manager is looking at points such as 'ROI' or 'Return on Investment', meeting regulatory compliance requirements and meeting the laid out business goals within the budgetary allocation. The solution architect owns up the priority tasks with the known business constraints and budget, then takes up the implementation and solution designs of the tasks.

Implementation of the Planned Scaling Tasks

This is a phase where a set of tasks would be executed which will lead to eventually scaling the system to handle higher volumes. This implementation phase is where all the techno-functional and technology expertise of the team will be challenged and utilized.

The technical team needs to plan each scaling task at a detailed level. They need to take into account that the changes they make to the system to handle more volumes will undergo a testing and certification process as a next step. Only if the changes conform to acceptance criteria, will the test team allow the changes to be made live. Therefore the development team needs to know clearly the acceptance criteria.

This could be a batch job which is processing let us say an x number of transactions per minute, and it now needs to handle 2x or even 3x number of transactions per minute. This then could be the acceptance criteria defined by the business owing to the increasing demand on the service. The development team needs the test data or a framework or a utility that can produce test data to test at 2x or 3x the current volume of transactions. At times, it is better to spend an additional time to develop such a test framework in order to test the change as effectively as possible.

The test driven development approach helps the implementation team meet the goal precisely and with specific focus. The actual solution of the scaling issue for a particular job could be an infra-only solution or it may involve application code changes. If the application has been already designed for achieving greater throughput with vertical scaling or horizontal scaling, then it could well be an infra-only change. This means that for processing higher volumes, more hardware will be commissioned in the deployment. The addition of the new hardware and processing power would be utilized by the application to process more records. A test framework would be used to measure the incremental gains achieved.

If there are impediments to achieving higher throughput in the application, then such application shortcomings or bottlenecks need to be identified and removed. The application needs to be vertically as well as horizontally scalable. If it is not, then the implementation task for scaling will need to focus on making the application vertically as well as horizontally scalable.

Let us look at Vertical Scaling and Horizontal Scaling in more detail to understand this topic well.

Vertical Scaling and Horizontal Scaling

As shown in the diagram, in vertical scaling the hardware is increased by adding more CPU, memory and storage to the existing server or servers. In horizontal scaling, you do not add more processing power or memory to the existing servers, however you add more servers to the cluster that processes the transactions or data.

In order to support vertical scaling, the application needs to be designed to be multi-threaded and allow parallel processing within a single process. Thus, when more CPU (or processing power) is added to the existing server, the application can scale by processing the transactions or data faster by doing parallel processing by using more CPU power.

For supporting horizontal scaling, the application needs to be designed to dynamically spawn new processes to process in parallel higher volumes of records by sending those processes to the new servers or nodes made available in the cluster. Within each spawned process, there could be parallel processing taking place using multi threading. There are limitations to vertical scaling because you can add only so much hardware to an existing server. Therefore, in addition to vertical scaling, supporting horizontal scaling makes the application that much more scalable. Apart from this, in the cloud based billing deployments, it is seen that horizontal scaling is cheaper than vertical scaling.

A careful study needs to be made of the existing design and limitation of the application, the business goal with respect to scaling, and then the task should be executed with a solution design to scale vertically, or horizontally or do both. The test driven development approach ensures that the changes made to the system are tested and results documented clearly showing the delta improvements in processing.

Testing and Certification

The test and certification team can reuse the test framework or utilities used by the development team, however they also need to verify the functional accuracy and test the performance or scaling improvements on a Production like environment. They need to run the system for a day to day operation, before and after changes and do comparison of their test reports, to certify the delta improvements.

The functional testing can rely on existing test automation suits or regression cycles (whether manual or automatic). A thorough end-to-end test cycle needs to be performed and test results documented as evidence of tests executed. A final go/no-go decision is made after looking at the test results and verifying if the acceptance criteria has been satisfied by the changes made to the application.

Benchmarking

The test team can make use of different load testing tools such as the Apache jMeter, Gatling and K6 in order to evaluate and measure the performance of the system. The benchmarking exercise helps set up a baseline with respect to the throughput taking into account the hardware used for setting up such a baseline. Then scaling can be measured by vertically and horizontally scaling the application and documenting the performance gains with respect to the baseline.

For example, with 2 batch nodes, a billing process job could be processing x number of customers to generate their invoices. This could be defined as the baseline performance, which could be useful for the smallest of the deployments. When scaled to 4 batch nodes, the billing process job could be achieving 3x the output, versus if the existing 2 batch nodes are scaled vertically to add double the CPU and memory, the throughput could be 2x and not 3x. These details for various scenarios need to be tested and benchmarked for future use.

Conclusion

Scaling a telecom billing solution involves planning, strategizing, solutioning, designing and deep expertise in technology as well as testing and certification process. It is an endeavor that touches many aspects of the business, project planning / management and use of technology. This article reviewed in detail these aspects and tried to bring together a mini-guide which can help telecom companies in their scaling exercises.