One System to Process Them All

Abstract — One Ring to Rule Them All

In today’s data-driven landscape, Fintechs in B2B firms grapple with increasing challenges. Scaling bulk job processing while managing diverse client service level agreements (SLA) is particularly complex. These challenges often result in inefficiencies, delays, and operational bottlenecks. At Perfios, we address these issues with a Centralized Bulk Job Processing Unit. This system delivers seamless, multi-tenant workflows designed for financial operations. It optimizes resource utilization by dynamically provisioning infrastructure and adaptively managing concurrency. This framework unifies diverse business unit requirements into one robust, scalable, and tech-independent solution, akin to the One Ring of efficiency and control.

Problem Statement — The Shadow Over Middle-earth

Like the Fellowship, the problem we face is complex, multifaceted, and requires diverse talents and strategies. It is an epic challenge that demands careful navigation, teamwork, and the right tools to succeed.

Processing diverse documents like PDFs, scanned images, government IDs, and financial reports is central to automation in the BFSI sector. Applications span lending, fraud detection, and regulatory compliance. These workflows rely heavily on accurate data extraction and validation to support hundreds of use cases. Solutions include aggregating data from multiple sources like government APIs, customer documents, private reports, etc while employing tools for parsing, ETL, and reporting. Scaling these processes for bulk operations introduces challenges like managing diverse SLAs, high computational loads, and cost optimization. FinTechs like Perfios are at the forefront of creating solutions tailored to address these challenges.

Proposed Solution — The One Ring to Rule Them All

Much like the One Ring’s central role in unifying the various forces of Middle-earth, our proposed solution brings together all the necessary components, strategies, and technologies into one unified force to address the challenge at hand.

A centralized processing unit is designed to operate without being constrained by specific infrastructure, programming languages, or technical stacks. Essentially functioning as Infrastructure as a Service (IaaS). BUs (Business Units) can register their jobs or workflows along with predefined constraints and seamlessly execute them, with the required infrastructure provisioned dynamically on demand.

This infrastructure is shared across multiple BUs enabling different teams or clients to utilize the same resources. As a result, optimal resource utilization is fostered for critical or time-sensitive jobs, the system can dynamically adjust priority levels and quota allocations. This ensures timely completion without starving other jobs. High-priority jobs are completed faster, while lower-priority ones continue to execute efficiently.

In scenarios where no competing jobs are present, the entire infrastructure can be deleted to maintain efficiency and cost-effectiveness, the system incorporates mechanisms for graceful degradation, ensuring smooth scaling down.

Prioritization and Quota Computation — The Steward’s Dilemma

In a bulk job processing unit, jobs can vary significantly in size, with some having hundreds of requests and others handling hundreds of thousands. Managing these effectively demands a robust queuing and prioritization system to ensure fairness, efficiency, and optimal resource utilization.

All job requests are placed in a queue, and prioritized based on their urgency and infrastructure needs. While API-based and asynchronous workflow requests do not require infrastructure provisioning, processing-intensive jobs necessitate the allocation of dedicated resources.

Cyclical Processing and Quota Allocation: The system operates in cycles, processing requests from all active jobs during each cycle. An intelligent algorithm allocates a dynamically calculated quota per job. The quota computation algorithm considers the following factors:

• Client Priority and Sensitivity: High-priority and sensitive jobs are allocated more requests per cycle, ensuring quicker processing.
• Waiting Time: Jobs with longer waiting times are allocated increasing quotas to prevent delays and maintain SLAs.

To prevent resource starvation, quotas are capped to ensure that no single job dominates the system. Low-priority jobs are always guaranteed a minimum allocation, avoiding neglect.

Read More@ https://medium.com/perfiostechblog/one-system-to-process-them-all-ac591fe16d5b