One App, Many Clients: Converting an APEX Application to Multi-Tenant

Editor's Notes

• #3: How to illustrate the concept of Multi-Tenant? First child…
• #4: Then comes two children…
• #5: Then three.
• #6: An elegant solution.
• #7: MULTI-TENANT APEX APPLICATIONS: HOW I STARTED State Youth Games is a multi-sport competition run by Youth Vision.
• #8: Churches throughout Western Australia send teams of young people aged 16-28 to compete in a wide variety of sports and games.
• #11: I've helped as a volunteer and coordinator, and in 2013 the director asked me to build a registration system for all the teams.
• #12: The first version went live that year and worked quite well; later that year the Masters Games, a similar competition run for people over 28, wanted to use the same system so I made a copy of the APEX application and they used that. Over the following years the same system was used by SportsFest and GI Games; it is now being used to manage half a dozen different competitions every year. This is an example of a Multi-Tenant application: one application used by more than one organisation, each with a private view of their own data without seeing or affecting data for other organisations. In my case, all the sites were hosted on a single Oracle database running APEX; the system was designed to support multiple competitions because the first organisation (SYG) ran a new competition every year.
• #13: This was Version 1.0. The data model includes a column in most of the tables called "Org/Year Code". This code identifies the data as relating to a particular organisation and competition year, e.g. SYG2013, MASTERS2013, SYG2014, MASTERS2014, SF2014, etc. I made a copy of the APEX application for each organisation and gave each copy an alias like "SYG", "MASTERS", etc. A substitution variable, set in the Application Definition, would tell the application what Year to use. In this way, I was able to make as many copies of the application as required, each pointing to the data for a particular organisation, for a particular competition year.
• #14: Since there was a separate application for each organisation, small customisations to the various pages were possible as required by each organisation. One benefit of this approach was being able to respond quickly to a variety of requests and bug reports for each competition - usually someone would call or email me with an issue and I'd have the fix ready within an hour or less. There was a significant problem here as well - every change or fix required redoing the changes again and again in the other copies. If I simply copied an application over the top of another, the customisations done for that other copy would be lost. It soon became impractical and I started planning the next iteration that would solve this particular problem.
• #15: In late 2014 I started Version 1.1. I took a copy of the application that was relatively "good", in that it had most of the good enhancements and bugfixes, and designated it "DEV". I reimplemented the remainder of customisations and bugfixes that had been done in the other copies into this new master copy - applying APEX conditions to ensure that customisations were only activated for the organisations that needed them.
• #16: The application had global Application Items like these: ORG - the org code such as SYG, MASTERS, SF, etc. YEAR - the competition year VERSION - the minor version number of the application
• #17: The application items were initialised for each session by copying corresponding application Substitution Variables.
• #19: These substitution variables are used to allow me to programmatically change the tenant for any copy of the application.
• #20: On the database side, database packages maintained an Application Context that stored these same variables for each user session. In this way APEX components could switch on or off - sometimes a component was only used by one particular ORG, or it was only applicable to a particular VERSION. In the same way, my PL/SQL code checked the Context variable to determine whether to run certain sections of code.
• #22: To deploy enhancements or bugfixes for an organisation, I would first make the change in my DEV application, and then I'd run a SQL*Plus script which took the following steps: 1. Call WWV_FLOW_UTILITIES.export_application_to_clob which returns a CLOB containing a SQL*Plus script that installs the application (Note: this API is not officially supported by Oracle and may change in future releases) - for more info refer to: https://jeffkemponoracle.com/2013/05/deploying-application-express-on-the-command-line/
• #23: 2. Prepend some extra initialisation code to the clob, which changes the Application ID, Alias and Name
• #25: 3. Do a search-and-replace on the resulting clob to change the substitution variables for ORG and YEAR 4. Save the clob to a file on disk. If it reported no errors I would then (a) check the script into source control, and (b) run the script to install the target application. I later changed the script so that it created a copy of the APEX application export for each application found in the workspace - that way it automatically create scripts for each application making it easy to update some or all of them when needed. I only had to make minor changes to the code when I upgraded to APEX 5.0. I was quite happy with this method so I used it for another registration system, one for youth camps and retreats, as well.
• #27: One downside to this approach is that Saved Reports in Interactive Reports are local to each application instance; I had some users who were involved in more than one competition who found it inconvenient that their saved reports did not carry over to the other competitions. On the other hand, this was sometimes convenient because they often had different reporting requirements for the different competitions. The main problem I started having with this approach was at the database schema level - the same PL/SQL code had to work for all different versions of the application that were in play at the time; so there were code fragments like "if minor_version >= 35 then ... end if" peppered throughout. I had to be careful with schema changes to ensure they didn't break things in any of the active versions. Another problem with this approach is that it didn't scale very well; every new organisation needed another copy of the application to be installed; deploying a small change could take several minutes as it installed and re-installed the application many times. After a while of this I realised having lots of separate copies of the application, largely identical, was not providing enough practical benefit. Instead, it would be easier if there was just one application which I could make changes to - and modifying the schema would be a simpler task as a result. In 2016 I decided to rebuild a significant portion of the application as a completely new application.
• #28: This was Version 2.0. This application was to use the Universal Theme, and would support all organisations and years in a single application. Instead of setting the org and year codes in Substitution Variables, I made a page that accepted a code via the URL.
• #29: I used Apache rewrites on the web server so that my clients didn't need to know what the valid codes were. They were each given a simple URL to publish on their websites and Facebook pages. So, for example, examplecomp.com.au gets directed to “examplecomp.com.au/apex/f?p=GF:FWD:::::P100_ORG_YEAR:SYG2017".
• #30: In a Page Load process, the page checks the value of the items, sets up the user session accordingly, and redirects to the home page which would use the session Context (as before) to show the data and customisations applicable to that organisation and year. If an invalid code is supplied, the page does not redirect; the page simply shows a list of links to valid codes as a kind of "graceful fallback".
• #31: The same method was implemented later for the camp registration system as well. This simple method has made building and deploying these applications much simpler. In addition, scaling to many clients is now possible; I can have a new client ready to go in under an hour.
• #32: MULTI-TENANT APPLICATIONS: THE THEORY The idea of making a system "multi-tenant" has been around for a long time; it forms the basis of "Software As A Service (SaaS)". Provide the same set of services to many tenants who do not share or see each other's data (e.g. in the cloud). A good example would be Gmail and wordpress.com; each user only sees their own data. Facebook and Twitter, conversely, are examples of systems which are most definitely NOT multi-tenant as they exist expressly for the purpose of sharing data between users and groups of users. Multi-tenant is equally applicable for services provided to different clients, or for services provided within a single company to different departments; the same principles might apply.
• #33: Purpose The purpose is to partition the data, and (potentially) the workload, so that you can build one system and then re-use (sell) it many times. The hope is that this will reduce costs, increase revenue, and/or more efficiently use available processing power and storage capacity.
• #34: Design goals for multi-tenant systems comprise four aspects. 1. Tenant Isolation 2. Optimise cloud resource cost (storage and CPU) 3. Ease of administration - self-service - allow tenants to perform maintenance, backup and recovery, troubleshooting, and customisation 4. Scalability - easily add more tenants, and add more capacity, without modifying the application itself
• #35: The goals of tenant isolation are: ensure no-one can see, or be impacted by, any other tenant's data or behaviour; make it possible to backup and restore a single tenant's data; implement tenant-specific customisations; protect from "noisy neighbours"; allow tenants to own their own data;
• #36: This goal was nicely expressed by Hadlow’s “First Law of Multi-Tenancy” – and this is partly from the point of view of the tenant, but also from the point of view of the application developer. If the fact of multi-tenancy can be abstracted away to some degree, it makes developing the application much simpler and easier.
• #37: Broadly, there are a few different architectures for designing a Multi-Tenant system. They differ on what level or technology they rely to provide tenant isolation: 1. Multiple servers: one server + database per tenant   - pros: excellent isolation for data, workload (dedicated resources) and customisations; if server fails, impact is limited to the customer on that server   - cons: high cost to add more tenants, administer, service, upgrade, and deploy changes; low opportunity to optimise cost of cloud resources - no resource sharing 2. Multiple databases: one (pluggable) database per tenant (Oracle) / one schema per tenant (SQL Server)   - pros: excellent isolation for data and customisations; better use of cloud resources   - cons: complexity of adding tenants, upgrades, changes 3. Multiple workspaces/schemas: one workspace+schema per tenant (Oracle APEX)   - pros: excellent isolation of data and customisations; optimal use of cloud resources   - cons: complexity of adding tenants, upgrades, changes
• #38: 4. Single schema: data disambiguated via table name prefix, e.g. tenant01_emp, tenant02_emp, etc.   - pros: query plans are based on stats for each tenant   - cons: too numerous to list! 5. Single schema: data disambiguated via tenant ID column   - pros: optimal use of cloud resources; simple to add tenants, upgrades, changes; new bugfixes/features are immediately available to everyone   - cons: tenant isolation must be carefully engineered into data model and application, higher risk of security issues; regression/new bugs immediately impact everyone
• #39: Should you maintain a single codebase for the application, or maintain a separate codebase for each tenant? Basically, if you're maintaining a separate codebase, or separate branches for each tenant, you don't really have a true Multi-Tenant system; instead, you've got a single-tenant system that you're copying, reusing, and customising into copies that lose a lot of the benefits of having a true multi-tenant system. Whether that loss is important depends on each case. If you have only a few clients, each with quite diverse requirements, this approach may well make a lot of sense. You then have freedom to make quite drastic customisations for each client. Of course, you then have a bigger maintenance burden but if you have the clients willing to pay for that flexibility, then why not?
• #41: Application Containers (Oracle 12.2)  common objects - share metadata for a multi-tenant application across multiple PDBs  application PDB seed - rapidly create new application PDBs from the seed  views can query all PDBs in the container  perform DML on objects in multiple PDBs  application versioning and application patches Container Maps - routing SQL to the appropriate PDB based on the value of a predicate used in the SQL statement - partitioning data at the PDB level
• #43: Workspace per tenant Build Options Application per tenant Build Options Single application/workspace Conditions or Authorizations?
• #44: To finish, I’d like to quickly show a very simple set of steps to take an existing APEX application and simply enable it to support multiple tenants. Step 1 – this is the only change necessary on the APEX side – a call to a database procedure in the Post-Authentication procedure.
• #45: Some schema changes will be required, including adding a “security group ID” column to each table (some global admin tables excepted, possibly).
• #46: To guarantee tenant isolation, I recommend VPD. Make sure to set update_check to “true” (the default is false) to ensure that users or the application cannot inadvertently change the security_group_id on any row.
• #47: Some changes to unique constraints and foreign key constraints may be required, if natural keys are used. However, in cases where surrogate keys are used, no modification should be required.
• #48: Refer to this article for all the details and sample code.