Three Unique Approaches for Dynamic Database Design Challenges- Impetus White Paper
7 likes4,721 views
This white paper discusses dynamic approaches for physical database design to address the challenges of changing workloads and data structures. It outlines three distinct methods: reusable columns, XML schema, and pivot tables, each with their limitations and benefits in accommodating evolving data needs without major structural changes. The paper emphasizes the importance of proactive design to facilitate future migration and modifications, with a recommendation for the pivot approach as the most effective dynamic solution.
Three Unique Approaches for Dynamic Database Design Challenges- Impetus White Paper
- 1. W H I T E P A P E R Three Unique Approaches for Dynamic Database Design Challenges Abstract Physical database design is the most vital aspect of database administration. While physical design caters to specific and static workloads for a certain period of time, both the design and workloads can change. Thus, there may be different approaches to focusing on dynamic physical design, which can account for time‐varying workloads. This white paper takes into consideration dynamic approaches as well as some constrained approaches. The goal is to recommend dynamic physical designs that reflect major workload trends. The white paper also presents its definition of a dynamic physical design problem and discusses several techniques for solving it. Impetus Technologies, Inc. www.impetus.com July ‐ 2011
- 2. Three Unique Ap pproaches for D Dynamic Datab base Design Ch hallenges Table o of Conte ents Introduction ..... ...................... ............................................................... ....... 2 ta design challenges ......... Dat ............................................................... ....... 3 Overcoming dat ta design challenges ................................................... ....... 4 Bus siness scenar .................. rio ............................................................... ....... 4 Solution approa aches ............. ............................................................... ....... 5 1.Reusa able Columns .............................................................. ....... 5 Limitations of Reusable Columns .......................................... ....... 6 Schema ......... 2.XML S ............................................................... ....... 7 Limitations of XML Schema ................................................... ....... 9 3.PIVOT T Table .......... . ............................................................... ..... 10 Limitations of the PIV VOT Table .............................................. ..... 12 nclusion ....... Con . ...................... ............................................................... ..... 13 In ntroduct tion Any informmation that is conceptual and needs to b be stored, is a a conceptual design for a a database. Itt describes the concepts too be saved. Converting the e conceptual design into a a physical des sign, or a plan n for actually implementin ng the database inn code, is calle ed the physic cal design of t the database. . A lot of work has been done for solving th he problem of automating physical data abase design. . Today, enteerprise data ddesign has beecome much m more comple ex than modeling traditional data stores. EEnterprise data design neeeds to cope wwith different types of data, changing data and evo olving databasse schema ov ver a period oof time. Various dattabase manag gement systeems offer a feature called aa database de esign advisor. For these design advisors, thhe physical daatabase desiggn is more of a a lem. For a giv static probl ven set of que eries which deescribe the wworkload with some const traints, the deesign advisor recommends a set of phy ysical databasse structures w with optimize ed indexes annd materializeed views, which minimize t the cost of execution. 2
- 3. Three Unique Ap pproaches for D Dynamic Datab base Design Ch hallenges Data a design challenges Several chaallenges can ccome up when work is on w with static da ata design, ssues related to data quality, data desig including is gn, data archiitecture, and processes. The limitatioons of the dessign advisors available with h various databases i is that these a advisors do nnot keep into account the c changes of workload o over time i.e. it may for ins stance, be req quired to chan nge the database’s physical desi ign for accum mulating the lo oad. A user wworking with a a dataset for just a few yeears may be c capable of ma aintaining cleaan data. How wever, as soon as multiple user rs are involved, errors and inconsistenc cies begin to c creep into a poorrly designed static databas se. If the inten ntion is to dessign a static database thhat manages itself according to the load and time, v various challenges and limitations may com me up with the static datab base design: • e most seriou The us enemy of c clean data in l long‐lived sta atic database sys stems is redun ndant copies of informatioon. • Forr accumulatin ng the load, if f there is a ne eed to change e the physical stru ucture of the database, th hen all the dep pendencies m must also be chaanged accordingly. • one manages to accumulat If o te the load th hrough complex design, witthout changinng the physicaal structure, t then it requires substantivve wo ork to managee both the back‐end and fr ront‐end. Signnificant redes sign and d coding are p possibly required. • If the actual prooblem of poor r database de esign is not ad ddressed, it w will con ntinue to affe ect future pro ojects. • Per rformance is likely to be si ignificantly im mpacted if the e existing obje ects are e to be mappeed with the changed datab base structure, resulting in n the erhead of mapping objects ove s to the databbase and the transformations req quired to supp port those mappings. • The e database do ocumentation n will have to o change again n and again cor rresponding t to physical str ructure and ddependency cchanges. • If refactoring is done or schema change is s required in t the existing dat tabase to reflect the new d data schema, , then the corrresponding app plication(s) w will also requir re changes. There will be a a need to iden ntify and d then fix all d data‐related problems, reqquiring signifi icant effort to o achhieve. • If the back‐end keeps changing in accorda ance with thee requirementt and loaad, then the in ntegration pooint for the ap pplication andd database would bec come a signifficant problem m. In some ca ases applicatio ons may be 3
- 4. Three Unique Ap pproaches for D Dynamic Datab base Design Ch hallenges rew written to usee the new acc cess approach h and ensure integrity with hin the e database. Overcom O ming data a design challenges If completee control as w well as a clean n database is r required, a dy ynamic physical database design is recom mmended tha at will also ke eep into accouunt the trend ds in increasing i input workloa ad. By implemmenting a dyn namic design, , one can overcome t the challengees associated with a static physical data abase design. The resulting da atabase can hhave the folloowing feature es: 1. Thee database is scalable withhout changing g the physical structure, an nd is flex xible enough to expand as s input worklo oad changes wwith time. 2. Thee database is easy to main ntain, as theree is no change e in the physiical stru ucture for acccommodating g the load. 3. Thee database ha as minimal reeconstruction of data. 4. Theere is an overrall reduction in developm ment time andd cost, through the acccommodation n of the changging requiremments and the e large numbe er of bussiness rules. In this whit te paper, we w will talk abou ut the solution n of a dynami ic physical database design. Busine ess scena ario Most of thee systems havve a common n business req quirement—to provide a storage mo odel whose scchema may be extended o or altered by its users after r the system is in n production——that is sche ema that will e enable users to define the eir own databa ase attributess, and collect data submittted on those attributes without changing thhe physical st tructure of the database. TThe objectivee, therefore, is to recommend the design, storage and data access s strategies for such a scenario. Let us assume SQL SERV VER is the prim mary databas se and the prooblem is of extending aan Employee Table for han ndling more aattributes without changin ng the physical str ructure. 4
- 5. Three Unique Ap pproaches for D Dynamic Datab base Design Ch hallenges So olution a approaches For addresssing the abov ve business sc cenario of dattabase designn, we can havve different strategies f for achieving the dynamici ity in the data abase. These strategies aree classified according tto the level of f dynamic behhavior. The foollowing are t the different methods through wh hich we can achieve dynam mic behavior in the physicaal database sttructure. 1. Reusable Columns s This is an obvious appro oach where thhe Employee table is created with a pre eset number of reusable colu umns, and a mmapping tablee is created fo or signifying t the type of attribute th hat is stored in the reusable columns. T Though, this iss not a 100 peercent dynamic ap pproach, it is simple enouggh to provide a certain leve el of dynamic c behavior. There can b be two tabless such as: 1. Emp ployeeePar rtial 2. Emp ployeePart tialColumn Insert dat ta into the following tables s as shown. The follow wing Query w will fetch the d desired colum mns in such a fashion that i it will seem to o be coming f from a single Employee table. . 5
- 6. Three Unique Ap pproaches for D Dynamic Datab base Design Ch hallenges Limitatio ons of Reusa able Column ns This appro oach is not fu ully dynamic, and dynamicity is constrai ined up to the e number of reusable colu umns. 6
- 7. Three Unique Ap pproaches for D Dynamic Datab base Design Ch hallenges 2. XML Schema This is a fully dynamic appproach, in w which the emp ployee table i is equipped wwith an extra coolumn as XML L data type, w which stores th he additional attributes an nd their valuess in the form of XML. Let’s have tthe table as Insert the e values in the e table as follo ows. The corr responding XML can be se eparately crea ated as shown n and stored in t the table. 7
- 8. Three Unique Ap pproaches for D Dynamic Datab base Design Challenges h The follow wing Query will fetch the e w entire Attribute, alongside their attribute names. 8
- 9. Three Unique Ap pproaches for D Dynamic Datab base Design Ch hallenges The follow wing are the e examples of A Adding / Upda ating / Deleting the Attributes from the e XML. Limitatio ons of XML S Schema 1. Thhe addition, u updating and deletion in X XML are very c complex. The e final query a also becomes s very co omplex due to XML manip pulations. XML columns cannot be indexed, which 2. X h hampers thee performanc ce of the querry. 9
- 10. Three Unique Ap pproaches for D Dynamic Datab base Design Ch hallenges 3. PIVOT Table This is also a fully dynam mic approach, , where colum mn values are e stored as rows in a value table and can be P PIVOT for the final result. Let us creatte the followi ing tables for r storing attrib bute types an nd the attribu utes values. ta into the following tables Insert dat s as shown. 10 0
- 11. Three Unique Ap pproaches for D Dynamic Datab base Design Ch hallenges We can cr reate a View after joining a all the tables so that the v view can be PIVOT to get th he desired result. The follow wing dynamic c query will give the desire ed result. 11 1
- 12. Three Unique Ap pproaches for D Dynamic Datab base Design Ch hallenges Limitatio ons of the PIVOT Table The only l limitation of t this approach h is its comple exity. Otherw wise, this is the e only preferr red approach h for achieving dynamic beh havior in data abase design. 12 2
- 13. Three Unique Ap pproaches for D Dynamic Datab base Design Ch hallenges Conclus sion The databa ase life cycle is a reminder of the fact thhat data in a d database needs to be changed d to a new or modified stru ucture in the future. Plannning ahead in database design can hel lp ease these future migra ations or moddifications. In the case of a fuully dynamic m model, where e new attribuutes need to bbe continuallyy defined and d altered to rrepresent an e evolving dataa scenario, thee query and tthe structure becomes more e complex. Al lthough, for a achieving such h dynamic flexibility a certain level of complexit ty is acceptedd. Among the e fully dynamic and most re ecommended d approaches s is the PIVOT approach, w where the de esign is complletely normalized and inde exing can be ddone on the underlying table for performa ance improvement. This ap pproach provides the followin ng advantage es: 1. Collumns can be e rearranged aand added/de eleted dynammically, withou ut the e need for a d dump/reload of the databa ase. Any new column data may be set to initial v value (virtually) in zero do owntime. 2. ews can be cre Vie eated out of tthe dynamic queries and b be used as virrtual tab bles in joins. About Impet tus Impetus Tech hnologies offers Product Eng gineering and TTechnology R&&D services for software prodduct development. With ongoing g investments in research an nd application o of emerging teechnology areaas, innovative b business mode els, and an agile apprroach, we partner with our client base com mprising large s scale ISVs and t technology inn novators to deliver cutting‐edgee software prodducts. Our expertise spans th he domains of Big Data, SaaS, Cloud Compu uting, Mobility Solutions, Te est Engineering g, Performancee Engineering, and Social Media among oth hers. Impetus Technologies, Inc. 5300 Stevens Creek Boulev vard, Suite 450 0, San Jose, CA 95129, USA Tel: 408.252.7111 | Email: inquiry@@impetus.com Regional Devvelopment Centers ‐ INDIA: • New Delhi • Bangalore • In ndore • Hydera abad To know mo ore visit: http:/ //www.impetus.com Di isclaimers The information conntained in this document is the prop prietary and exclus sive property of Im mpetus Technologi ies Inc. except as o otherwise indicate ed. No part of is document, in wh thi hole or in part, ma ay be reproduced, , stored, transmitted, or used for de esign purposes without the prior wri itten permission o of Impetus Technologies Inc. 13 3