Case Study: SRM 2.0 - A next generation shared resource management system built on SpringSource dm Server

Editor's Notes

• #3: So the audience for this presentation is primarily going to be software developers that use the Spring framework, probably several who are interested in leveraging OSGi by way of Spring Dynamic Modules, which is the Java modularity runtime provided by SpringSource dm Server, the application server on which SRM 2.0 will reside. In that interest I'm going to provide them with a brief introduction to what St. Jude is, followed by an overview of our research landscape and the concept of shared resources. We'll then step through SRM 1.x, how it was developed, its architecture, technology stack, and data model, and then look at many of the issues with SRM 1.x. We'll then discuss the crossroads provided by the IS strategic plan that called for the massive expansion of SRM and prompted the development of SRM 2.0. We'll look at the key goals for SRM 2, it's architecture, data model, and tech stack, and the describe the many benefits that we've realized from our adoption of Spring DM and OSGi. I'll follow up this discussion with a brief demo of SRM 2.0 and add functionality to the application by way of our OSGi-based plugin system. Next we'll look at SRM 2.0's partner system, the SRM Data Bus, which provides integration between SRM 2.0 and its own Data Mart, the upcoming Pediatric Cancer Genome Project data warehouse, our Pallas data analysis workflow engine, and our research archives. Finally we'll have a brief demo of the Data Bus and take Q&A
• #4: St. Jude represents the marriage of life sciences research and clinical care, rallied around our mission, “to advance cures, and means of prevention, for pediatric catastrophic diseases through research and treatment. Consistent with the vision of our founder Danny Thomas, no child is denied treatment based on race, religion or a family’s ability to pay.
• #5: Our founder, Danny Thomas, was a struggling entertainer that one day offered up a simple prayer to St. Jude, the patron saint of hopeless causes, that if he would help Danny find his way in life, Danny would build him a shrine. Danny went on to a fruitful entertainment career, perhaps best known for his starring role in the TV sitcom, “Make Room for Daddy.” St. Jude Children’s Research Hospital is the promised shrine. We are supported primarily by funds from volunteer contributions raised by ALSAC, the fund-raising arm of St. Jude. Some basic information about St. Jude is here for you in the slides.
• #6: St. Jude is best known for advancing the treatment of childhood leukemia, though we also research and treat many other catastrophic diseases, including sickle cell, influenza, HIV/AIDS, and other childhood cancers.
• #7: And the fruits of our labor over the past 47 years show a massive increase in the national average survival rate for multiple cancers, including an ALL survival rate increase from 4% in 1962 to 94% in 2006.
• #8: Part of what has made all of these discoveries possible is a very vibrant research community, both in basic sciences and clinical sciences spaces. Just to give you a cross section of the research landscape at St. Jude, we have 16 academic departments and 5 academic divisions, ranging from Biochemistry to Tumor Cell Biology. We also have the first and only National Cancer Institute designated Comprehensive Cancer Center solely focused on pediatric cancer, consisting of the six programs listed here. Supporting all of this research is our ecosystem of...
• #9: ...shared resources, which are centralized facilities providing services and access to specialized equipment for research activities. These are very often centered around technology too expensive for individual research programs to maintain their on infrastructure. Some examples of these are our High-throughout DNA Sequencing facility, our Functional Genomics facility providing gene expression array technology, and our Electron Microscopy facility. St. Jude currently has 41 shared resources, of which a little more than half are laboratory-based.
• #10: These shared resources all have a common set of operational issues, a few of which include ordering or requisition for services. These can be a simple as a basic form process for service requests to sophisticated scheduling systems for shared instrumentation. Nearly all of these facilities require some form of workflow or process management, where by they track the flow of samples or other items through their facilities, collecting and generating data, (TODO: work on the flow here)...all of this is describe by the umbrella term of “LIMS” or Lab Information Management System. Finally, most of these facilities operate on a chargeback basis, where by they recover costs by charging investigators grants and other centers of funding for services.
• #11: These shared resources all have a common set of operational issues, a few of which include ordering or requisition for services. These can be a simple as a basic form process for service requests to sophisticated scheduling systems for shared instrumentation. Nearly all of these facilities require some form of workflow or process management, where by they track the flow of samples or other items through their facilities, collecting and generating data, (TODO: work on the flow here)...all of this is describe by the umbrella term of “LIMS” or Lab Information Management System. Finally, most of these facilities operate on a chargeback basis, where by they recover costs by charging investigators grants and other centers of funding for services.
• #12: These shared resources all have a common set of operational issues, a few of which include ordering or requisition for services. These can be a simple as a basic form process for service requests to sophisticated scheduling systems for shared instrumentation. Nearly all of these facilities require some form of workflow or process management, where by they track the flow of samples or other items through their facilities, collecting and generating data, (TODO: work on the flow here)...all of this is describe by the umbrella term of “LIMS” or Lab Information Management System. Finally, most of these facilities operate on a chargeback basis, where by they recover costs by charging investigators grants and other centers of funding for services.
• #13: These shared resources all have a common set of operational issues, a few of which include ordering or requisition for services. These can be a simple as a basic form process for service requests to sophisticated scheduling systems for shared instrumentation. Nearly all of these facilities require some form of workflow or process management, where by they track the flow of samples or other items through their facilities, collecting and generating data, (TODO: work on the flow here)...all of this is describe by the umbrella term of “LIMS” or Lab Information Management System. Finally, most of these facilities operate on a chargeback basis, where by they recover costs by charging investigators grants and other centers of funding for services.
• #14: These shared resources all have a common set of operational issues, a few of which include ordering or requisition for services. These can be a simple as a basic form process for service requests to sophisticated scheduling systems for shared instrumentation. Nearly all of these facilities require some form of workflow or process management, where by they track the flow of samples or other items through their facilities, collecting and generating data, (TODO: work on the flow here)...all of this is describe by the umbrella term of “LIMS” or Lab Information Management System. Finally, most of these facilities operate on a chargeback basis, where by they recover costs by charging investigators grants and other centers of funding for services.
• #15: These shared resources all have a common set of operational issues, a few of which include ordering or requisition for services. These can be a simple as a basic form process for service requests to sophisticated scheduling systems for shared instrumentation. Nearly all of these facilities require some form of workflow or process management, where by they track the flow of samples or other items through their facilities, collecting and generating data, (TODO: work on the flow here)...all of this is describe by the umbrella term of “LIMS” or Lab Information Management System. Finally, most of these facilities operate on a chargeback basis, where by they recover costs by charging investigators grants and other centers of funding for services.
• #21: Monolithic Deployment - our core service components and laboratory specific components resided in the same source tree. While we always intended to maintain a degree of separation between these, the lack of a true inheritance model in EJB 2.0 made this extremely difficult. Where a straightforward application of the Strategy pattern in a POJO model would have solved many problems, the inability to do this with EJB would often result in hack solutions where laboratory-specific code would slowly creep into the common components. Another problem occurred when a useful “utility” class present in the web layer would find an application in the service layer. Rather than factoring out into a common package, the shared source tree facilliated the creation of a circular dependency. Of course neither of these issues is insurmountable with proper discipline, but the programming model did little to prevent it.
• #22: Monolithic Deployment - our core service components and laboratory specific components resided in the same source tree. While we always intended to maintain a degree of separation between these, the lack of a true inheritance model in EJB 2.0 made this extremely difficult. Where a straightforward application of the Strategy pattern in a POJO model would have solved many problems, the inability to do this with EJB would often result in hack solutions where laboratory-specific code would slowly creep into the common components. Another problem occurred when a useful “utility” class present in the web layer would find an application in the service layer. Rather than factoring out into a common package, the shared source tree facilliated the creation of a circular dependency. Of course neither of these issues is insurmountable with proper discipline, but the programming model did little to prevent it.
• #23: That said, the core services platform eventually became fairly stable. As we were constantly developing new laboratory-specific extensions, that’s here the bugs were. Unfortunately, the monolithic model also led to an isolated defect creating distributed suffering. In this case, we’d often need to create downtime for all facilities served by the application to deploy a work stopping fix during the business day. Not good. The “resistance to rapid development” created by EJB inspired the use of alternative solutions - i.e. go around the service layer with direct DB access - within the web layer to simply “get things done.” This inevitably resulted in data integrity issues, as different services were working with the data differently.
• #24: That said, the core services platform eventually became fairly stable. As we were constantly developing new laboratory-specific extensions, that’s here the bugs were. Unfortunately, the monolithic model also led to an isolated defect creating distributed suffering. In this case, we’d often need to create downtime for all facilities served by the application to deploy a work stopping fix during the business day. Not good. The “resistance to rapid development” created by EJB inspired the use of alternative solutions - i.e. go around the service layer with direct DB access - within the web layer to simply “get things done.” This inevitably resulted in data integrity issues, as different services were working with the data differently.
• #25: That said, the core services platform eventually became fairly stable. As we were constantly developing new laboratory-specific extensions, that’s here the bugs were. Unfortunately, the monolithic model also led to an isolated defect creating distributed suffering. In this case, we’d often need to create downtime for all facilities served by the application to deploy a work stopping fix during the business day. Not good. The “resistance to rapid development” created by EJB inspired the use of alternative solutions - i.e. go around the service layer with direct DB access - within the web layer to simply “get things done.” This inevitably resulted in data integrity issues, as different services were working with the data differently.
• #61: Going back to our original problem of isolated defects creating distributed suffering, OSGi provides us with a way to circumvent this problem. Now that we can implement our highly-specific, and thus more volatile components as plugins, we have the ability to truly isolate that functionality. OSGi and Spring DM provide us with the ability to expose our core API as OSGi services and then consume those services from the plugins. Further more, as the Spring programming model now ties in nicely with the OSGi lifecycle, it becomes absolutely trivial to register our plugins with the SRM core platform using Spring lifecycle callbacks such as @PostConstruct. And of course, in the event our plugins become more generally useful, nothing whatsoever prevents us from sharing these plugins amongst multiple laboratories.
• #62: Going back to our original problem of isolated defects creating distributed suffering, OSGi provides us with a way to circumvent this problem. Now that we can implement our highly-specific, and thus more volatile components as plugins, we have the ability to truly isolate that functionality. OSGi and Spring DM provide us with the ability to expose our core API as OSGi services and then consume those services from the plugins. Further more, as the Spring programming model now ties in nicely with the OSGi lifecycle, it becomes absolutely trivial to register our plugins with the SRM core platform using Spring lifecycle callbacks such as @PostConstruct. And of course, in the event our plugins become more generally useful, nothing whatsoever prevents us from sharing these plugins amongst multiple laboratories.
• #63: 1. Tour of the metadata editor and downstream user facing functionality 2. Look at 3 Lims Home Screens 3. Do a quick plugin of Lims Home 4. Show the functional lims home dashboard
• #84: Demonstrate the round trip stuff: 1) Deploy the Lims Home Client 2) Deploy the Databus Plugin 3) Demonstrate round trip functionality