![Exchanging large files – how to avoid OutOfMemoryerrors?Solution: employ RMIIO library (RemoteInputStream[Server] and RemoteOutputStream[Server] classes)Figure illustrates downloading a file to client](https://image.slidesharecdn.com/obrona-100114030428-phpapp02/85/Dissertation-defense-28-320.jpg)
![References[1] M. Pomocka, P. Nowakowski, and M. Bubak, Integrating EGEE Storage Services with the Virtual Laboratory. Poster presented as partof theCracowGridWorkshop ’09, Krakow, Poland, 12-14 October 2009.[2] M. Pomocka, P. Nowakowski, and M. Bubak, Integrating EGEE Storage Services with the Virtual Laboratory. In Marian Bubak, Michał Turała, and Kazimierz Wiatr, editors, Proceedings of Cracow Grid Workshop – CGW’09, October 2009, Krakow, Poland. ACC-Cyfronet AGH.to appear[3] Lana Abadie et al., Grid-Enabled Standards-based Data Management. In Mass Storage Systems and Technologies, 2007. MSST 2007. 24th IEEE Conference on, pages 60–71, Sept. 2007.[4] Marian Bubak et al., Virtual Laboratory for Collaborative Applications, In: M. Cannataro (Ed.) Handbook of Research on Computational Grid Technologies for Life Sciences, Biomedicine and Healthcare, Information Science Reference, 2009, IGI Global[5] Matthias Assel et al. : A Collaborative Environment Allowing Clinical Investigations on Integrated Biomedical Databases. In Tony Solomonides et al. (Ed.): Healthgrid Research, Innovation and Business Case; Proceedings of HealthGrid 2009, Studies in Health Technology and Informatics, vol 147, IOS Press, ISSN 0926-9630, pp 51 -61[6] M. Malawski, T. Bartynski, and M. Bubak, "Invocation of operations from script-based grid applications," Future Generation Computer Systems, vol. In Press, Accepted Manuscript, 2009.36](https://image.slidesharecdn.com/obrona-100114030428-phpapp02/85/Dissertation-defense-36-320.jpg)
Dissertation defense
- 1. Data source registrationin the VirtualLaboratoryMarek Pomockamajor: applied computer sciencespecialisation: computer techniques in science and technologyFaculty of Physics and Applied Computer Science,AGH University of Science and TechnologySupervisor: Marian Bubak, Ph.D.Consultants:PiotrNowakowski, M.Sc. Daniel Harężlak, M.Sc.Master’s thesis defenseNovember 13, 2009
- 2. OutlineIntroduction to Grid technologies and Virtual LaboratoriesMotivation and ObjectivesConceptual view onto the solutionChallenges and solutionsApplicationsFuture workSummaryReferences
- 3. Grid technologies and Virtual Laboratories3
- 4. Gridis a distributedcomputingarchitecturewithcross-organizationalaccess, providingnontrivialquality of service for participatingactors.
- 5. Notable applications includehigh-energy physics (LHC)Weather forecastingNatural disaster modellingComplex parameter studies in biomedicine and biochemistryDigital image archives
- 6. Gridis a computer infrastructure.. dedicated to conducting in-silico researchcreated by many partnerswho share supercomputers, computer clusters, storage andresearch instrumentsTASKPCSSICMWCSSCYFRONET
- 7. to create common space for e-Science
- 8. which are dynamic by their natureGrid users are Virtual Organizations(VOs)VO approach simplifies access managementCYFRONETPSNCCYFRONETPSNC
- 9. Examples of GridsEGEE, DEISATeraGridOpen Science Grid
- 10. Virtual Laboratories (VLs)supplyhigher-level services and abstractlow-leveldetailsrelated to Grid services invocations, security etc. awayfromend-users.VirtualLaboratoryGridmiddlewareMany VLsendeavor to be general purposein-silico(orvirtual)experiment design and execution environment, Gridinfrastructuree.g. GridSpace VirtualLaboratory.
- 11. Others are often designed for specific purposesuch as remote access to scientific instruments (e.g. VLAB)supporting research in meteorology (LEAD)research and decision support in virology(ViroLab)
- 12. Virtual experiments in VLs are expressed using script-based languages (e.g. in GridSpace, Athena, Geodise)if (condition) then …else …end… or using workflow languages (e.g. in VL-e, VLAB, myExperiment, myGridTaverna, Kepler, Triana, Pegasus)Virtual LaboratoryVLs made Grids available to non-computer scientists. GridUsers
- 14. Hello, I’m a chemist. I use Gaussian program and work mostly with files. I’d like to use Grids, but filesystem is far too complex for me.... the security system is complicated too.Yes, I do agree. We won’t use Grids until there is an easy way of using Grid file catalogues from virtual experiments.
- 15. ObjectivesThe objective of the dissertation is to meet these needs by enabling access to LFC data sources from GridSpace scripts concealing most of interactions with Grid Security Infrastructure (GSI).This goal entails several other objectives:Data SourceRegistryreorganizationIntegration with GridSpace Engineextending DSR EPE plug-inDAC2GSEngineLFC DS
- 16. Conceptual view onto the solution
- 18. Not to comprise GSEngine portabilityWindowsLinuxScientific Linux 4 (SL4)UNIXMac OS XIsolation of platform dependent code into a remote serviceSolution:GScript LFC integrationGSEngineLFC connectorLFC client libraryLFC DS ServerPlatform independentPlatform dependent
- 19. Serve multiple users utilizing inherently single user gLite libraries.Solution:ChemPo command wrappers – each command is run in new JVM with prepared UNIX environment.Worker 1 JVMLFC DS ServerCert1Key1(ServerJVM)Worker 2 JVMCert2Key2Instead of permanent place for a credentials (e.g. ~/.globus/), use temporary files and specify paths dynamically in UNIX environment of created JVM processes.
- 20. Enabling access to Grid files without downloading them to GSEngine machineFirst, download file to LFC DS Server. Then, stream it to client. Grid File Access Library (GFAL)ChemPo command wrappers do not support such a mode of operation (streaming to client)Vice—versa for sending file to Grid, i.e. stream file to LFC DS Server, then send it to Grid.
- 21. Streaming representation in GridSpace scriptsSolution: User receives modified version of Ruby IO object (sending file to Grid happens on file close operation while retrieving a file from Grid during object initialization) Reading a Grid fileds.open("mpomocka/test_file", "r") do |file|file.each {|line| puts line}endf = ds.open("mpomocka/test_file", :r)f.each {|line| puts line}f.close Writing to a Grid filef = ds.open("mpomocka/test_file",:write)f.puts "First line of the file test_file"f.puts "Second line of the file test_file"f.close Alternativelyds.open("mpomocka/test_file",:w) do |f|f.puts "Another way to write to a file"f.puts "Note that close is not necessary“end
- 22. Need for a descriptive and intuitive APImimicking Ruby file operations, e.g. exist?, file?e.g. create_directory instead of mkdirDAC2 LFC DS methodsMethod name, AliasescreateDirectory(parent,child),create_directorycreateDirectory(path),create_directorydelete(path),delete_file, deleteFiledeleteFile(filename)directory?(filename),isDirectory, is_directoryexist?(path), exist, exists, exist?file?(path),isFile, is_filegetFile(filename),get_filegetSize(path),size, size?, get_sizelistFiles(path),list_filesopenFile(path, mode, &b),open, open_filestoreFile(payload, filename),store_filezero?(path)
- 23. SecurecommunicationSecurityTunnelling is simplerTransport Layer SecurityNeed to manage keystoresCredentials managementProxy certificate generation Java CoG KitData Source RegistryCredentials are stored in DSRCredentials can be set static, i.e. shared with other authenticated users
- 24. Proxy generated automatically during initialization
- 25. Information needs – previous DSR structure did not enable storage of LFC data sources information nor gLite credentials.Solution:RelationalDataSourcesDataSourcesDataSources++LFCDataSourcesLFCCertDataLFCDSConnectionsAlso changes to DAC2 and DSR EPE Plug-in DSR access modules.
- 26. GUI for registering data source of new typeCreated as a new form in EPE DSR Plug-inIn addition, some new DSR access methods were created in DSR EPE Plug-in.
- 27. Selection of distributed computing approach
- 28. Exchanging large files – how to avoid OutOfMemoryerrors?Solution: employ RMIIO library (RemoteInputStream[Server] and RemoteOutputStream[Server] classes)Figure illustrates downloading a file to client
- 29. Figure – sending a file from client to serverAdditional benefits of using RMIIO: Compressed socket-based communicationAutomaticretry
- 30. Solution scales linearlyFigure – download and upload times up to 2Gb when tested locally on ChemPo server
- 31. PL-Grid:Polish Infrastructure for Information Science Support in the European Research Space.Chemistry Portal – ChemPoApplications
- 32. Finer-grained securityPseudo memory mapped-file API (Pseudo MMAP)Future work
- 33. Summary33
- 34. LFC DS ServerLFC DS client Java libraryNew DAC2 APIDAC2 LFC connectorDAC2 LFC DS methodsMethod name, AliasescreateDirectory(parent,child),create_directorycreateDirectory(path),create_directorydelete(path),delete_file, deleteFiledeleteFile(filename)directory?(filename),isDirectory, is_directory….
- 35. Automated and transparent handling of Grid credentialsExtended EPE DSR Plug-inReorganized DSR Schema
- 36. References[1] M. Pomocka, P. Nowakowski, and M. Bubak, Integrating EGEE Storage Services with the Virtual Laboratory. Poster presented as partof theCracowGridWorkshop ’09, Krakow, Poland, 12-14 October 2009.[2] M. Pomocka, P. Nowakowski, and M. Bubak, Integrating EGEE Storage Services with the Virtual Laboratory. In Marian Bubak, Michał Turała, and Kazimierz Wiatr, editors, Proceedings of Cracow Grid Workshop – CGW’09, October 2009, Krakow, Poland. ACC-Cyfronet AGH.to appear[3] Lana Abadie et al., Grid-Enabled Standards-based Data Management. In Mass Storage Systems and Technologies, 2007. MSST 2007. 24th IEEE Conference on, pages 60–71, Sept. 2007.[4] Marian Bubak et al., Virtual Laboratory for Collaborative Applications, In: M. Cannataro (Ed.) Handbook of Research on Computational Grid Technologies for Life Sciences, Biomedicine and Healthcare, Information Science Reference, 2009, IGI Global[5] Matthias Assel et al. : A Collaborative Environment Allowing Clinical Investigations on Integrated Biomedical Databases. In Tony Solomonides et al. (Ed.): Healthgrid Research, Innovation and Business Case; Proceedings of HealthGrid 2009, Studies in Health Technology and Informatics, vol 147, IOS Press, ISSN 0926-9630, pp 51 -61[6] M. Malawski, T. Bartynski, and M. Bubak, "Invocation of operations from script-based grid applications," Future Generation Computer Systems, vol. In Press, Accepted Manuscript, 2009.36