gLite Data Management System
Download as PPT, PDF1 like778 views
The document provides an overview of the architecture of the gLite data management system. It discusses the challenges of data management in grids, including heterogeneity, distribution and data description. It defines key concepts like the Storage Element, File Catalogue and Storage Resource Manager interface. It also covers file naming conventions, the different types of Storage Elements, and commands to interact with the file catalogue and manage data replication.
gLite Data Management System
- 1. Architecture of the gLite Data Management System Leandro Neumann Ciuffo INFN-Catania (Italy) EELA-2 Tutorial Montevideo, 22.07.2009
- 2. Outline Challenges of data management in a Grid infrastructure Initial definitions Types of Storage Elements File naming conventions File catalogue Practical exercises (hands on) Be prepared for a bunch of acronyms! gLite DMS – EELA-2 Tutorial, 22.07.2009
- 3. Challenges Heterogeneity Data are stored on different storage systems using different access technologies Distribution Data are stored in different locations (in most cases there is no shared file system or common namespace) Data need to be moved between different locations Data description Data are stored as files (need to describe and locate them according to their content) gLite DMS – EELA-2 Tutorial, 22.07.2009 Storage Resource Manager interface File Catalogue File Transfer Service Metadata Service
- 4. Getting started The Storage Element (SE) is the service which allows users and applications (programs) to store/retrieve data (files) The DMS provide services for location, access and transfer of files User do not need to know the file location, just its logical name. Files can be replicated or transferred to several locations (SEs) as needed. Files are shared within a VO Files are write-once, read-many Files cannot be changed unless remove or replaced No intention of providing a global file management system gLite DMS – EELA-2 Tutorial, 22.07.2009
- 5. Getting started Files located in the Storage Elements (SEs)… Are mostly write-once, read-many. Accessible by users and applications from “anywhere” in the Grid. Several replicas of one file can be replicated at different sites. Cannot be changed unless remove or replaced. Storage Elements (SEs)… Provide storage space for files. Provide transfer protocol (GSIFTP) ~ GSI based FTP server Provide an interface for the management of disk and tape storage resources: Storage Resource Manager (SRM) gLite DMS – EELA-2 Tutorial, 22.07.2009
- 6. Types of Storage Elements dCache Consists of a server and one or more pool nodes. Centralized admin.: single point of access to the SE. Files are presented in the disk pools under a single virtual filesystem tree. Uses the GSI dCache Access Protocol (gsidcap). CERN Advanced STORage manager (CASTOR) Files are migrated from a disk buffer frontend to a tape mass storage Uses the insecure Remote File I/O protocol (RFIO) Disk Pool Manager (DPM) Used for fairly small SEs (max 10 TB of total space) with disk-based storage only. Uses secure RFIO protocol gLite DMS – EELA-2 Tutorial, 22.07.2009
- 7. Storage Resource Manager (SRM) B C Worker Nodes A User Interface SE - CASTOR SE - DPM dCache submit read input read input store output gLite DMS – EELA-2 Tutorial, 22.07.2009 myJOB
- 8. Storage Resource Manager (SRM) You as a user need to know all the systems!!! SRM I talk to them on your behalf I will even allocate space for your files And I will use transfer protocols to send your files there SE CASTOR SE DPM SE dCache The SRM is a single interface that takes care of local storage interaction and provides a Grid interface to the outside world. gLite DMS – EELA-2 Tutorial, 22.07.2009
- 9. File Naming conventions (1) Grid Unique IDentifier (GUID) Every file has a GUID A non-human-readable unique identifier, e.g.: guid:38ed3f60-c402-11d7-a6b0-f53ee5a37e1d Note: all replicas of a file will share the same GUID Logical File Name (LFN) An a lias that can be used to refer to a file, e.g.: lfn://grid/gilda/users/mario/myfile.dat gLite DMS – EELA-2 Tutorial, 22.07.2009 Logical File Name 1 Logical File Name N GUID ...
- 10. File Naming conventions (2) Storage URL (SURL) or Physical File Name (PFN) The location of an actual file on a storage system, e.g.: srm://aliserv6.ct.infn.it/dpm/home/gilda/project1/test.dat Note: Used by the system to find where the replica is physically stored Transport URL (TURL) Complete URI with the necessary information to access a file in a SE (including the access protocol) e.g.: rfio://lxshare0209.cern.ch//data/alice/ntuples.dat Logical File Name 1 Logical File Name N GUID ... ... Physical File SURL N Physical File SURL 1 TURL 1 TURL 1 ... gLite DMS – EELA-2 Tutorial, 22.07.2009
- 11. SRM interactions SRM The client asks the SRM for the file providing an SURL The SRM asks the Storage Element to provide the file The Storage Element notifies the availability of the file and its location The SRM returns a TURL (Transfer URL), i.e. the location from where the file can be accessed The client interacts with the storage using the protocol specified in the TURL 2 3 5 1 4 SE gLite DMS – EELA-2 Tutorial, 22.07.2009 Client
- 12. Needles in a haystack How do I keep track of all files I have on the Grid? Even if I remember all the LFN’s of my files, what about someone else's files? How does the Grid keep track of the mapping between LFN(s), GUID and SURL(s)? LFC = L CG F ile C atalogue LCG = LHC Compute Grid LHC = Large Hadron Collider gLite DMS – EELA-2 Tutorial, 22.07.2009 File Catalogue
- 13. File Catalogue Is the service which maintains mappings between LFN(s), GUID and SURL(s) It keeps track of the location of copies (replicas) of files It consists of a unique catalogue, where the LFN is the main key Looks like a “top-level” directory in the Grid For each of the supported VO a separate subdirectory exists under the "/grid" directory. All members of a given VO have read-write permissions in such a directory gLite DMS – EELA-2 Tutorial, 22.07.2009
- 14. The LFC Service User Interface SE B SE A SE C File Catalogue lfn:/grid/gilda/tcaland/mpi.txt gLite DMS – EELA-2 Tutorial, 22.07.2009
- 15. The LFC Service srm://host.example.com/foo/bar host.example.com /grid/dteam/dir1/dir2/file1.root LFN GUID 38ed3f60-c402-11d7 -a6b0… Replicas /grid/dteam/mydir/mylink Symlink Further LFNs can be added as symlinks to the main LFN. LCF key SURLs User Metadata System Metadata gLite DMS – EELA-2 Tutorial, 22.07.2009
- 16. Job submission – example 1 User Interface CE Worker Nodes WMS Small files: InputSandbox / OutputSandbox gLite DMS – EELA-2 Tutorial, 22.07.2009
- 17. Data Management – example 2 User Interface CE Worker Nodes WMS LFC SE SE gLite DMS – EELA-2 Tutorial, 22.07.2009
- 18. LFC commands Interact with the catalogue only gLite DMS – EELA-2 Tutorial, 22.07.2009 Add/replace a comment lfc-setcomment Set file/directory access control lists lfc-setacl Remove a file/directory lfc-rm Rename a file/directory lfc-rename Create a directory lfc-mkdir List file/directory entries in a directory lfc-ls Make a symbolic link to a file/directory lfc-ln Get file/directory access control lists lfc-getacl Delete the comment associated with the file/directory lfc-delcomment Change owner and group of the LFC file-directory lfc-chown Change access mode of the LFC file/directory lfc-chmod
- 19. lcg-utils commands Copy files to/from/between SEs. Keep the SEs and the Catalogue up to date. The RPM containing these tools (lcg_util) is installed in the WNs and UIs. gLite DMS – EELA-2 Tutorial, 22.07.2009 lcg-cp Copies a grid file to a local destination lcg-cr Copies a file to a SE and registers the file in the catalog lcg-del Delete one file lcg-rep Replication between SEs and registration of the replica lcg-gt Gets the TURL for a given SURL and transfer protocol lcg-sd Sets file status to “Done” for a given SURL in a SRM request
- 20. Environment Variables Make sure to use the correct BDII and LFC BDII - LCG_GFAL_INFOSYS export LCG_GFAL_INFOSYS=gilda-bdii.ct.infn.it:2170 LFC - LFC_HOST export LFC_HOST=lfc-gilda.ct.infn.it gLite DMS – EELA-2 Tutorial, 22.07.2009
- 21. Let’s practice! Reference: https://grid.ct.infn.it/twiki/bin/view/GILDA/DataManagement
- 22. Environment Variables Pointing to the right BDII Pointing to the right LFC echo $ LCG_GFAL_INFOSYS export LCG_GFAL_INFOSYS =gilda-bdii.ct.infn.it:2170 echo $ LFC_HOST export LFC_HOST =lfc-gilda.ct.infn.it gLite DMS – EELA-2 Tutorial, 22.07.2009
- 23. Before starting… voms-proxy-init --voms gilda gLite DMS – EELA-2 Tutorial, 22.07.2009 Make sure to have a proxy created
- 24. LFC: Listing file and directory lfc-ls -l /grid/gilda Remember that LFC has a directory tree structure /grid/ <VO_name> / <user directory> Defined by the user LFC Namespace You can set LFC_HOME variable to use relative paths export LFC_HOME =/grid/gilda/tutorials lfc-ls gLite DMS – EELA-2 Tutorial, 22.07.2009
- 25. LFC: creating a directory lfc-mkdir /grid/gilda/tutorials/ yourname Create your own personal directory inside: /grid/gilda/tutorials/ <your dir> You can check the creation typing: lfc-ls /grid/gilda/tutorials gLite DMS – EELA-2 Tutorial, 22.07.2009
- 26. Downloading a file lcg-cp --vo gilda lfn:/grid/gilda/users/example/alien.txt file://$HOME/alien.txt First of all, let ’s download a file from a SE to start “playing” with it. Basic Usage: Try it: lcg-cp --vo <vo name> <LFN origin> <local destination> gLite DMS – EELA-2 Tutorial, 22.07.2009
- 27. Copying and registering a file lcg-cr --vo <vo name> -l <LFN destination> -d <SE> <local file> lcg-cr Copies a file to a SE and registers the file in the catalogue This command will return the GUID for your file gLite DMS – EELA-2 Tutorial, 22.07.2009 Make sure to have a directory in the LFC ( /grid/gilda/users/sagrid/yourname/ ) Use the lcg-info or lcg-infosites commands to figure out the available SEs lcg-infosites --vo gilda se Avail Space(Kb) Used Space(Kb) Type SEs ---------------------------------------------------------- 1100000000 1145007 n.a gilda-se.rediris.es 1030000000 32 n.a fn2.hpcc.sztaki.hu 295250000 75945624 n.a aliserv6.ct.infn.it n.a 999999 n.a se-edu.grid.acad.bg 60440000 3280565 n.a iceage-se-01.ct.infn.it 1008437 8844236 n.a se.hpc.iit.bme.hu 53160000 440416 n.a vega-se.ct.infn.it 2430000000 440450 n.a se1-egee.srce.hr 97890000 440423 n.a dgt02.ui.savba.sk lcg-cr --vo gilda -l lfn:/grid/gilda/tutorials/ yourname/yourfile.txt -d aliserv6.ct.infn.it file://$HOME/alien.txt
- 28. Replicate a file between SEs lcg-rep --vo gilda -d gilda-se.rediris.es lfn:/grid/gilda/tutorials/ yourname/yourfile.txt Basic Usage: Try it: lcg-rep --vo <vo name> -d <destination SE> <LFN of your file> gLite DMS – EELA-2 Tutorial, 22.07.2009
- 29. Listing the replicas Use the same lcg-lr command used previously: The command will return the SURL of all replicas A file can be stored on multiple SE's so that a job can download it from the closest SE while is running. lcg-lr --vo gilda lfn:/grid/gilda/tutorials/ yourname/yourfile.txt gLite DMS – EELA-2 Tutorial, 22.07.2009
- 30. Adding metadata information lfc-setcomment /grid/gilda/tutorials/ yourname/yourfile.txt “ This is my comment ” This is the only user-defined metadata that can be associated with catalogue entries. Basic Usage: Try it: lfc-setcomment <LFC file path> "Your comments" gLite DMS – EELA-2 Tutorial, 22.07.2009
- 31. Listing with comments lfc-ls --comment /grid/gilda/tutorials/ yourname/ Try it: gLite DMS – EELA-2 Tutorial, 22.07.2009
- 32. Creating a symbolic link Two different LFNs will point to the same file. Basic Usage: Try it: Check your link typing: lfc-ln -s /grid/gilda/tutorials/ yourname/yourlink.txt /grid/gilda/tutorials/ yourname/yourfile.txt lfc-ln -s <your symbolic link> <original file> lfc-ls -l /grid/gilda/tutorials/ yourname/ gLite DMS – EELA-2 Tutorial, 22.07.2009
- 33. Downloading a file lcg-cp --vo gilda lfn:/grid/gilda/tutorials/ yourname/yourfile.txt file://$HOME/ yourfile.txt Basic Usage: Try it: lcg-cp --vo <vo name> <LFN origin> <local destination> gLite DMS – EELA-2 Tutorial, 22.07.2009
- 34. Deleting a file lcg-del -a --vo gilda lfn:/grid/gilda/tutorials/ yourname/yourfile.txt Basic Usage: Try it : lcg-del -a --vo <vo name> <LFN> gLite DMS – EELA-2 Tutorial, 22.07.2009
- 35. Removing a LFC directory Basic Usage: Try it : lfc-rm -r <LFC file path> lfc-rm -r /grid/gilda/tutorials/ yourname gLite DMS – EELA-2 Tutorial, 22.07.2009
- 36. Get the file SURL Basic Usage: Try it: Some advanced Data Management commands (File Transfer Service, for instance) requires the SURL of a file lcg-lr --vo gilda lfn:/grid/gilda/tutorials/ yourname/yourfile.txt lcg-lr --vo <vo name> <LFN> gLite DMS – EELA-2 Tutorial, 22.07.2009
- 37. Get the file TURL lcg-gt <paste the file SURL: srm://…> gsiftp Basic Usage: Try it: lcg-gt <file SURL> <protocol supported by the SE> gLite DMS – EELA-2 Tutorial, 22.07.2009