How to Build a new under filesystem in Alluxio: Apache Ozone as an example
0 likes50 views
The document outlines key topics from the Data Orchestration Summit 2020, focusing on building a new under filesystem in Alluxio using Apache Ozone. It covers various features and functionalities, including service discovery and dependency management within Alluxio UFS. Additionally, it provides implementation guidance, links for further information, and community engagement resources.
How to Build a new under filesystem in Alluxio: Apache Ozone as an example
- 1. DATA ORCHESTRATION SUMMIT 2020 How to build a new Under Filesystem in Alluxio Apache Ozone as an example Baolong Mao | Sr. System Engineer Alluxio PMC Apache Ozone Committer
- 2. DATA ORCHESTRATION SUMMIT Contribution From Tencent ● Ozone & COSN UFS ● Support using IP as connect host ● Make PermissionChecker Configurable ● Prometheus + Grafana monitor dashboard template & docs ● Mount table webUI ● Generic tarball script to generate customized hadoop ● More metrics: Lock pool size and block remover metrics ● Add ETAG header to s3 api proxy ● ……………………….
- 3. DATA ORCHESTRATION SUMMIT Agenda ● Alluxio global namespace ● Apache Ozone ● Alluxio UFS framework ●
- 4. DATA ORCHESTRATION SUMMIT 2020 ● Alluxio global namespace Introduction How the unified namespace works in Alluxio with different under file systems
- 7. DATA ORCHESTRATION SUMMIT 2020 Apache Ozone Introduction Introduce Apache Ozone shortly
- 10. DATA ORCHESTRATION SUMMIT 2020 Alluxio UFS framework Introduce about Alluxio UFS
- 11. DATA ORCHESTRATION SUMMIT •./bin/alluxio fs mount --option alluxio.underfs.hdfs.configuration=<DIR>/ozone-site.xml:<DIR>/ core-site.xml /ozone o3fs://<OZONE_BUCKET>.<OZONE_VOLUME>/ How Alluxio UFS works
- 12. DATA ORCHESTRATION SUMMIT Service Discovery ● Dynamically loaded ● Java ServiceLoader . Implements the alluxio.underfs.UnderFileSystemFactory interface. ● Pointing to the class implementation by META_INF/services/alluxio.underfs.UnderFileSystemFactory How Alluxio UFS works
- 13. DATA ORCHESTRATION SUMMIT Dependency Management ● Fat jar ● Shaded jar ● Isolated classloading How Alluxio UFS works
- 15. DATA ORCHESTRATION SUMMIT Implementing an Under Storage Extension •Implementing the required under storage interface
- 16. DATA ORCHESTRATION SUMMIT Implementing an Under Storage Extension •Declaring the service implementation
- 17. DATA ORCHESTRATION SUMMIT Implementing an Under Storage Extension •Add maven plugin to pom.xml for build a package <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> </plugin> <plugin> <groupId>com.coderplus.maven.plugins</groupId> <artifactId>copy-rename-maven-plugin</artifactId> </plugin> </plugins> </build>
- 18. DATA ORCHESTRATION SUMMIT 2020 Introduce how to implement Ozone UFS
- 19. DATA ORCHESTRATION SUMMIT alluxio fs cat /ozone/B
- 21. DATA ORCHESTRATION SUMMIT Hadoop compatible File system(HCFS)
- 25. DATA ORCHESTRATION SUMMIT How to Join Alluxio Community Thank you! alluxio.io/slack