How to Build a new under filesystem in Alluxio: Apache Ozone as an example
0 likes172 views
The document provides an overview of the Data Orchestration Summit, focusing on the integration of Apache Ozone with Alluxio and the unified namespace concept. It discusses various technical aspects including service discovery, dependency management, and implementing under storage extensions for Alluxio. Additionally, resources and links are provided for further exploration of Alluxio and its features.
How to Build a new under filesystem in Alluxio: Apache Ozone as an example
- 1. DATA ORCHESTRATION SUMMI T How to build a new Under Filesystem in Alluxio Apache Ozone as an example Baolong Mao | Sr. System Engineer Alluxio PMC Apache Ozone Committer
- 2. DATA ORCHESTRATION SUMMIT Contribution From Tencent ● Ozone & COSN UFS ● Support using IP as connect host ● Make PermissionChecker Configurable ● Prometheus + Grafana monitor dashboard template & docs ● Mount table webUI ● Generic tarball script to generate customized hadoop ● More metrics: Lock pool size and block remover metrics ● Add ETAG header to s3 api proxy ● ……………………….
- 3. DATA ORCHESTRATION SUMMIT Agenda ● Alluxio global namespace ● Apache Ozone ● Alluxio UFS framework ●
- 4. DATA ORCHESTRATION SUMMIT 2020 ● Alluxio global namespace Introduction How the unified namespace works in Alluxio with different under file systems
- 7. DATA ORCHESTRATION SUMMIT 2020 Apache Ozone Introduction Introduce Apache Ozone shortly
- 10. DATA ORCHESTRATION SUMMIT 2020 Alluxio UFS framework Introduce about Alluxio UFS
- 11. DATA ORCHESTRATION SUMMIT •./bin/alluxio fs mount --option alluxio.underfs.hdfs.configuration=<DIR>/ozone-site.xml:<DIR>/ core-site.xml /ozone o3fs://<OZONE_BUCKET>.<OZONE_VOLUME>/ How Alluxio UFS works
- 12. DATA ORCHESTRATION SUMMIT Service Discovery ● Dynamically loaded ● Java ServiceLoader . Implements the alluxio.underfs.UnderFileSystemFactory interface. ● Pointing to the class implementation by META_INF/services/alluxio.underfs.UnderFileSystemFactory How Alluxio UFS works
- 13. DATA ORCHESTRATION SUMMIT Dependency Management ● Fat jar ● Shaded jar ● Isolated classloading How Alluxio UFS works
- 15. DATA ORCHESTRATION SUMMIT Implementing an Under Storage Extension •Implementing the required under storage interface
- 16. DATA ORCHESTRATION SUMMIT Implementing an Under Storage Extension •Declaring the service implementation
- 17. DATA ORCHESTRATION SUMMIT Implementing an Under Storage Extension •Add maven plugin to pom.xml for build a package <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> </plugin> <plugin> <groupId>com.coderplus.maven.plugins</groupId> <artifactId>copy-rename-maven-plugin</artifactId> </plugin> </plugins> </build>
- 18. DATA ORCHESTRATION SUMMIT 2020 Introduce how to implement Ozone UFS
- 19. DATA ORCHESTRATION SUMMIT alluxio fs cat /ozone/B
- 21. DATA ORCHESTRATION SUMMIT Hadoop compatible File system(HCFS)
- 25. DATA ORCHESTRATION SUMMIT How to Join Alluxio Community Thank you! alluxio.io/slack