Building Scalable Data Ingestion
Download as PPTX, PDF0 likes59 views
Michael Pearce discusses building scalable data ingestion solutions at Peak AI, emphasizing the importance of simplicity, security, and automation in data workflows. The document outlines various technical considerations for integrating AWS services, such as SFTP, S3, and EC2, to enhance data transfer and management. Additionally, it highlights job opportunities at Peak AI, including roles in development and UX design.
Building Scalable Data Ingestion
- 1. Building Scalable Data Ingestion Michael Pearce Senior Engineering Manager (DevOps) @ Peak AI
- 2. 01 02 03 About Mike Automation Infrastructure Security
- 3. About Peak Helping customers overcome Barrier to Entry for AI Identify Opportunity, Quantify, Business Case Kick Off, Onboarding, Live Service & Support “Off the Shelf” Solutions Customer AI Demand AI Supply AI End to End AI Platform Ingest Unify, Transform, Model Provide Insights AWS Advanced Consulting Partner ML Competency Retail Competency
- 4. AI needs data. Collecting it should be: ● Simple ● Effective
- 5. Secure File Transfer Protocol SFTP
- 6. AWS Transfer for SFTP Seamlessly migrate your file transfer workflows to AWS Transfer for SFTP Integrating existing authentication (IAM) providing DNS routing (Route 53) Nothing changes for your customers and partners, or their applications… There is no infrastructure to buy and setup.
- 7. So why build your own!? Compatibility Issues Configurability Product Maturity
- 8. Must be: ● Scalable ● Highly Available ● Fault Tolerant Building a custom solution
- 9. EC2 ● Private Subnet ● Load Balancer ● Security Groups ● Route 53 Making it more secure and simple
- 10. S3 ● Separate Servers and Data ○ Sync with S3 ■ AZ replication ● IAM instance role and policy to manage access OR ● Private Link S3 Endpoint ○ Traffic stays in AWS network ○ Access bucket privately without authentication when you access the bucket from a VPC that has an endpoint to S3 ● Versioning, Logging, etc. ○ Auditing, recover from accidental deletion Added resilience, availability and security - Data
- 11. Auto Scaling Group ● Launch Template ○ AMI, Instance Type etc. ● Health check replacements ● Scaling Policies ○ Elastic ○ Cost effective ● AZ spread Fault tolerance and availability - Servers
- 12. User Management ● Systems Manager (SSM) ● Secrets Manager ● Step Functions Simple and Automatable
- 13. Disk Management ● CloudWatch Agent ○ CloudWatch Alarms ● Step Functions ● Still using EBS as an intermediary ○ Data duplication ○ Scaling out could mean big cost implications ● Or just use EFS!? ○ Or Multi Attach EBS Provisioned IOPS volumes (NEW) Proactive monitoring and resolution
- 16. Find out more
- 19. Engineering Opportunities Open Positions: ● Head of Development ● UX Designer ● Lead Product Manager Coming Soon: ● Senior DevOps ● Support Engineer Manchester, UK https://peak.ai/company/careers For all our latest roles, please head to our Peak careers page!
- 20. Building Scalable Data Ingestion Michael Pearce Senior Engineering Manager (DevOps) @ Peak AI
Editor's Notes
- #2: My name is Michael, Mike for short. I’m a DevOps Engineer @ Peak AI. I’m based in Manchester and I work to enable developers and data scientists to be autonomous and self sufficient, empowering them to do great things with data!
- #3: I’m passionate about building great Automation Infrastructure security with Linux and AWS.
- #4: Barriers to Entry - building systems and hiring ml and ds Also help client identify opportunities in their data From connecting to data sources to Ingest data, unifying that data, exploring it, transforming it, building ML models to extract even for insight from it then providing valuable output in the form of visual dashboards, api endpoints or even just raw data. ‘off the shelf’ solutions - speed up putting ai into the enterprise right the way through the business. Last but not least, Consulting partners, ML Retail Status
- #5: Especially from a client perspective A few different options using AIS - ingest agent, signed url, webhook, but there was something else customers kept asking for...
- #6: One tried and tested method of data transfer, still very popular - SFTP. Secure File Transfer Protocol. Hands up to has heard of SFTP? Keep hands up if you’ve used it? Keep your hands up if you’ve had some kind of hard time with SFTP? Managing servers Server dies Disk fails or fills up Managing users Security
- #7: AWS Transfer for SFTP Seamlessly migrate your file transfer workflows to AWS integrating existing auth - IAM DNS routing (Route 53) Nothing changes for your customers and partners, or their applications No infrastructure to buy and setup. With your data in S3, you can use it with AWS services for processing, analytics, machine learning, and archiving. Getting started with AWS Transfer for SFTP (AWS SFTP) is easy;
- #8: Compatibility Issues Mainly Windows! Transfer endurance (temp file names for part upload) Preserving timestamps Configurability Gotcha - VPC, use Network LB Only login with an access key is supported. That means: There is simply no access via username and password. General Product Maturity Classic AWS development, just watch this space!
- #9: A simple EC2 with a SFTP service running would be good, but we want it to be great! Following some architecture best practices (and doing anything being woken up by pagerduty at 3am to reboot a server) How to AWS do it? Could we do that, plus extra customisation? Moving onto how we developed it, as we went along...
- #10: Put the instance in a Private Subnet Load Balancer (also helps absorb DDoS attacks!) Could even use CloudFront Locked down using Security Groups Route traffic using Domain name (not an IP address)
- #11: Why we use S3 Followed AWS Transfer example Quick transfer, quick setup Multi AZ Why not EFS? EBS Provisioned IOS now multi attach EC2!!
- #14: We didn’t consider EBS Not predicting much load yet As the disk fills up, so does S3 (Duplication) If you scale out, you need all the EBS volumes attached to be big enough to sync all the data! EFS would provide Multi AZ One shared disk Elastic, no need for complicate resizing Why we use S3 Followed AWS Transfer example Quick transfer, quick setup Multi AZ
- #16: EFS alternative - no need for EBS, Resize Step function or S3
- #18: Read some of our tech blogs on Medium or the Website
- #19: Open sourced projects on Github
- #21: Any Questions?