Building Infrastructure for an IT Organization
- 1. Building Infrastructure for an IT Organization Daniel Jude Gonsalves Plan9 Labs
- 2. Agenda: A Cloud Architect Project Introduction to Plan 9 Abstract Requirements Current System Research Work Proposed System Real World Testing Timeline of the project Final costs Summary BUILDING INFRASTRUCTURE FOR AN IT ORGANIZATION
- 3. Introduction to Plan 9 Plan 9 Labs is an automation company that provides data robots to companies needing data transfer between data sources or CRMs. Removing Noise from existing data and provisioning robots that can sync data across multiple services is the key aim of Plan9 to secure proper data tagging, routing, and mapping between data sources. BUILDING INFRASTRUCTURE FOR AN IT ORGANIZATION
- 4. PLAN 9 AUTOMATION Web App Servers Blogs and Websites Adobe Experience Cloud CRMs Milestones Database Dashboard and Reporting Plan 9 Bot Plan 9 Bot Plan 9 Bot Segment Destination Data lake Destination Web App source Server source Multiple other sources
- 5. Requirements To upgrade Plan 9’s Infrastructure
- 6. Requirements SERVER Single Server with Backups and a multi hour MTTR (manual restore). OS Support for FreeBSD Must run Ubuntu 22.04 LTS No desktop GUI/ CLI only SPECS 8-16 Core CPU 32-64 GB RAM 500-2 TB STORAGE (SSD) MONITORING & BACKUP Live monitoring SMS alerting Backup with AWS S3 buckets BUILDING INFRASTRUCTURE FOR AN IT ORGANIZATION
- 7. Current System Linode Server with 8 Cores, 16GB RAM and 300 GB RAM. Support for Ubuntu 22.04 LTS , but no support for FreeBSD Backup done through AWS S3 buckets. Whole images of data backed up daily and can be restored in 4-6 Hours BUILDING INFRASTRUCTURE FOR AN IT ORGANIZATION AWS S3 Linode VPS Cloud Server
- 8. Research Work Determining the Type of Server Virtual Private Server, Virtual Data Center or Hosted Dedicated Server Backup and Failover Solutions Having a Failover of less than 2 hours if the server is down is important Best Server Location Strategies To provide for the least latency, the server should be located on the US east coast. Monitoring and Alerts via SMS and Email For instant response to restore servers manually (automatic restore not required). 1 2 3 4 BUILDING INFRASTRUCTURE FOR AN IT ORGANIZATION
- 9. Research Work Determining the Type of Server Virtual Private Server, Virtual Data Center or Hosted Dedicated Server 1 BUILDING INFRASTRUCTURE FOR AN IT ORGANIZATION Virtual Private Server- It is a server virtualized on physical server hardware, providing scalability and flexibility of configurations with its secondary storage virtualized using LUNs with stacks of HDD or SSD units. This is advantageous for scalability and flexibility only, but read/write speeds are significantly slower Virtual Data Center- It is a collection of VPSes connected over virtualized networks and having databases, monitoring servers, and other servers all on a single physical virtualized infrastructure. Only provided by Ionos Hosted Dedicated Server: It is a physical rack-based server that is configured with physical components and is hosted over a network. High performance but loses on Virtualization features. Finally, the best type of Server boils down to the pricing and what fits in Plan9’s budget.
- 10. Research Work Plan9 had a budget of 600 USD(per month) to spare for their whole infrastructure. We had to configure everything including servers, backup, monitoring and failover for that price Research work was started to find the best servers for the money, according to the specifications provided. This table is a small snippet of the prices we found. (in USD per Month) We chose InterServer and Ionos as our finalists because of the price, and a hosted dedicated server because of performance. We could thus use our budget to configure better backup infra. Specifications AWS Digital Ocean Google Cloud InterServer Ionos 32GB RAM Server 720 252 908 98 185 64GB RAM Server - 504 - 110 210 128GB RAM Server - 1008 - 209 256 Type of Server VPS VPS VPS DS & VPS DS VPS & VDC
- 11. Research Work Backup and Failover Solutions Having a Failover of less than 2 hours if the server is down is important 2 BUILDING INFRASTRUCTURE FOR AN IT ORGANIZATION Plan 9’s current system consists of a single server with images being pushed to AWS S3 bucket server. There is also a redundant backup which is pushed to dropbox. However, with the new system, we propose a 2 server system, where the second server takes over immediately in case of the first server failing. PostgreSQL data is pushed to both servers at the same time, properly syncing data between the two servers. Bots still push a snapshot of data daily to Amazon S3 (more snapshots daily to be implemented soon). Amazon S3 data can be restored onto a new server configured within minutes and the bots redeployed in less than an hour just like the previous configuration. Dropbox is still used in case Amazon S3 fails
- 12. Research Work Best Server Location Strategies To provide for the least latency, the server should be located on the US east coast 3 BUILDING INFRASTRUCTURE FOR AN IT ORGANIZATION Plan 9’s main servers need to be located on the US east coast to prevent any latency and network issues that may arise. Having the server in the same regional location is also beneficial for application and data processing performance. The Linode server that Plan 9 uses is based in New York, and Plan 9 needed something similar for their server upgrade. Both our finalists Ionos and InterServer are located in New Jersey, which is a city a few miles from New York, and on the east coast. For the second server, we needed a server from a different region and location to prevent data center failures, or environmental hazards which can cause a certain server in a region to fail (ex- storms, hurricanes, tornados or earthquakes) Therefore, we configured the second server in central US- Las Vegas
- 13. Research Work Monitoring and Alerts via SMS and Email For instant response to restore servers manually (automatic restore not required) 4 BUILDING INFRASTRUCTURE FOR AN IT ORGANIZATION 13 Plan 9 has always used a monitoring system to alert them when the server was down. Linode offered good services for their VPS products with SMS alerting and stats monitoring. The main open-source monitoring solutions that we discovered were Zabbix and Nagios. These are cli configurable monitoring that requires a server to run and lots of configuration. This was however very complex to install and run and provided features that wouldn’t be useful to us. Therefore, we dropped that and used a domain monitoring solution called UptimeRobot which could monitor the status of the domain (which we configured using Namecheap, for our server) This was also within budget and costed only 30 USD per month.
- 14. Proposed System Main Server is located in NJ and Backup Server in LA All the bots are run as Cron jobs (scheduled tasks) We use an AWS S3 server as an archive for the data being created or run through the robots on the main server. PostgreSQL database is run on both servers and synced together through the AWS S3 server. ML Training is done on the main server or Plan 9’s development laptop. AWS S3 Main Server Backup Server
- 15. Proposed System AWS S3 Main Server Backup Server UptimeRobot Monitoring Web Server All the data is also pushed through to Dropbox, in case the S3 instance fails. It takes longer to restore from Dropbox, but it is a proper archiving solution. Uptime Robot monitors both servers. The web app is run on the backup server, and the website is on a web server (Plan 9 could switch to a hosting provider in the future) Dropbox
- 16. Real World Testing For every product or IT asset, Testing is important to ensure that the product is functional and performs well. Such was the case with our finalists- Ionos Intel Xeon 6126 and InterServer’s AMD EPYC 7402P. We tested it using a high-resource-demanding bot created from Plan 9 The resultant scores were disappointing against the Plan 9 development laptop. This made us switch to a desktop-class processor which would perform better than the Plan 9 laptop – AMD Ryzen 9 5950X. This config had less RAM, but this processor was much faster than the other Specifications Bot Write Copy Linode VPS- 16GB RAM 8 Core Intel 27.9 13.7 1.6 Ionos Intel Xeon 6126 25.6 16.8 1.4 InterServer AMD EPYC 7402P 18.7 10.9 1.3 Plan 9 Development Laptop 20.2 8.3 0.8 InterServer AMD RYZEN 5950X 14.6 8.1 1.2 server processors, which was more beneficial. We also chose a lower spec of the Ryzen (5900X) for the backup server.
- 17. Project Timeline Project Requirement Allocation Jan 2 Budget and Metrics Calculation Jan 4 Application Analysis Jan 5 Research Work to find the best providers Jan 10 Plan9 chooses the best finalists, starts work on testing Jan 16 Servers Provisioned Jan 18 Performance Testing Jan 20 Provisioning AMD 5950X Jan 25 Installing Software – python, bots etc Feb 2 Research work for monitoring solutions Feb 5 UptimeRobot Configured Feb 10 Backup Server configured Feb 15 Server Migration Feb 20 AWS S3 Feb 21 Fixing Bugs and Issues Feb 23 Dropbox config Feb 27 Project Audit Mar 1 Project Complete Mar 10 BUILDING INFRASTRUCTURE FOR AN IT ORGANIZATION
- 18. Final Costs Budget- USD 600 P.M. When we started with this budget, it was challenging to build a solution to fit this limited budget. However, we now have a fully-fledged solution, which is high performing and functional. BUILDING INFRASTRUCTURE FOR AN IT ORGANIZATION Component Product Cost Main Server InterServer AMD Ryzen 9 5950X 128GB RAM, 2TB NVMe 209 USD Backup/ Webapp Server InterServer AMD Ryzen 9 5900X 128GB RAM, 2TB NVMe 125 USD Domain NameCheap 20 USD Storage Backup Amazon S3 2TB 10 USD Archive Backup Dropbox Premium 2TB 20 USD Monitoring UptimeRobot 30 USD Total Costs Per Month 414 USD
- 19. Summary As an infrastructure solutions architect, the aim of this project was to make Plan 9’s infrastructure have the best specifications to suit their company’s application while sticking to the budget- thus creating the most value. I was able to save Plan 9 around 176 USD per month while building a solution that will benefit Plan 9 for years to come. BUILDING INFRASTRUCTURE FOR AN IT ORGANIZATION
- 20. Thank you Daniel Jude Gonsalves 111720036021 In collaboration with Plan 9 Labs