Minio ♥ Go
Download as PPTX, PDF0 likes1,164 views
This document provides an overview and instructions for using Minio, an open source object storage server. Minio is lightweight and can be bundled with applications. It is compatible with Amazon S3 and supports storing large objects up to 5TB in size. The document describes how to install, run and deploy Minio in standalone, distributed and erasure coded modes. It also covers key Minio features like distributed locking, data distribution, protection against bitrot, and client tools.
![Data Distribution (Details)
// encodeData - encodes incoming data buffer into
// dataBlocks+parityBlocks returns a 2 dimensional byte array.
func encodeData(dataBuffer []byte, dataBlocks, parityBlocks int) ([][]byte, error) {
rs, err := reedsolomon.New(dataBlocks, parityBlocks)
if err != nil {
return nil, traceError(err)
}
// Split the input buffer into data and parity blocks.
var blocks [][]byte
blocks, err = rs.Split(dataBuffer)
if err != nil {
return nil, traceError(err)
}
// Encode parity blocks using data blocks.
err = rs.Encode(blocks)
if err != nil {
return nil, traceError(err)
}
// Return encoded blocks.
return blocks, nil
}](https://image.slidesharecdn.com/kvvs1jl7tkulbkcn6bxl-signature-78f96e88cbb8f48c72e7df42a1bd40758a18e4be540cff9641b10ea2b3b915fc-poli-170724175918/85/Minio-Go-10-320.jpg)
![On disk format
Describes each object uniquely for Erasure Code and Bitrot.
Human readable and parser friendly.
Replicated across all disks.
"parts": [ {
"etag": "7f614da9329cd3aebf59b91aadc30bf0",
...
}, ... ],
"erasure": {
"checksum": [ {
"hash": "HASH-1",](https://image.slidesharecdn.com/kvvs1jl7tkulbkcn6bxl-signature-78f96e88cbb8f48c72e7df42a1bd40758a18e4be540cff9641b10ea2b3b915fc-poli-170724175918/85/Minio-Go-14-320.jpg)
![Minio Client (MC)
A cross platform modern alternative to UNIX commands like ls, cp,
diff etc.
Supports filesystems and Amazon S3 compatible object storage.
Install
$ go get -u github.com/minio/mc
$ mc ls play
[2017-02-12 22:35:16 PST] 0B ferenginar/
[2017-02-12 15:08:45 PST] 0B andoria/
[2017-02-12 23:24:21 PST] 0B romulus/](https://image.slidesharecdn.com/kvvs1jl7tkulbkcn6bxl-signature-78f96e88cbb8f48c72e7df42a1bd40758a18e4be540cff9641b10ea2b3b915fc-poli-170724175918/85/Minio-Go-16-320.jpg)
Minio ♥ Go
- 1. Minio ♥ Go A Cloud Native Object Storage Harshavardhana
- 2. What is Minio? ☕ Minio is an object storage server released under Apache License v2.0. Compatible with Amazon S3 cloud storage service. Best suited for storing photos, videos, backups and container / VM images. Size of an object can range from a few KBs to a maximum of 5TB. Is light enough to be bundled with the application stack, similar to NodeJS, Redis.
- 3. Install Binary $ wget https://dl.minio.io/server/minio/release/linux-amd64/minio $ chmod +x minio $ ./minio version Version: 2017-01-25T03:14:52Z Release-Tag: RELEASE.2017-01-25T03-14-52Z Commit-ID: f8e4700a11065967242b8857045ce7dee607722d Docker $ docker run --name minio -p 9000:9000 minio/minio server /export Visit Docker quickstart guide for more details.
- 4. Run Standalone FS (Filesystem) $ ./minio server /dir1 Standalone Erasure $ minio server /dir1 /dir2 /dir3 ... /dir8 Distributed Erasure $ minio server http://192.168.1.11/dir1 http://192.168.1.12/dir2 http://192.168.1.13/dir3 http://192.168.1.14/dir4 http://192.168.1.15/dir5 http://192.168.1.16/dir6 http://192.168.1.17/dir7 http://192.168.1.18/dir8 Distributed FS (Filesystem)
- 7. Dsync (Distributed Syncing) No master node concept. Resilient applies quorum techniques. Drop-in replacement for sync.RWMutex / sync.Locker. Automatically reconnect to (restarted) nodes. Limited scalability and fixed configuration: up to 16 nodes. Will not (re)acquire any stale locks.
- 8. Example Write Lock func lockSameResource() { // Create distributed mutex to protect resource 'test' dm := dsync.NewDRWMutex("test") dm.Lock() log.Println("first lock granted") go func() { time.Sleep(5 * time.Second) log.Println("first lock unlocked") dm.Unlock() // Unblock first lock }() log.Println("about to lock same resource again...") dm.Lock() // Waits here. log.Println("second lock granted") time.Sleep(2 * time.Second) dm.Unlock() } Visit github.com/minio/dsync for more examples.
- 10. Data Distribution (Details) // encodeData - encodes incoming data buffer into // dataBlocks+parityBlocks returns a 2 dimensional byte array. func encodeData(dataBuffer []byte, dataBlocks, parityBlocks int) ([][]byte, error) { rs, err := reedsolomon.New(dataBlocks, parityBlocks) if err != nil { return nil, traceError(err) } // Split the input buffer into data and parity blocks. var blocks [][]byte blocks, err = rs.Split(dataBuffer) if err != nil { return nil, traceError(err) } // Encode parity blocks using data blocks. err = rs.Encode(blocks) if err != nil { return nil, traceError(err) } // Return encoded blocks. return blocks, nil }
- 11. Data Protection
- 12. Bitrot (blake2b hash) SIMD accelerated pure Go implementation using Go Assembly. Three flavors: AVX2, AVX and SSE Speed up over pure go (non-assembly): 3.94x Performance comparison with other checksum techniques on AVX2. $ go test -bench=ComparisonAVX2 BenchmarkComparisonMD5-12 1000 1726121 ns/op 607.48 MB/s BenchmarkComparisonSHA1-12 500 2005164 ns/op
- 13. Bitrot (sha256 hash) on ARM64 Used in AWS S3 payload signature verification and bitrot on GOARCH:ARM64 Speed up over pure go on ARM64: 105x | Processor | Package | Speed | Improvement | | --------------------------------- | ---------------------------- | --- --------:| -----------:| | 1.2 GHz ARM Cortex-A53 | minio/sha256-simd (ARM64) | 638.2 MB/s | 105x | | 2.4 GHz Intel Xeon CPU E5-2620 v3 | minio/sha256-simd (AVX2) (*) | 355.0 MB/s | 1.88x | | 2.4 GHz Intel Xeon CPU E5-2620 v3 | minio/sha256-simd (AVX) | 306.0 MB/s | 1.62x | | 2.4 GHz Intel Xeon CPU E5-2620 v3
- 14. On disk format Describes each object uniquely for Erasure Code and Bitrot. Human readable and parser friendly. Replicated across all disks. "parts": [ { "etag": "7f614da9329cd3aebf59b91aadc30bf0", ... }, ... ], "erasure": { "checksum": [ { "hash": "HASH-1",
- 15. Client Tool and Libraries
- 16. Minio Client (MC) A cross platform modern alternative to UNIX commands like ls, cp, diff etc. Supports filesystems and Amazon S3 compatible object storage. Install $ go get -u github.com/minio/mc $ mc ls play [2017-02-12 22:35:16 PST] 0B ferenginar/ [2017-02-12 15:08:45 PST] 0B andoria/ [2017-02-12 23:24:21 PST] 0B romulus/
- 17. Minio Client Library - Upload a local file. package main import ( "log" minio "github.com/minio/minio-go" ) func main() { // Initialize a new s3Client. s3Client, err := minio.New("play.minio.io:9000", "Q3AM3UQ867SPQQA43P2F", "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG", true) if err != nil { log.Fatalln(err)
- 18. Questions? Community Channel slack.minio.io Documentation docs.minio.io