使用 Raspberry pi + fluentd + gcp cloud logging, big query 做iot 資料搜集與分析

Linker Networks Corp.
http://www.linkernetworks.com
Use Raspberry Pi + Fluentd + GCP
Cloud Logging, BigQuery
as IoT Data Collect & Analysis
Simon Su @ LinkerNetworks

var simon = {/** I am at GCPUG.TW **/};
simon.GDE = 'https://developers.google.com/experts/people/peihsin-su';
simon.nodejs = ‘http://opennodes.arecord.us';
simon.googleshare = 'http://gappsnews.blogspot.tw'
simon.nodejsblog = ‘http://nodejs-in-example.blogspot.tw';
simon.blog = ‘http://peihsinsu.blogspot.com';
simon.slideshare = ‘http://slideshare.net/peihsinsu/';
simon.email = ‘simonsu.mail@gmail.com’;
simon.say(‘Let’s go Cloud!');

What we use...
● Docker
● Fluentd
● Google Cloud Logging
● Google BigQuery
● SpreadSheet

Docker
● Build, Ship and Run any
App, Anywhere
● DevOps spirit
● Fast to scale

Fluentd
● Easy and fast
● Message delivery for at
most once, at least
once, exactly once

BigQuery
● TB Level data analysis
● Fast mining tool
● SQL like interface
● Multi-dataset join support
● Cheap & Pay by Use
Google BigQuery - Big data with SQL like query feature, but fast...

Architecture
● BigQuery: Data Warehouse and
support query and analysis
● Cloud Logging: Log gateway
and easy to transfer to one or
multi destination with guarantee
● Fluentd: Device layer gateway
and support multi source for
user to extend http(s) input
tcp/udp input
forward / secure forward
other inputs
● Customize schema
● Co-relation
● Higher quota and
performance
● Web console
● Export available for BQ,
GCS, Pub/Sub
● Configure input source

Prepare Log Gateway using
Cloud Logging

Still no GCP project?? (https://cloud.google.com/free-trial/)

curl -sSO https://dl.google.com/cloudagents/install-logging-agent.sh
sudo bash install-logging-agent.sh
service google-fluentd start
Install Cloud Log Agent From: https://cloud.google.com/logging/docs/agent/installation

Configure
# /etc/google-fluentd/google-fluentd.conf
<source>
type http
port 9880
</source>
<source>
@type forward
port 24224
</source>
<filter *.**>
@type stdout
</filter>
# Do not collect fluentd's own logs to avoid infinite loops.
<match fluent.**>
type null
</match>
# Configure all sources to output to Google Cloud Logging
<match *.**>
type google_cloud
# Set the chunk limit conservatively to avoid exceeding the limit
# of 10MB per write request.
buffer_chunk_limit 2M
flush_interval 5s
# Never wait longer than 5 minutes between retries.
max_retry_wait 300
# Disable the limit on the number of retries (retry forever).
disable_retry_limit
# Use multiple threads for processing.
num_threads 8
</match>
Set config to open fluentd forward
and also screen output for debug..

Test run & watch log...
# service google-fluentd start
# tail -f /var/log/google-fluentd/google-fluentd.log
2016-09-02 09:46:44 +0000 hddusage.simon-project.a_54ERXdr4inKCGiAAAA:
{"catg":"hddusage","ctime":null,"size":null,"used":null,"avail":null,"use":null,"linker_devid":"simon-project.a_54ERXdr4inKCGiAAAA"}
….(skip)

Install Docker into Fluentd
http://blog.hypriot.com/downloads/

Build the Fluentd Docker for Pi

Getting start with fluentd… (https://peihsinsu.gitbooks.io/fluentd-in-action/content/)

Build your Raspberry Pi Fluentd - Dockerfile
# Dockerfile
FROM hypriot/rpi-alpine-scratch
MAINTAINER linkeriot
RUN adduser -D -g '' -u 1000 docker
ENV FLUENTD_VERSION 0.14
RUN apk --update add
build-base
ca-certificates
ruby-dev
&&
rm -rf /var/cache/apk/* &&
echo 'gem: --no-document' >> /etc/gemrc &&
gem install fluentd -v $FLUENTD_VERSION &&
apk del build-base
WORKDIR /fluentd
RUN mkdir log etc plugins config.d
COPY fluent.conf /fluentd/etc/
ONBUILD COPY fluent.conf /fluentd/etc/
ONBUILD COPY plugins/* /fluentd/plugins/
ENV FLUENTD_OPT=""
ENV FLUENTD_CONF="fluent.conf"
EXPOSE 24224
EXPOSE 8888
VOLUME ["/fluentd/log"]
CMD fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT

Build your Raspberry Pi Fluentd - fluentd.conf
# fluentd.conf
@include /fluentd/config.d/*.conf
<label @mainstream>
<filter **>
@type record_transformer
<record>
linker_devid "#{ENV['TAG']}"
</record>
</filter>
<match **>
@type forward
send_timeout 60s
recover_wait 10s
heartbeat_interval 1s
phi_threshold 16
hard_timeout 60s
<server>
name loggateway
host "#{ENV['FDN']}"
port "#{ENV['FPORT']}"
weight 60
</server>
<secondary>
@type file
path /var/log/fluent/forward-failed
</secondary>
</match>
</label>

Build and publish
● docker build -t your-image-name .
● docker push your-image-name

Run from your images
docker run -d -p 24224:24224 --restart=always
-e FDN=[remote-server-ip]
-e FPORT=[remote-server-port]
-e TAG=[your-tag]
-v /var/log:/fluentd/log
-v /data/cfg:/fluentd/config.d
yout-image-name

Run using linker build images
docker run -d -p 24224:24224 --restart=always
-e FDN=[remote-server-ip]
-e FPORT=[remote-server-port]
-e TAG=[your-tag]
-v /var/log:/fluentd/log
-v /data/cfg:/fluentd/config.d
linkeriot/iot-log-agent

Enable log forward to BigQuery...

Create Apps Script from Google SpreadSheet

Using my sample...
Get source from gist: http://goo.gl/J90QD6

使用 Raspberry pi + fluentd + gcp cloud logging, big query 做iot 資料搜集與分析

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to 使用 Raspberry pi + fluentd + gcp cloud logging, big query 做iot 資料搜集與分析 (20)

More from Simon Su (16)

Recently uploaded (20)

使用 Raspberry pi + fluentd + gcp cloud logging, big query 做iot 資料搜集與分析