Storage, retreival and process of continuous streaming data in a widearea frequency measurement system

Project Report on

Storage, retreival and process of continuous streaming data in a Wide
Area Frequency Measurement System

by

Nitesh Pandit
Kedar Khandeparkar

under the guidance of

Prof. A.M. Kulkarni

Department of Electrical Engineering
Indian Institute of Technology, Bombay
Mumbai-400076

July 2010

Contents

0.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . 2
0.1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . 2
0.2 Current Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
0.2.1 Packet Format . . . . . . . . . . . . . . . . . . . . . . 3
0.2.2 Need of Project . . . . . . . . . . . . . . . . . . . . . . 4
0.3 Data Reception . . . . . . . . . . . . . . . . . . . . . . . . . . 5
0.3.1 Requirement . . . . . . . . . . . . . . . . . . . . . . . 5
0.3.2 Current Approach . . . . . . . . . . . . . . . . . . . . 5
0.3.3 Our Approach . . . . . . . . . . . . . . . . . . . . . . 5
0.4 Data Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
0.4.1 Requirement . . . . . . . . . . . . . . . . . . . . . . . 7
0.4.2 Current Approaches . . . . . . . . . . . . . . . . . . . 7
0.4.3 Possible Approaches . . . . . . . . . . . . . . . . . . . 7
0.4.4 Our Approach . . . . . . . . . . . . . . . . . . . . . . 7
0.4.5 Pseudocode for Insertion & Updation in Database . . 8
0.5 Data Display . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
0.5.1 Requirement . . . . . . . . . . . . . . . . . . . . . . . 9
0.5.2 Current Approach . . . . . . . . . . . . . . . . . . . . 9
0.5.3 Possible Approaches . . . . . . . . . . . . . . . . . . . 9
0.5.4 Our Approach . . . . . . . . . . . . . . . . . . . . . . 9
0.6 Handling database issues . . . . . . . . . . . . . . . . . . . . . 11
0.6.1 Running the program . . . . . . . . . . . . . . . . . . 12
0.7 Literature Survey . . . . . . . . . . . . . . . . . . . . . . . . . 14
0.7.1 MYSQL . . . . . . . . . . . . . . . . . . . . . . . . . 14
0.7.2 PHP . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
0.8 Future Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
0.8.1 Interpolation of Frequencies . . . . . . . . . . . . . . . 15
0.8.2 Daily triggers . . . . . . . . . . . . . . . . . . . . . . . 15
0.8.3 Threading . . . . . . . . . . . . . . . . . . . . . . . . . 15
0.9 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

1

0.1 Problem Statement
Storage, retreival and process of continuous streaming data in a Wide Area
Frequency Measurement System and further analysis to generate the flags
and alerts.

0.1.1 Introduction
Power System Network is continuously subjected to disturbances in the form
of sudden load and generation changes. These disturbances give rise to os-
cillations in the rotor angle which are also seen in the frequency. To study
these oscillations wide area frequency measurement setup is implemented in
this project. The setup are strategically placed in five different places in
India and are time synchronized via Network Time Protocol (NTP). The
frequency is measured every 20 ms and is time stamped and send through
internet to a server in IITB.

The server program which is continuously receiving the packets, checking
the correctness of packet information, doing the insertion in database ac-
cording their time and displaying it on the web.

2

0.2 Current Setup
In ‘Wide area frequency measurement system’ frequencies measured at dif-
ferent places in India are sent to a IITB Server. Frequencies are measured
by sensors at every 20 ms. Local frequency measured is stamped with the
time and place and sent to IITB Server.
The IITB server is continuously listening for the packets on UDP port 6000.
When a packet is received, it extracts the data and writes it in a file. A sep-
arate file is created for each sensor and when any file reaches its maximum a
new file is created for each sensor with incremented serial number. Plotting
of this data is done offline.

0.2.1 Packet Format

Where,
Source Port = 6000
Destination Port = 6000
Checksum = Default
Data = “( Sensor name, year-month-day hours minutes seconds.milliseconds,
Frequency value )”

3

0.2.2 Need of Project
In existing scenario, data from each packet is stored in files at IITB sever.
The analysis of data in these files is a difficult task and is done later offline.
This project deals with online display of the incoming frequency data. Also
it does constraint checking of each packet such as authenticity of location
of sensor, validity of frequency range etc. If a packet passes such validation
checks it is stored in the database for further analysis.

Hardware Requirements
1) Intel P4 processor
2) RAM (512 MB)

Software Requirements
1) C (gcc compiler)
2) Perl (libperl-dev, libdbd-mysql-perl)
3) MySql (mysql-server, mysql-client, libmysqlclient15-dev)
4) PHP (php5, php5-gd, php5-mysql, php-cli)
5) Apache (apache2)
6) Bzip2 (with perl libraries)

4

0.3 Data Reception
0.3.1 Requirement
Data from multiple sensors coming continuously at a very small time inter-
val need to be stored in an organized manner for further analysis.

0.3.2 Current Approach
Data is dumped into flat files as it coming, with a separate file for each
sensor. When any file reaches its maximum limit new files are created for
each sensor with incremented serial number attached as a part of file name.
There is no way for soft real time display of such data. It is also very difficult
to gather data from all the sensors in a particular time range and compare
them. Also current approach does not handle constraint checking.

0.3.3 Our Approach
A UDP Server implemented in C language. It is continuously receiving the
packets from multiple sensors.

Server program comprises of three functions
1. main():

Continuously senses for the packets on UDP port 6000 and on reception
creates a thread by calling a fuction check constraint has calculate(). To
this function the message received is passed as an argument.

2. check constraint hash calculate():
Splits the received, message checks validity of each field and performs hash
calculation. After that it calls insert packets().

5

Logic for ‘check constraint hash calculate()’
I. Each packet has Sensor location Name, Date Time, Frequency"
for ex. "surat, 2010-06-10 12 39 23.545, 49.654". We split
the message by comma and place the values in columns array,
for ex. columns[0]=’surat’ columns[1]=’2010-06-10 12 39 23.545’
columns[2]=49.654.

II.We then perform some validity checks like
a) Check if columns[0] contains the valid sensor name, if not
then entry is made toLog_Error table and no further
processing is done for the packet.

b) Check if columns[1] contains a non null value, if not then
entry is made to Log_Error table and no further processing
is done for the packet.
c) Check if columns[2] contains frequency value that lies between
47 and 52, if not then entry is made to Log_Error table and no
further processing is done for the packet.

d) If the packet passes all the validation checks then we further
process the packet. Splits the received, message checks validity
of each field and performs

d.i) Split Date Time value in packet by space ’ ’
e.g. columns[1]=2009-09-07 10 12 322.743 will be split as
datetime[0]=2009-09- 07,datetime[1]=10 ,datetime[2]=12,
datetime[3]=322.743

d.ii) Split Date by ’-’
e.g. datetime[0]=2009-09-07 will be split as date[0]=2009,
date[1]=09,date[2]=07

d.iii) Split millisecond time(ms) by ’.’
e.g.datetime[3]=322.743 as time[0]=322,time[1]=743

III. Now we calculate the hash value of each packet.
It is the number of millisiseconds for the date time value in
the packet. i.e. 2009-09-07 10 12 322.743 =>(x millisec)

IV. On performing hash calculation we call a function insert_packets()
and pass the calculated hash (i.e. x from above ) and
columns[] as arguments.

6

0.4 Data Storage
0.4.1 Requirement
Storage and retrieval of data should be efficient.

0.4.2 Current Approaches
Data is stored in files, but to retrieve this data and do analysis is very diffi-
cult.

0.4.3 Possible Approaches
Two ways of data storage
a) Store as flat files as is done in the previous approach
b) Use database.

0.4.4 Our Approach
We have used Mysql database for data storage. Logic for insertion of packets
into the database table is a part of main server program i.e. insert packets()
called from check constraint hash calculate() contains the logic to insert
newly arrived packet values in the database table.

7

0.4.5 Pseudocode for Insertion & Updation in Database

Algorithm 0.4.1: InsertPackets(newHash, datetime, f requency)

comment: Insert the packet with $datetime$ and $frequency$ at correct position in the table
global currentHash, minHash, nextHash
local tdif f, tmod, k, i, t2, t3, tempHash
if currentHash! = 0

if newHash >= currentHash


 

 tdif f ← newHash − currentHash

 
if tdif f <= 10

 



  then {UPDATE TABLE(currentHash,$datetime$,$frequecy$)



 
 

 


 
 tmod ← tdif f %20;


 
 if tmod <= 10


 
 
 {

 
 






  then t2 ← tdif f − tmod;



 
 
 t3 ← (integer)(t2)

 
 
 {

  

20

 
 else t2 ← tdif f − tmod + 20;
 then
 


 
  t3 ← (integer)(t2)

 
 else for k ← 0 to t3 − 1 20

 


 
 
 

 
 
 $currentHash$ ← $nextHash$

 
 


 
  do $nextHash$ ← $currentHash$ + 20


 
 
 

 
 
 INSERT INTO TABLE(currentHash,NULL,NULL)

 
 


 
 
$currentHash$ ← $nextHash$

 
 


 
 $nextHash$ ← $currentHash$ + 20


 
 




  INSERT INTO TABLE(currentHash,$datetime$,$frequency$)



 tdif f ← currentHash − newHash

 
tmod ← tdif f %20


then 
if newHash >= minHash

 


 
 

 
 if tmod  10
<=

 
 


 
 
 t2 ← tdif f − tmod;

 
 


 
  then t3 ← currentHash − t2;


 
 

  then


 
   UPDATE TABLE(t3,$datetime$,$frequecy$)

 
 
 t2 ← tdif f − tmod + 20;

 
 


 
  else t3 ← currentHash − t2;


 
 
 

 
 UPDATE TABLE(t3,$datetime$,$frequecy$)

  

 else  if tmod <= 10

 
 {

 
 





  then t2 ← tdif f − tmod;



 
 
 t3 ← (integer)(t2)

 
 
 {

 
 

20





  else t2 ← tdif f − tmod;



 
 
 t3 ← (integer)(t2)

 
 else 20

 
 tempHash ← currentHash

 
 

 
 for k ← 1 to t3 − 1


 
 
 {

 
 






  do tempHash ← tempHash − 20;

 tempHash ←

 
 
 INSERT INTO TABLE(tempHash,NULL,NULL)

 
 


 
 INSERT INTO TABLE( currentHash,$datetime$,$frequency$)


 
 
 8
 minHash ← tempHash
currentHash ← minHash ← newHash
else newHash ← currentHash + 20

INSERT INTO TABLE(currentHash,$datetime$,$frequency$)

0.5 Data Display
0.5.1 Requirement
Graphical display of frequency Values on web.

0.5.2 Current Approach
Offline plotting of frequency stored in files is done in Matlab, below graph
is generated using Matlab and has to be done offline.

0.5.3 Possible Approaches
1. Java Applets
2. JPGraph
3. PHP
4. Asp.net

0.5.4 Our Approach
We have chosen PHP and GD library to plot graphs of frequency data on
web as it is an open source allows to build dynamic graphs. It also has less
overhead compared to other web designing techniques.

Server Requirements a) Apache server
b) PHP (with GD)
c) MySql

9

Steps involved in Display:

1. In the first step get the fixed size graph (image) on the web browser
by using the following functions,

ImageCreate ( int $width , int $height ).
It returns an image identifier representing a blank image of specified size. In
addition to ImageCreate() that created a graph, there are other functions
used to fill the information in the graph such as drawing X-axis and Y-axis
lines, plotting appropriate axis names, defining the color values etc.

ImageLine (resource $image,int $x1,int $y1,int $x2,int $y2,int $color)
Imageline function draws line between given points. This function will draw
a line between the points ($x1,$y1) & ($x2,$y2) and assign the color speci-
fied by $color.

ImageString (resource $image,int $font,int $x,int $y,string $string,int
$color)
ImageString function is used to display a text on the given coordinates of
graph.
Here ($x,$y) is a coordinate point on the graph where $string will be written.

ImageColorAllocate (resource $image,int $red,int $green,int $blue)
ImageColorAllocate returns a color identifier representing the color com-
posed of the given RGB components. To plot a line or write a string on the
graph we require colors. Imagecolorallocate() is used to associate color with
objects

.
2. After the first step a blank graph is created and axis are drawn php-
mysql database connection is established.
We use $con=mysql connect ( ’localhost’, ’username’, ’password’ ); mysql select db
(‘database name’, $con);

3. In third step we run mysql queries to fetch the records from table
sensor data. A view named display is created that will fetch last 10 sec-
onds records from the table sensor data. Also, there are other queries to
obtain minimum and maximum frequency values for the records obtained
from view. These are required to decide the limits of values on Y-axis on
each page refresh.

10

4. Fourth step is actual plotting of the frequency values. Time in seconds
on X-axis and frequency on Y-axis. On each page refresh, which happened
every second a new record set is fetched from the sensor data table. This
fetched data is plotted dynamically by adjusting X axis and Y axis. There
are two files, WAFMES.html and graph.php. WAFMES.html page calls
graph.php every second on page refresh. Every new second value is plotted
from right to left side. If the frequencies change suddenly it will be seen on
the display as spikes.

5. In the fifth step we handle exceptions and errors. If any of data source
failed and the data is not coming at server we raise a flag. This is done by
displaying a message “X Currently Off” where X is the name of the data
source. Null values in between the rows are also handled at display time.
These Null values in the database table may be due to UDP packet lost or
transmission delay or sensor side packet generation. These null values are
displayed as sudden spikes on graphs. To overcome this problem we have
taken previous frequency value in place of null only at display time and
plot. The actual entry in the database table is left intact. However if there
are continuous null values for a period of 10 seconds then we consider that
sensor is down and raise the flag message.

0.6 Handling database issues
Since approximately 4.5 millions of rows are inserted to ‘sensor data’ table
every day and its maximum limit is 5.0 billion records we need to constantly
move the records from the table to make the space for the new data. For
this ‘Backup.pl’ script is run on daily basis. The job is scheduled in cron-
job. This will run at 1 am every day and move the last day entries from
table to a file compress the file with bzip2 and store it in the folder sen-
sor data backup. The system date is part of filename. After taking the
backup of the day those rows are deleted from the table sensor data.

11

0.6.1 Running the program
Step 1

A shell script is written, so that when need to add a new sensor name,
add only his name in text file (but it must be in sequence as per in php file
and also need to update php file according to changes)

first run ./setup.sh to create database
then ./run.sh to run the server

Otherwise
Making the database ready
Need to be done only first time.
CREATE DATABASE WAFMES;
USE WAFMES;
CREATE TABLE sensor data (
hash decimal(14,0) NOT NULL,
surat_time varchar(25),
surat_value float ,
sangli_time varchar(25),
sangli_value float,
ahm_time varchar(25),
ahm_value float,
mum_time varchar(25),
mum_value float,
pune_time varchar(25),
pune_value float,
bhu_time varchar(25),
bhu_value float,
PRIMARY KEY(hash)
);
CREATE TABLE Log Error (
sensor_name varchar(10),
time varchar(30),
freq varchar(10));

CREATE VIEW display AS
SELECT * FROM sensor_data where
hash>=((SELECT max(hash) FROM sensor_data)-12000) && hash<=((SELECT
max(hash) FROM sensor_data)-2000)ORDER BY hash DESC;

12

Make an entry in cronjob for the file Backup.pl to be run on daily basis.
It will get the last one day entries (approx 4.5 million) from database ta-
ble (sensor data) and write in a file and compress, take the date as file name.

Step 2

When a new sensor is added we need to alter the table and create columns
for the new sensor. For ex. if x is a sensor location that has been newly
established, then make the following changes to the table.

ALTER TABLE sensor data add x time varchar(25), x value float);

Changes also need to be reflected in the server.c and graph.php.

Step-3

Compile the server program as ./compile.sh

Step-4

Run the server program as ./run.sh

13

0.7 Literature Survey
0.7.1 MYSQL
The MySQL software delivers a very fast, multi-threaded, multi-user, and
robust SQL (Structured Query Language) database server. The MySQL
software is Dual Licensed. Users can choose to use the MySQL software as
an Open Source product under the terms of the GNU General Public License
(http://www.fsf.org/licenses/) or can purchase a standard commercial
license from Oracle. See http://www.mysql.com/company/legal/licensing/
for more information on our licensing policies.

It is the one of the most scalable (upto 5 Billion entries) and fast database
to meet our requirement.

0.7.2 PHP
PHP (Hypertext Processor) is the language to create dynamic web devel-
opment. PHP is an interpreted language and is executed on server side
(just like CGI or ASP scripts) contrary to scripts executed on client side (a
Javascript or a Java applet executes on your computer). It is usually asso-
ciated with Apache and Mysql database. PHP with GD library to create
images, graphs on the web browser.

14

0.8 Future Scope
0.8.1 Interpolation of Frequencies
We are currently storing the frequency values from multiple sensors in a
20 millisecond time interval. For eg. if a packet that has arrived has the
following data ’surat,2010-07-11 10 12 32.320,47.8’. Its calculated hash is
1278843152320. It is stored in the database table against the hash(which is
a primary key) 1278843152330. Instead of storing the frequency against the
hash 1278843152330(2010-07-11 10 12 32.340) we can interpolate between
the current frequency value and the previous frequency of the sensor and
store it against 1278843152330, for that we need to store last one frequency
value for each sensor.

0.8.2 Daily triggers
Alarms and Triggers need to be genarated for the variations in the frequen-
cies between the sensors over a particular time interval.

0.8.3 Threading
Current program running sequencialy so packet loss may occure when the
number of sensor increased, in order to avodie such problems can be use
threading to achive the parallelism with every packet. We can create a
thread for each incoming packet and perform the constraint check and
database insertion.

15

0.9 References
1. For mysql http://dev.mysql.com/doc/refman/5.1/en/index.html
2. For PHP http://php.net/manual/en/book.image.php
3. For threads https://computing.llnl.gov/tutorials/pthreads

16

Storage, retreival and process of continuous streaming data in a widearea frequency measurement system

More Related Content

What's hot (17)

Similar to Storage, retreival and process of continuous streaming data in a widearea frequency measurement system (20)

More from Nitesh Pandit (6)

Recently uploaded (20)

Storage, retreival and process of continuous streaming data in a widearea frequency measurement system