cloudcomputing.pptx

What is Cloud Computing?
• Cloud computing is a model for enabling convenient,
on-demand network access to a shared pool of
configurable computing resources (e.g., networks,
servers, storage, applications, and services)
[Mell_2009], [Berkely_2009].
• It can be rapidly provisioned and released with minimal
management effort.
• It provides high level abstraction of computation and
storage model.
• It has some essential characteristics, service models,
and deployment models.
2

Essential Characteristics
• On-Demand Self Service:
• A consumer can unilaterally provision computing capabilities,
automatically without requiring human interaction with each
service’s provider.
• Heterogeneous Access:
• Capabilities are available over the network and accessed
through standard mechanisms that promote use by
heterogeneous thin or thick client platforms.
3

• Resource Pooling:
• The provider’s computing resources are pooled to serve
multiple consumers using a multi-tenant model.
• Different physical and virtual resources dynamically assigned
and reassigned according to consumer demand.
• Measured Service:
• Cloud systems automatically control and optimize resources
used by leveraging a metering capability at some level of
abstraction appropriate to the type of service.
• It will provide analyzable and predictable computing
platform.
4
Essential Characteristics (cont.)

Service Models
• Cloud Software as a Service (SaaS):
• The capability provided to the consumer is to use the
provider’s applications running on a cloud infrastructure.
• The applications are accessible from various client devices
such as a web browser (e.g., web-based email).
• The consumer does not manage or control the underlying
cloud infrastructure including network, servers, operating
systems, storage,…
• Examples: Caspio, Google Apps, Salesforce, Nivio,
Learn.com.
5

• Cloud Platform as a Service (PaaS):
• The capability provided to the consumer is to deploy onto
the cloud infrastructure consumer-created or acquired
applications created using programming languages and tools
supported by the provider.
cloud infrastructure.
• Consumer has control over the deployed applications and
possibly application hosting environment configurations.
• Examples: Windows Azure, Google App.
6
Service Models (cont.)

• Cloud Infrastructure as a Service (IaaS):
• The capability provided to the consumer is to provision
processing, storage, networks, and other fundamental
computing resources.
• The consumer is able to deploy and run arbitrary software,
which can include operating systems and applications.
cloud infrastructure but has control over operating systems,
storage, deployed applications, and possibly limited control
of select networking components (e.g., host firewalls).
• Examples: Amazon EC2, GoGrid, iland, Rackspace Cloud
Servers, ReliaCloud.
7

Service Model at a glance: Picture From http://en.wikipedia.org/wiki/File:Cloud_Computing_Stack.svg
8


Private Cloud:

The cloud is operated solely for an organization. It may be
managed by the organization or a third party and may exist on
premise or off premise.

Community Cloud:

The cloud infrastructure is shared by several organizations and
supports a specific community that has shared concerns.

It may be managed by the organizations or a third party and
may exist on premise or off premise


Public Cloud:

The cloud infrastructure is made available to the general public
or a large industry group and it is owned by an organization
selling cloud services.

Hybrid cloud:

The cloud infrastructure is a composition of two or more
clouds (private, community, or public).

Advantages of Cloud Computing
 Cloud computing do not need high quality equipment
for user, and it is very easy to use.
 Provides dependable and secure data storage center.
 Reduce run time and response time.
 Cloud is a large resource pool that you can buy on-
demand service.
 Scale of cloud can extend dynamically providing nearly
infinite possibility for users to use internet.

Infrastructure as a Service
(IaaS)
Amazon EC2

What is Infrastructure as a Service ?
• A category of cloud services which provides capability to
provision processing, storage, intra-cloud network connectivity
services, and other fundamental computing resources of the
cloud infrastructure.
Source- [ITU –Cloud Focus Group]
Diagram Source: Wikipedia

Highlights of IaaS
• On demand computing resources
• Eliminate the need of far ahead planning
• No up-front commitment
• Start small and grow as required
• No contract, Only credit card!
• Pay for what you use
• No maintenance
• Measured service
• Scalability
• Reliability

What is EC2 ?
 Amazon Elastic Compute Cloud (EC2) is a web service
that provides resizeable computing capacity that one
uses to build and host different software systems.
 Designed to make web-scale computing easier for
developers.
 A user can create, launch, and terminate server
instances as needed, paying by the hour for active
servers, hence the term "elastic".
 Provides scalable, pay as-you-go compute capacity
 Elastic - scales in both direction

EC2 Concepts
• AMI & Instance
• Region & Zones
• Storage
• Networking and Security
• Monitoring
• Auto Scaling
• Load Balancer

Amazon Machine Images (AMI)
 Is an immutable representation of a set of disks that contain an operating
system, user applications and/or data.
 From an AMI, one can launch multiple instances, which are running copies
of the AMI.

AMI and Instance
• Amazon Machine Image (AMI) is a template for
software configuration (Operating System,
Application Server, and Applications)
• Instance is a AMI running on virtual servers in the
cloud
• Each instance type offers different compute and
memory facilities
Diagram Source: http://docs.aws.amazon.com

Region and Zones
• Amazon have data centers in different region across
the globe
• An instance can be launched in different regions
depending on the need.
• Closer to specific customer
• To meet legal or other requirements
• Each region has set of zones
• Zones are isolated from failure in other zones
• Inexpensive, low latency connectivity between zones in same
region

Storage
• Amazon EC2 provides three type of storage option
• Amazon EBS
• Amazon S3
• Instance Storage
Diagram Source: http://docs.aws.amazon.com

Elastic Block Store(EBS) volume
 An EBS volume is a read/write disk that can be created by an AMI
and mounted by an instance.
 Volumes are suited for applications that require a database, a file
system, or access to raw block-level storage.

Amazon S3
 S3 = Simple storage Service
 A SOA – Service Oriented Architecture which provides
online storage using web services.
 Allows read, write and delete permissions on objects.
 Uses REST and SOAP protocols for messaging.

Amazon SimpleDB
 Amazon SimpleDB is a highly available, flexible, and
scalable non-relational data store that offloads the work
of database administration.
 Creates and manages multiple geographically
distributed replicas of your data automatically to enable
high availability and data durability.
 The service charges you only for the resources actually
consumed in storing your data and serving your
requests.

Networking and Security
• Instances can be launched on one of the two platforms
• EC2-Classic
• EC2-VPC
 Each instance launched is assigned two addresses a private
address and a public IP address.
 A replacement instance has a different public IP address.
• Instance IP address is dynamic.
• new IP address is assigned every time instance is launched
 Amazon EC2 offers Elastic IP addresses (static IP addresses) for
dynamic cloud computing.
• Remap the Elastic IP to new instance to mask failure
• Separate pool for EC2-Classic and VPC
• Security Groups to access control to instance

Monitoring, Auto Scaling, and Load
Balancing
• Monitor statistics of instances and EBS
• CloudWatch
• Automatically scales amazon EC2 capacity up and
down based on rules
• Add and remove compute resource based on demand
• Suitable for businesses experiencing variability in usage
• Distribute incoming traffic across multiple instances
• Elastic Load Balancing

How to access EC2
• AWS Console
• http://console.aws.amazon.com
• Command Line Tools
• Programmatic Interface
• EC2 APIs
• AWS SDK

References
 Mobile cloud computing: Big Picture by M. Reza Rahimi
 http://aws.amazon.com/ec2,
http://docs.aws.amazon.com
 Amazon Elastic Compute Cloud – User Guide, API
Version 2011-02-28.
• Above the Clouds: A Berkeley View of Cloud Computing
- Michael Armbrust et.al 2009
• International telecommunication union – Focus Group
Cloud Technical Report

Hadoop, a distributed
framework for Big Data

Introduction
1. Introduction: Hadoop’s history and advantages
2. Architecture in detail
3. Hadoop in industry

What is Hadoop?
• Apache top level project, open-source
implementation of frameworks for reliable, scalable,
distributed computing and data storage.
• It is a flexible and highly-available architecture for
large scale computation and data processing on a
network of commodity hardware.
• Designed to answer the question: “How to
process big data with reasonable cost and
time?”

Search engines in 1990s
1996
1996
1997
1996

Google search engines
1998
2013

Hadoop’s Developers
2005: Doug Cutting and Michael J. Cafarella
developed Hadoop to support distribution for
the Nutch search engine project.
The project was funded by Yahoo.
2006: Yahoo gave the project to Apache
Software Foundation.

Some Hadoop Milestones
• 2008 - Hadoop Wins Terabyte Sort Benchmark (sorted 1 terabyte
of data in 209 seconds, compared to previous record of 297 seconds)
• 2009 - Avro and Chukwa became new members of Hadoop
Framework family
• 2010 - Hadoop's Hbase, Hive and Pig subprojects completed, adding
more computational power to Hadoop framework
• 2011 - ZooKeeper Completed
• 2013 - Hadoop 1.1.2 and Hadoop 2.0.3 alpha.
- Ambari, Cassandra, Mahout have been added

What is Hadoop?
• An open-source software framework that supports data-intensive
distributed applications, licensed under the Apache v2 license.
• Abstract and facilitate the storage and processing of large and/or rapidly
growing data sets
• Structured and non-structured data
• Simple programming models
• High scalability and availability
• Use commodity (cheap!) hardware with little redundancy
• Fault-tolerance
• Move computation rather than data

Hadoop MapReduce Engine
A MapReduce Process (org.apache.hadoop.mapred)
• JobClient
•Submit job
•JobTracker
•Manage and schedule job, split job into tasks;
• Splits up data into smaller tasks(“Map”) and sends it to the
TaskTracker process in each node
•TaskTracker
• Start and monitor the task execution;
•reports back to the JobTracker node and reports on job
progress, sends data (“Reduce”) or requests new jobs
•Child
•The process that really executes the task

Hadoop’s Architecture: MapReduce Engine

Hadoop’s MapReduce Architecture
• Distributed, with some centralization
• Main nodes of cluster are where most of the computational
power and storage of the system lies
• Main nodes run TaskTracker to accept and reply to MapReduce
tasks, Main Nodes run DataNode to store needed blocks closely
as possible
• Central control node runs NameNode to keep track of HDFS
directories & files, and JobTracker to dispatch compute tasks to
TaskTracker
• Written in Java, also supports Python and Ruby

Hadoop Distributed FileSystem
• Tailored to needs of MapReduce
• Targeted towards many reads of filestreams
• Writes are more costly
• Open Data Format
• Flexible Schema
• Queryable Database
• Fault Tolerance
• High degree of data replication (3x by default)
• No need for RAID on normal nodes
• Large blocksize (64MB)
• Location awareness of DataNodes in network

HDFS
NameNode:
• Stores metadata for the files, like the
directory structure of a typical FS.
• The server holding the NameNode
instance is quite crucial, as there is
only one.
• Transaction log for file deletes/adds,
etc. Does not use transactions for
whole blocks or file-streams, only
metadata.
• Handles creation of more replica
blocks when necessary after a
DataNode failure
DataNode:
• Stores the actual data
in HDFS
• Can run on any
underlying filesystem
(ext3/4, NTFS, etc)
• Notifies NameNode of
what blocks it has
• NameNode replicates
blocks 2x in local
rack, 1x elsewhere

HDFS Replication
Replication Strategy:
• One replica on local node
• Second replica on a remote
rack
• Third replica on same remote
rack
• Additional replicas are
randomly placed
•Clients read from nearest replica
Use Checksums to validate data –
CRC32
• File Creation
• Client computes checksum per 512 byte
• DataNode stores the checksum
• File Access
• Client retrieves the data anD checksum
from DataNode
• If validation fails, client tries other
replicas
•Client retrieves a list of DataNodes on which to place replicas of a block
• Client writes block to the first DataNode
•The first DataNode forwards the data to the next DataNode in the
Pipeline
•When all replicas are written, the client moves on to write the next

Hadoop Usage
• Hadoop is in use at most
organizations that handle big data:
o Yahoo!
o Yahoo!’s Search Webmap
runs on 10,000 core Linux
cluster and powers Yahoo!
Web search
o Facebook
o FB’s Hadoop cluster hosts
100+ PB of data (July,
2012) & growing at ½
PB/day (Nov, 2012)
o Amazon
o Netflix
• Key Applications
• Advertisement (Mining
user behavior to
generate
recommendations)
• Searches (group related
documents)
• Security (search for
uncommon patterns)

Hadoop Usage
• Non-realtime large dataset computing:
o NY Times was dynamically generating PDFs
of articles from 1851-1922
o Wanted to pre-generate & statically serve
articles to improve performance
o Using Hadoop + MapReduce running on EC2
/ S3, converted 4TB of TIFFs into 11 million
PDF articles in 24 hrs

Hadoop Usage: Facebook Messages
• Design requirements:
o Integrate display of email,
SMS and chat messages
between pairs and groups of
users
o Strong control over who users
receive messages from
o Suited for production use
between 500 million people
immediately after launch
o Stringent latency & uptime
requirements

• System requirements
o High write throughput
o Cheap, elastic storage
o Low latency
o High consistency
(within a single data
center good enough)
o Disk-efficient
sequential and random
read performance

• Classic alternatives
o These requirements typically met using large
MySQL cluster & caching tiers using Memcache
o Content on HDFS could be loaded into MySQL or
Memcached if needed by web tier
• Problems with previous solutions
o MySQL has low random write throughput… BIG
problem for messaging!
o Difficult to scale MySQL clusters rapidly while
maintaining performance
o MySQL clusters have high management overhead,
require more expensive hardware

• Facebook’s solution
o Hadoop + HBase as foundations
o Improve & adapt HDFS and HBase to scale to FB’s
workload and operational considerations
 Major concern was availability: NameNode is
SPOF & failover times are at least 20 minutes
 Proprietary “AvatarNode”: eliminates SPOF,
makes HDFS safe to deploy even with 24/7
uptime requirement
 Performance improvements for realtime
workload: RPC timeout. Rather fail fast and try
a different DataNode

58
Cloud Computing for Mobile
and Pervasive Applications
Mobile Music: 52.5%
Mobile Video:25.2%
Mobile Gaming: 19.3%
Sensory Based Applications
Augmented Reality
Mobile Social
Networks and
Crowdsourcing
Multimedia and
Data Streaming
Location Based
Services (LBS)
Due to limited resources on mobile devices,
we need outside resources to empower mobile apps.

59
Mobile Cloud Computing
Ecosystem
Wired and Wireless
Network Providers
Local and Private
Cloud Providers
Devices, Users
and Apps
Public Cloud Providers
Content and Service
Providers

60
Tier 2: Local
Cloud
(+) Low Delay, Low Power,
(-) Not Scalable and Elastic
Tier 1: Public Cloud
(+) Scalable and Elastic
(-) Price, Delay
Wi-Fi Access
Point
3G Access
Point
RTT:
~290ms
RTT:
~80ms
IBM: by 2017 61% of
enterprise is likely to be on
a tiered cloud
2-Tier Cloud Architecture

61
Mobile Cloud Computing
Ecosystem
Wired and Wireless
Network Providers
Local and Private
Cloud Providers
Devices, Users
and Apps
Public Cloud Providers
Content and Service
Providers

How can we Optimally and Fairly assign services to mobile
users using a 2-tier cloud architecture (knowing user mobility
pattern) considering power consumed on mobile device, delay
users experience and price as the main criteria for optimization.
62
Modeling Mobile
Apps
Mobility-Aware
Service Allocation
Algorithms
Scalability
Middleware
Architecture and
System Design

Modeling Mobile Applications
as Workflows
• .Model apps as consisting of a series of logical
steps known as a Service with different
composition patterns:
63
S1
S2
S4
S3
S5
S7
S8
S6
0
1
Par1
Par2
3
Start End
S1 S2 S3
S1
S2
S4
S3
S1
S1
S2
S4
S3
SEQ LOOP
AND: CONCURRENT FUNCTIONS XOR: CONDITIONAL FUNCTIONS
k
1
1
P1
P2
𝑷𝟏 + 𝑷𝟐 = 𝟏, 𝑷𝟏, 𝑷𝟐 ∈ {𝟎, 𝟏}

64
t1 t2 t4
t3 tN
l2
l1
l3
ln
W1
Wk+1
Wk
Wj+1
Wj
Location-Time Workflow
• It could be formally defined as:
𝑾(𝒖𝒌)𝚻
𝑳
≝ (𝒘 𝒖𝒌 𝒕𝒎𝟏
𝒍𝒏𝟏 , 𝒘 𝒖𝒌 𝒕𝒎𝟐
𝒍𝒏𝟐 ,….,𝒘 𝒖𝒌 𝒕𝒎𝒌
𝒍𝒏𝒌 )
Modeling Mobile Applications as
Workflows

Quality of Service (QoS)
𝒒(𝒖𝒌
𝒔𝒊,𝒍𝒋)𝒑𝒐𝒘𝒆𝒓 power consumed on 𝒖𝒌 cellphone when he is in l𝐨𝐜𝐚𝐭𝐢𝐨𝐧 𝒍𝒋 using 𝒔𝒊.
65
• The QoS could be defined in two different
Levels:
• Atomic service level
• Composite service level or workflow level.
• Atomic service level could be defined as (for
power as an example):
• The workflow QoS is based on different patterns.
QoS SEQ AND (PAR) XOR (IF-ELSE-THEN) LOOP
𝑾𝒑𝒐𝒘𝒆𝒓
𝒊=𝟏
𝒊=𝒏
𝒒(𝒖𝒌
𝒔𝒊,𝒍𝒋)𝒑𝒐𝒘𝒆𝒓
𝒊=𝟏
𝒊=𝒏
𝒒(𝒖𝒌
𝒔𝒊,𝒍𝒋)𝒑𝒐𝒘𝒆𝒓
𝒎𝒂𝒙
𝒊
𝒒(𝒖𝒌
𝒔𝒊,𝒍𝒋)𝒑𝒐𝒘𝒆𝒓 𝒒(𝒖𝒌
𝒔𝒊,𝒍𝒋)𝒑𝒐𝒘𝒆𝒓 × 𝒌

66
• different QoSes have different dimensions (Price->$, power-
>joule, delay->s)
• We need a normalization process to make them comparable.
Normalization
𝑾(𝒖𝒌)𝒑𝒐𝒘𝒆𝒓 ≝
𝑾(𝒖𝒌)𝒑𝒐𝒘𝒆𝒓
𝒎𝒂𝒙
− 𝑾(𝒖𝒌)𝒑𝒐𝒘𝒆𝒓
𝒎𝒂𝒙
− 𝑾(𝒖𝒌)𝒑𝒐𝒘𝒆𝒓
𝒎𝒊𝒏
, 𝑾(𝒖𝒌)𝒑𝒐𝒘𝒆𝒓
𝒎𝒂𝒙
≠
𝒎𝒊𝒏
𝟏, 𝒆𝒍𝒔𝒆
𝑾(𝒖𝒌)𝜯
𝑳
𝒑𝒐𝒘𝒆𝒓
≝
𝑾(𝒖𝒌)𝜯
𝑳
𝒎𝒂𝒙
− 𝑾(𝒖𝒌)𝜯
𝑳
𝑾(𝒖𝒌)𝜯
𝑳
𝒎𝒂𝒙
− 𝑾(𝒖𝒌)𝜯
𝑳
𝒎𝒊𝒏 , 𝑾(𝒖𝒌)𝜯
𝑳
𝒎𝒂𝒙
≠
𝑾(𝒖𝒌)𝜯
𝑳
𝒎𝒊𝒏
𝟏, 𝒆𝒍𝒔𝒆
The normalized
power, price
and delay is the
real number in
interval [0,1].
The higher the
normalized
QoS the better
the execution
plan is.
M. Reza. Rahimi, Nalini Venkatasubramanian, Sharad Mehrotra and Athanasios Vasilakos, "MAPCloud: Mobile
Applications on an Elastic and Scalable 2-Tier Cloud Architecture", In the 5th IEEE/ACM International Conference
on Utility and Cloud Computing (UCC 2012), USA, Nov 2012.

𝒎𝒂𝒙
𝟏
|𝑼|
𝒖𝒌
𝒎𝒊𝒏 𝑾(𝒖𝒌)𝚻
𝑳
, 𝑾(𝒖𝒌)𝚻
𝑳
𝒑𝒓𝒊𝒄𝒆
, 𝑾(𝒖𝒌)𝚻
𝑳
𝒅𝒆𝒍𝒂𝒚
𝑺𝒖𝒃𝒋𝒆𝒄𝒕 𝒕𝒐:
𝟏
|𝑼|
𝑾(𝒖𝒌)𝚻
𝑳
≤ 𝑩𝒑𝒐𝒘𝒆𝒓,
𝟏
|𝑼|
𝑾(𝒖𝒌)𝚻
𝑳
𝒑𝒓𝒊𝒄𝒆
≤ 𝑩𝒑𝒓𝒊𝒄𝒆,
𝟏
|𝑼|
𝑾(𝒖𝒌)𝚻
𝑳
𝒅𝒆𝒍𝒂𝒚
≤ 𝑩𝒅𝒆𝒍𝒂𝒚,
𝜿 ≤ 𝑪𝒂𝒑(𝑳𝒐𝒄𝒂𝒍_𝑪𝒍𝒐𝒖𝒅𝒔)
𝜿 ≜ 𝑵𝒖𝒎𝒃𝒆𝒓 𝒐𝒇 𝒎𝒐𝒃𝒊𝒍𝒆 𝑼𝒔𝒆𝒓𝒔 𝒖𝒔𝒊𝒏𝒈
𝒔𝒆𝒓𝒗𝒊𝒄𝒆𝒔 𝒐𝒏 𝒍𝒐𝒄𝒂𝒍 𝒄𝒍𝒐𝒖𝒅
∀ 𝒖𝒌 ∈ 𝒖𝟏, … , 𝒖|𝑼|
• In this optimization problem our goal is to maximize the
minimum saving of power, price and delay of the mobile
applications. 67
𝑭𝒂𝒊𝒓𝒏𝒆𝒔𝒔 𝑈𝑡𝑖𝑙𝑖𝑡𝑦
Optimal Service Allocation for
Single Mobile User

68
Service Allocation Algorithms for
Single Mobile User and Mobile Group-Ware
Applications
Brute-Force Search
(BFS)
Simulated
Annealing Based
Genetic Based
Greedy Based
Random Service
Allocation (RSA)
• MuSIC: Mobility Aware Service AllocatIon on Cloud.
• based-on a simulated annealing approach.

69
QoS-Aware
Service DB
Mobile User
Log DB
Optimal Service Scheduler
Cloud Service Registry
Mobile Client
MAPCloud
Web
Service
Interface
MAPCloud Middleware
MAPCloud
Runtime
Local and
Public
Cloud Pool
MAPCloud LTW
Engine
MAPCloud Web Service Interface
MAPCloud Middleware
Architecture

• M. Satyanarayanan, P. Bahl, R. Cáceres, N. Davies " The Case for VM-
Based Cloudlets in Mobile Computing",PerCom 2009.
• M. Reza Rahimi, Jian Ren, Chi Harold Liu, Athanasios V. Vasilakos,
and Nalini Venkatasubramanian, "Mobile Cloud Computing: A
Survey, State of Art and Future Directions", in ACM/Springer Mobile
Application and Networks (MONET), Special Issue on Mobile Cloud
Computing, Nov. 2013.
• Reza Rahimi, Nalini Venkatasubramanian, Athanasios Vasilakos,
"MuSIC: On Mobility-Aware Optimal Service Allocation in Mobile
Cloud Computing", In the IEEE 6th International Conference on Cloud
Computing, (Cloud 2013), Silicon Valley, CA, USA, July 2013
70

cloudcomputing.pptx

More Related Content

Similar to cloudcomputing.pptx (20)

More from Siva453615 (20)

Recently uploaded (20)

cloudcomputing.pptx