TechEd 2012 - Сценарии хранения и обработки данных в windows azure

Windows Azure Keynote

Знакомство Создание приложений Решение задач
с Windows Azure Windows Azure с Windows Azure

Windows Azure Internals Сценарии хранения Media Services - обработка медиа
и обработки данных
Windows Azure Q&A в Windows Azure
Миграция имеющихся
Использование Azure ACS приложений в Windows Azure
Безопасность для решения задач
Windows Azure аутентификации Service Bus – связь облака
пользователей с сервисами предприятия

Лабораторные работы
Введение в Windows Azure Virtual Machines Продвинутые техники выявления проблем
Развертывание Active Directory в Windows Azure для решений на базе Windows Azure
Миграция приложения ASP.NET на Windows Azure Развертывание фермы SharePoint в Windows Azure

4.5
4.03
Кол-во объектов в триллионах

4
3.5
3 2.89

2.5
2 1.7
1.5 1.33
0.94
1
0.5
0

Axis Title

Источник: http://blogs.msdn.com/b/windowsazurestorage/archive/2012/07/20/windows-azure-storage-4-trillion-objects-and-counting.aspx

average peak
1000000
Кол-во запросов в секунду

900000
800000
700000
600000
500000
400000
300000
200000
100000
0

Источник: http://blogs.msdn.com/b/windowsazurestorage/archive/2012/07/20/windows-azure-storage-4-trillion-objects-and-counting.aspx

Некоторые клиенты Windows Azure

West US East US

South Central US

Container Blobs
https://<account>.blob.core.windows.net/<container>

Account Table Entities
https://<account>.table.core.windows.net/<table>

Queue Messages
https://<account>.queue.core.windows.net/<queue>

http://myaccount.blob.core.windows.net/mycontainer/myblob
http://myaccount.blob.core.windows.net/$root/myblob
http://myaccount.blob.core.windows.net/myblob

http://myaccount.blob.core.windows.net/mycontainer/path/myblob

string storageConnectionString =
CloudConfigurationManager.GetSetting("StorageConnectionString");

CloudStorageAccount storageAccount =
CloudStorageAccount.Parse(storageConnectionString);

CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

CloudBlobContainer container = blobClient.GetContainerReference("files");
container.CreateIfNotExists();

BlobContainerPermissions publicPermissions = new BlobContainerPermissions
{
PublicAccess = BlobContainerPublicAccessType.Blob
};
container.SetPermissions(publicPermissions);

using (Stream stream = System.IO.File.OpenRead(@"D:blob.txt"))
{
CloudBlockBlob blob = container.GetBlockBlobReference("blob.txt");
blob.UploadFromStream(stream);

CloudBlockBlob blob2 = container.GetBlockBlobReference("demo/blob2.txt");
blob2.UploadFromStream(stream);

CloudBlockBlob blob3 = container.GetBlockBlobReference("demo/blob3.txt");
blob3.UploadFromStream(stream);
}

foreach (IListBlobItem item in container.ListBlobs("demo/"))
{
// Do some action
}

Прогресс обработки периодически сохраняется в сообщении
Увеличение visibility timeout на 5 минут
7:04 AM
7:00
7:09 AM
7:07 Получение сообщения с 5 минутным visibility timeout
Expires @ 7:05AM
Expires @ 7:09AM

Azure Queue
7:09
7:05
7:14

Получение прогресса и
продолжение обработки

CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();

CloudQueue queue = queueClient.GetQueueReference("booking");
queue.CreateIfNotExists();

CloudQueueMessage message = new CloudQueueMessage("Flight: Kharkov - Moskow");
queue.AddMessage(message);

...

CloudQueueMessage message = queue.GetMessage();

queue.DeleteMessage(message);

CloudTableClient tableClient = storageAccount.CreateCloudTableClient();

CloudTable table = tableClient.GetTableReference("people");

TableQuery<CustomerEntity> query = new TableQuery<CustomerEntity>()
.Where(TableQuery.GenerateFilterCondition("PartitionKey",
QueryComparisons.Equal, "Smith"));

foreach (CustomerEntity entity in table.ExecuteQuery(query))
{
Console.WriteLine("{0}, {1}t{2}t{3}", entity.PartitionKey,
entity.RowKey,
entity.Email, entity.PhoneNumber);
}

«In "Avatar," every blade of grass, every cloud in the sky, every vine
in the jungle existed digitally and had to be stored somewhere.»

Jon Landau, Producer

Источник: http://www.microsoft.com/microsoftservices/en/uk/article_Microsoft_Role_In_Avatar.aspx

Более миллиона уникальных пользователей
посетили сайт компании в период проката
фильма в 2010 году.

«We developed an interactive game where users try to navigate a
course maze in a Volvo car in the fastest time and using the shortest
route.»

David Holecek, Digital Strategy Manager at Volvo Car Corporation

Около 400 000 активных пользователей игры

Источник: http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?casestudyid=4000008490

Microsoft Zune Media
Storage and Delivery

Язык запросов HiveQL
CREATE TABLE logs(t1 string, t2 string, t3 string, t4 string, t5 string, t6
string, t7 string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ' ';

LOAD DATA LOCAL INPATH 'sample.log' OVERWRITE INTO TABLE logs;

SELECT t5 AS sev, COUNT(*) AS cnt FROM logs
WHERE t5 LIKE '[%'
GROUP BY t5;

“We take in raw data and make it into something that is
actionable for our consumers, brands, and partners.”

David Mariani Vice President of Engineering at Klout

“Klout chose the best of both worlds by marrying the
Microsoft BI platform with Hadoop and Hive.”

David Mariani Vice President of Engineering at Klout

Источник: http://www.microsoft.com/casestudies/Microsoft-SQL-Server-2012-Enterprise/Klout/Data-Services-Firm-Uses-Microsoft-BI-and-
Hadoop-to-Boost-Insight-into-Big-Data/710000000129

Account Server Database

• Работа с • Информация о • Аналог SQL
порталом базах данных Server базы
• Инструмент • Аутентификация данных
оплаты и безопасность • Содержит
таблицы,
процедуры,
индексы, и т.п.

SQL Azure Network Applications use standard SQL client
Topology Application libraries: ODBC, ADO.Net, PHP, …

Internet
Azure Cloud

TDS (tcp)
Security Boundary Load balancer forwards „sticky‟ sessions to
LB TDS protocol tier

TDS (tcp)

Gateway Gateway Gateway Gateway Gateway Gateway

Gateway: TDS protocol gateway, enforces AUTHN/AUTHZ policy; proxy to CloudDB
TDS (tcp)

L SQL SQL SQL SQL

Scalability and Availability: Fabric, Failover, Replication, and Load balancing

Запрос к базе SQL Azure
1
TDS Gateway
Front-end Node
TDS Session Protocol Parser
2
3
Gateway Logic

Master Cluster

Master Node
4
Master Node Components

7
5 8 6

Backend Node 1 Backend Node 2 Backend Node 3
SQL Instance SQL Instance SQL Instance
SQL DB SQL DB SQL DB

Scalability and and Availability: Fabric,Failover,Replication, and Load balancing
Scalability Availability: Fabric, Failover, Replication, and Load balancing

Federations

SalesDB
CustomerFederationeration
CustomerFederation

Federation Root
Federation Members

Federations member: Range [1000, 2000)

SalesDB AU AU AU

PK=5 PK=25 PK=35

AU AU AU

CustomerFederation
PK=5 PK=25 PK=35
AU AU AU
PK=1005 PK=1025 PK=1035

Federation Root
Federation Members Atomic Units

Central Tables

Federations

SalesDB Federated Tables
CustomerFederation
Reference Tables

User Database and
Federation root Federation members

USE FEDERATION CustomerFederation(customer_id = 5075) …

SalesDB
CustomerFederation

Range Distribution [min,1000, 2000, 3000 ….. 5000, 10000, Max]

USE FEDERATION CustomerFederation(cid = 55)
WITH RESET, FILTERING=OFF
GO

FILTERING=OFF
Работаем с данными всего Federation member
Неограниченный доступ к данным базы: Всѐ равно что соединиться по имени БД
DDL, DML и доступ ко всем Atomic Units внутри Federation Member

Хорошо для…
Management Tasks: Обновление схемы данных
Fan-out Querying – получение данных нескольких
atomic units
member: Range [100,200)

Products
(referece) Customer_id=55
SalesDB
CustomerFederation
CustomerFederation Customers Orders
(federated) (federated)

Microsoft Confidential Microsoft Confidential

USE FEDERATION CustomerFederation(cid = 55)
WITH RESET, FILTERING=ON
GO

FILTERING=ON
Работаем с данными одного Atomic Unit
Полностью доступны данные Reference Tables
Запрещены любые изменения глобального состояния Federation Member
Хорошо для…
Безопасной модели разработки
Предотвращения утечки данных


Products
(referece) Customer_id=55
SalesDB
CustomerFederation
CustomerFederation Customers Orders
(federated) Customer_id=55 (federated)

Microsoft Confidential

ALTER FEDERATION CustomerFederation SPLIT AT (tenant_id=7500)

SalesDB
Orders_federation
CustomerFederation

Range Distribution [min,1000, 2000, 3000 ….. 5000, 10000, Max]

App

PostalCode
SalesDB (reference)

CustomerFederation Customer CustomerAddress
(federated) (federated)

Member: Range [100,150) Member: Range [150,200)

“We can be selling a few hundred tickets an hour for most
of the week, but when a big event goes on sale at 9:00 in
the morning, we get an enormous spike in load against
our application.”

Matthew Davey, Founder, TicketDirect

Источник: http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?casestudyid=4000005890

Windows Azure Roles Client Applications

Service Bus Castellan.Azure
• Box Office sales
• Ticket Printing
• System Administration
• Venue/Event
http:// TicketDirect .* Dynamic Worker Distributed Partitioner WCF Management
(tasks uploaded Cache Worker Worker • Partitioning
as blobs)

Windows Azure Storage On Premises
Blobs to -- -- Tables to
-- Queues for
SQL Server
store web
and worker --- record
--- server &
communication
between clients
role
resources
---- partition
information
and roles

Castellan Venue

SQL Azure
Castellan
Castellan Castellan
Castellan Castellan
Castellan
Castellan Venue DB 1
Venue
Venue Venue DB 2
Venue
Venue Venue DB N
Venue
Venue ...
Partition(s)
DB‟s Partition(s)
DB‟s Partition(s)
DB‟s Castellan.old (VB6)

One application DB, many venue DB‟s – each partitioned in to many parts (40+)

Monitor/Comm
[SLA & Usage]
Monitor/Comm
[Performance]
[Traces &[SLA & Usage]
Logs]
[Performance]
[Traces & Logs]

RS Engine
RS Engine

Источник: http://weblogs.asp.net/scottgu/archive/2012/08/21/windows-azure-media-services-and-the-london-2012-olympics.aspx

http://sigops.org/sosp/sosp11/current/2011-
Cascais/printable/11-calder.pdf

http://research.google.com/archive/mapreduce-osdi04.pdf

http://social.technet.microsoft.com/wiki/contents/articles/1695.i
nside-windows-azure-sql-database.aspx

http://www.microsoft.com/casestudies

denisreznik@live.ru
http://reznik.uneta.com.ua

TechEd 2012 - Сценарии хранения и обработки данных в windows azure

More Related Content

What's hot (8)

Viewers also liked (8)

Similar to TechEd 2012 - Сценарии хранения и обработки данных в windows azure (20)

More from Денис Резник (12)

TechEd 2012 - Сценарии хранения и обработки данных в windows azure

Editor's Notes