Real World Azure - Dev

Editor's Notes

• #52: Here’s the datacenter in the cloudA collection of commodity hardwareA collection of storage servers; triple replicationLoad BalancersFabric Controller: the “Brains” behind it all. Web Portal: where to deploy and manage applicationsService – any app you want to run is the service.It’s about running your service in the Microsoft datacenter. Windows Azure is not a SKU that you would install onsite.
• #53: = Service Deployment (So easy, even a CEO can do it) =Service, the application you want to runModel, service configuration; tells what the service looks like, how many you want to run, etc.Today, you must deploy your service through the portal. In the future, there will be an API available that will you to deploy your service through command-line, TFS build procedures, and other types of automation In this scenario, we are deploying our service through the portal. We upload the two files (the service package and model configuration). The Fabric Controller reads the model configuration which describes how to deploy our service. In this case, we are deploying our service to 3 machines. The Fabric Controller determines which 3 machines to deploy to, copies the service package to the 3 machines, and fires up the services. [Transition] The Fabric Controller then configures the DNS so you have an endpoint exposed for your services for the outside world to communicate with your services. From there, it configures the load balancers and routers. That’s it. It’s completed automated.
• #57: Managed partner pipeline review -opps in Seibel, partner, PAM; get together and collaborate on opps - more social collaboration, ability to comment, ability to bring people inDidn’t do much in SQL Azure as it wasn’t available at the timeAccomplished goals in 5 mos.
• #59: Web role - support for multi-tennancy (host multiple customers or segments on set of infrastructure) - web service for updating the opportunity information.NET Service Bus was used to integrate on the backend with SeibelMoved worker role inside firewall as it made more sense (on premise)Heavy use of Tables and BlobsMost Queue work is done with the .NET Service Bus under-the-covers; not a whole lot of work writing directly to QueuesDuring development, SDS did a reset and became SQL Azure; use Azure storage until SQL Azure becomes available (one of the best decisions they made)
• #60: Community wants to control Personalization, Content, Membership
• #61: TODO: Convert to Whiteboard template
• #62: TODO: Convert to Whiteboard template
• #67: We will bend over backwards to help you out.
• #80: BlobsBlobs are stored in containers. There are 0 or more blobs per container and 0 or more containers per account. (since you can have 0 containers, but then you would not have any blobs either)Typically url in the cloud is http://accountname.blob.core.windows.net/container/blobpathBlob paths can contain the / character, so you can give the illusion of multiple folders, but there is only 1 level of containers.Blob capacity at CTP is 50gb.There is an 8k dictionary that can be associated with blobs for metadata.Blobs can be private or public:Private requires a key to read and writePublic requires a key to write, but NO KEY to read.Use blobs where you would use the file system in the past.
• #87: Tables are simply collections of Entities.Entites must have a PartitionKey and RowKey – can also contain up to 256 other properties.Entities within a table need not be the same shape! E.g.:Entity 1: PartitionKey, RowKey, firstnameEntity 2: PartitionKey, RowKey, firstname, lastnameEntity 3: PartitionKey, Rowkey, orderId, orderData, zipCodePartitions are used to spread data across multiple servers. This happens automatically based on the partition key you provide. Table “heat” is also monitored and data may be moved to different storage endpoints based upon usage.Queries should be targeted at a partition, since there are no indexes to speed up performance. Indexes may be added at a later date.Its important to convey that whilst you could copy tables in from a local data source (e.g. sql) it would not perform well in the cloud, data access needs to be re-thought at this level. Those wanting a more traditional SQL like experience should investigate SDS.
• #88: Hosted in Microsoft data centersFor massive scalable applicationsWe’re talking millions of rows, terabytes of dataStorage is one of the pieces of the Azure PlatformIt is NOT relationalWe can take hot data, the data that’s being requested more often, and scale those out to more serversDesigned for online, high-scale, high-throughput applicationsFamiliar if you’re used to .NET, LINQ and ADO.NET Data ServicesShared-key authentication between client and server
• #89: 05:17Flexible schema, tables don’t have to have the same columns (properties) defined up front; this is not an ‘Add Column’ or ‘Update Schema’Optimistic concurrency, protects you from someone else from modifying your data while you’re working on it
• #90: 07:17It’s an “Entity Store”, you can store entities, can retrieve entities, do simple querying on these entitiesPartitioned SQL Server: - A-M on this server - N-Z on this serverTop 5 customers that ordered the most, you have to poll 26 servers and aggregate the dataThat’s sort of what we have with Azure Table storage. We went with a highly partitioned approach upfront to gain scale and gain availability. We’ve had to sacrifice some of the complex queries such as joins to support this. It’s just a different way of having to deal with your data.
• #91: 09:18Entities (Rows)Properties (Columns)An Entity has 2 “Key” properties: -PartitionKey: How to partition data (ie. First-letter of last name, zip code) - every entity will be on the SAME physical server, this sets us up for scale - a good strategy is to keep your partitions small - RowKey: a unique identifier within a partition
• #92: 11:53Getting the all of dunnry’s post it fast because we’re selecting the entities by a partition keyGetting all of the posts after a certain is slow because we may have to traverse across multiple servers because we’re selecting entities that span partition keysA query without the partition key is really a scan
• #93: 14:58Keep partitions small, this increases scalability; this allows us to replicate data when its hot and spread data across multiple servers
• #94: 16:34Whitepaper: Windows Azure Table - Programming Azure Tables, http://go.microsoft.com/fwlink/?LinkId=153401Locality, in working with your entities, you want them to be on the same server or close together; you gain locality by having large partitionsBut… for scalability, we said that you want smaller partitions so we can help manage the distribute the data and therefore the loadScalability is defined on who well you can manage your partition sizesConsiderations: - the only transaction we have our within the same table with the same partition key; take your transactional requirements into consideration when defining you want to partition your data - query efficiency; think through the top queries you want to execute - scalability: managing the size of your partitions - flexible partitioning : A-M, N-Z; change to A-M, N-R, S-Z - partitioning scheme by months (holiday sales, by month but in december by day)
• #95: 21:51A simple formula for choosing keys (more info in the whitepaper)Potential keys… what are you going to query onOrder by importanceMay need additional information to make it unique (add a GUID)
• #97: 23:35If you do a query on a non-key property, that’s a scan (very inefficient)Improve performance by scoping - by partition key, and range of row keys - scope by tables (came up in some customer discussions); users have arbitrary queries, no good set of keys (only querying from top 1000 items), make a table of just those 1000 items - data will be replicated but you can determine which partition to query from based on query typeIs it OK to duplicate data? Efficiency over consistency and slowing down updates - Certain types of applications
• #99: 30:29One of the more important things to understand*NOT* a relational databaseBad – indexes, foreign keys, referential integrity don’t existGood – it scales!Have to write lower level code for Azure table access vs. SQL ServerStorage in triplicateOur experience is Azure storage has been remarkably stable during the CTP; amazing as it’s a CTP and its brand new codeIt is CASE SENSITIVELot of learning in the community
• #100: 46:44During development, Azure Storage Client library was a sample; they couldn’t use a sample; since then its shipping as part of the SDK (production quality)Leverage client library where you canAzure storage is available via RESTThe hardest part is getting the signature right; its very picky (tutorial available on bartr’s blog)Client library uses ADO.NET Data ServicesSometimes its faster to just pull back a simple string then spin up a bunch of ADO.NET Data Services objectsREST is easier for dealing with versioning
• #101: 49:23REST requires a lot of parsingLINQ and XML make it easy, but sometimes parsing is easier (and maybe the only thing needed)CLOSE your HTTP connections; wrap in ‘using’ block or physically close themStorage names are picky
• #102: 50:32Drop table/Create table has about 30-40 sec lag timeOften better to get more data then you need (goes against T-SQL best practices)
• #103: 52:34List<> is your friend; works well, serializes well, binds wellThe team used this to work with the azure table data
• #118: 53:20(watch Dave Campbell session on SQL Azure)
• #119: 54:32SQL Server is very familiar, Azure tables are new and unknownSQL Server will not scale as well for V1; make the right choice for your needs
• #120: 56:09SQL Azure costs more, but delivers moreDBA’s don’t have the tools available yet to monitor SQL Azure… system catalog is different in the cloud then on premise… this is forthcoming
• #122: Queues are simple:Messages are placed in queues. Max size is 8k (and it’s a string)Message can be read from the queue, at which point it is hidden.Once whatever read the message from the queue is finished processing the message, it should then remove the message from the queue. If not the message is returned to the queue after a specific user defined time limit. This can be used to handle code failures etc.
• #123: Use queues as a way of communicating w/ the backend worker rolesWRs call getmessage and pass timeoutTimeout value is importantExpiration time is important; message is marked in the queue as invisible; for duration of timeout it’s invisibleWhen we’re done processing, we call a message to remove the message through a deleteTh reason we do this is imagine we have a second worker role; if something goes wrong, once the timeout expires, the message becomes visible, and the next person to do a get message will get the message
• #176: C:Program Files (x86)Microsoft .NET Services SDK (July 2009 CTP)SamplesServiceBusGettingStartedEcho
• #183: If you are or know someone starting a software business, get started building applications for Windows Azure * by joining BizSpark!*inclusion of Windows Azure in the BizSpark program is still being finalized and there is no public information on this topic we can share