SlideShare a Scribd company logo

Gray_Compass99.ppt

Download as PPT, PDF
P
PokinMorakrant

Jim Gray gave a presentation on Microsoft SQL Server and database research. He discussed SQL Server's goals of being easy to use and scalable. He outlined enhancements to SQL Server 7 including improved replication, query processing, and data warehousing capabilities. Gray also discussed challenges around managing the growing volume of data being created and the importance of data analysis. He concluded by previewing new capabilities for future versions of SQL Server like support for XML and object-relational features.

Technology
1 of 68
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 1 Microsoft SQL Server, Scalability, & Database Research Jim Gray Researcher Microsoft Corporation
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 2 Outline Summary of what you heard. (10 min) The database scene in general. (10 min) Scaleability: Farms, Clones,Parts & Packs (15 min) Microsoft DB research focus. (15 min) • TerraServer (design and ops). • RAGS. • Data Mining Q&A (10 min)
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 3 Organizations Are Going Online Building a digital nervous system. Inexpensive hardware means huge databases are possible. But, we are drowning in data. Databases help organize information. Microsoft’s goal: • Information at your fingertips. • Make it easy to capture, manage, and analyze information.
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 4 Microsoft SQL Server 7 Goals Easy  Dynamic self management  Multi-site management  Operation Scripting  Job scheduling and execution  Alert/response management  Scriptable Install+upgrade  DBA profiling/tuning tools  Unicode  English Language Query  Integrated with NT Security  Integrated with NT files Scalability Data Warehousing
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 11 Scalability  Win9x/NTW version  Dynamic row-level locking  Improved query optimizer  Intra-query parallelism  VLDB improvements  Replication improvements  Distributed query  High Availability Clusters Easy Scalability Data Warehousing
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 12 Scale Down to Windows 95-98 Full function (same as NTW) Integration with Access 97 MSDE in Office2000 and MSDN WinCE version demonstrated
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 13 Replication Transactional and Merge Remote update ODBC and OLE DB subscribers Wizards Performance 2PC, RPC Subscriber DB2 CICS Subscriber Subscriber VSAM OS 390 DB2 Publisher Updating Subscriber (immediate updates) Distributor Subscriber
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 14 Query Processor Enhancements Parallelism Improved scan, fetch, & sort Smart hash & merge join Large joins & grouping Better query optimization Multi-index operations Automatic statistics maintenance Distributed Query Heterogeneous Query Focus on Complex Queries
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 15 •# of emp. per group •total inc. per group Local Agg. 4 x 50 rows + + + + Disks 50,000 rows Global Agg. Result 50 rows + Parallel Query SMP & Disk Parallelism Plus Distributed Plus Hash Join (fanciest on the planet) Plus Optimized Partitioned views
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 16 Distributed Heterogeneous Queries Data Fusion / Integration Join spread sheets, databases, directories, Text DBs etc. Any source that exposes OLE DB interfaces SQL Server as gateway, even on the desktop Database (DB2, VSAM, Oracle, …) Spreadsheet Photos Mail Maps Documents and the Web Directory Service SQL 7.0 Query Processor
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 17 Utilities The Key to LARGE Databases  Backup • Fuzzy • Parallel • Incremental • Restartable Recovery • Fast • File granularity Reorganize • shrinks file • reclusters file Auto-Repair Index creation ~2x faster than 6.5 DBCC • not required, • a good practice • 5x - 100x faster 0 10 20 30 40 50 60 1 2 3 4 SQL Server 6.5 SQL Server 7.0 # of indices Recovery time ( secs )
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 20 Data Warehousing  Warehousing Framework  Visual data modeler  Microsoft repository  Data transformation services (DTS)  Plato & Dcube - Multi Dimensional Data Cubes  English query 2.0 Easy Scalability Data Warehousing
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 22 Data Warehouse / Data Analysis Data Transformation Services to get data into the warehouse CUBE (OLE/DB OLAP) to analyze data Operational Data Extact & Load Data Warehouse Storage OLAP
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 25 Source table Partition 1 ROLAP Partition 2 Partition 3 ROLAP Europe USA Asia MD SQL SQL Plato and Data Cube and HOLAP “Plato” server “Plato” Designer Dcube Client app User 1 Dcube Client app User 2 RED WHITE BLUE By Color By Make & Year By Color & Year By Make By Year Sum
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 26 English Query
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 27 Easy Scalable Data Warehousing
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. “Shiloh” The Next SQL Server Shiloh (H1’00) - Strengthen Position • Data Warehousing leadership  Materialized Views  Cascading Referential Integrity (#1 requested user-group feature)  XML support • Scalability  WinCE support  W2K VLM (36 and 64 bit)  Multi-instance support Yukon – Next Big Step • Scalability (Clusters, Partitions) • Programmability • Ease of Use (Self Tuning, Auto Config)
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 29 Outline Summary of what you heard. (10 min) The database scene in general. (10 min) Scaleability: Farms, Clones,Parts & Packs (15 min) Microsoft DB research focus. (15 min) • TerraServer (design and ops). • RAGS. • Data Mining Q&A (10 min)
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 30 Info Capture You can record everything you see or hear or read. What will you do with it? How will you organize & analyze it? Most data will never be seen Analysis an summarization are key technologies Video 8 PB per lifetime (10GBph) Audio 30 TB (10KBps) Read or write: 8 GB (words) See: http://www.lesk.com/mlesk/ksg97/ksg.html Yotta Zetta Exa Peta Tera Giga Mega Kilo A Book .Movi e All books (words) All Books MultiMedia Everything ! Recorded A Photo
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 31 Data Tidal Wave  Seagate 47GB drive @ 783$ (= 1.7 ¢/mb) • 100 GB penny per MB drive coming in 2000  10 $/GB = 10 k$/ Terabyte! • “Everyone” can afford one  What’s a terror bite? • If you sell ten billion items a year (e.g Wal-Mart) • And you record 100 bytes on each one • Then you get a TeraByte/year  Where will the terror bytes come from? • Multimedia (like the TerraServer) and...
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 32 Reducing Data’s Cost-of-Ownership Self-Managing data Cost of ownership: One admin/TB (100K$ vs 10K$) Admin cost exceeds storage cost. SQL 7: Suggests indices Migrates data away from end of file Truncates file Someday: Automatic move files to balance disks Online defragmentation & restructuring Online physical redesign
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 33 OBJECT RELATIONAL The Next Great DBMS Wave All DB vendors have added objects to DB Microsoft is adding DBs to Objects Integration with COM+ Gives user-defined types and objects Plug-ins will be Billion dollar industry • Blades for SQL Server razor
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 34 Why Is XML Important? Self-describing data “ABC47-Z”, “100”, “STL”, “C”, “3”, “28” Data stream in a typical interface… <INVENTORY> <PART_NUM>ABC47-Z</PART_NUM> <QUANTITY>100</QUANTITY> <WAREHOUSE>STL</WAREHOUSE> <ZONE>C</ZONE> <AISLE>3</AISLE> <BIN>28</BIN> </INVENTORY> Same data stream in XML…
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 35 table.xsl bar.xsl art.xsl
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 36 XML Applications Exposing Software as a “Service” • Websites without UI’s • Exposed services with common scheme • Integration points at the enterprise, value- chain, workgroup, desktop and intelligent gizmo “levels” B2B value chains • Uses XML to transmit wide range of date to a broad set of stakeholders (regulatory agencies, suppliers, customers, etc.). • Leverage for prior efforts like EDI • BizTalk a key industry effort in this regard
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 37 XML Message XML Document XML Message XML: BizTalk Framework Order Processing MVS CICS SAP R/3 XML XML Service Interface Browser Client Apps New Form Factors XML Document Another Service JD Edwards XML XML Library www.biztalk.org XML XML schema
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 38 Outline Summary of what you heard. (20 min) The database scene in general. (10 min) Scaleability: Farms, Clones,Parts & Packs (10 min) Microsoft DB research focus. (15 min) • TerraServer (design and ops). • RAGS. • Data Mining Q&A (15 min)
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 39 Terminology for scaleability Farms of servers: • Clones: identical  Scaleability + availability • Partitions:  Scaleability • Packs  Partition availability via fail-over Farm Clone Partition Pack
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 40 Unpredictable Growth The TerraServer Story: • We expected 5 M hits per day • We got 50 M hits on day 1 • We peak at 15-20 M hpd on a “hot” day • Average 5 M hpd after 1 year Most of us cannot predict demand • Must be able to deal with NO demand • Must be able to deal with HUGE demand
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 41 An Architecture for Internet Services? Need to be able to add capacity • New processing • New storage • New networking Need continuous service • Online change of all components (hardware and software) • Multiple service sites • Multiple network providers Need great development tools • Change the application several times per year. • Add new services several times per year.
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 42 Premise: Each Site is a Farm Buy computing by the slice (brick): • Rack of servers + disks. Grow by adding slices • Spread data and computation to new slices Two growth styles: • Clones: anonymous servers • Parts+Packs: Partitions fail over within a pack In both cases, remote farm for disaster recovery Switched Ethernet Switched Ethernet www.microsoft.com (3) search.microsoft.com (1) premium.microsoft.com (1) European Data Center FTP Download Server (1) SQL SERVERS (2) Router msid.msn.com (1) MOSWest Admin LAN SQLNet Feeder LAN FDDI Ring (MIS4) Router www.microsoft.com (5) Building 11 Live SQL Server Router home.microsoft.com (5) FDDI Ring (MIS2) www.microsoft.com (4) activex.microsoft.com (2) search.microsoft.com (3) register.microsoft.com (2) msid.msn.com (1) FDDI Ring (MIS3) www.microsoft.com (3) premium.microsoft.com (1) msid.msn.com (1) FDDI Ring (MIS1) www.microsoft.com (4) premium.microsoft.com (2) register.microsoft.com (2) msid.msn.com (1) Primary Gigaswitch Secondary Gigaswitch Staging Servers (7) search.microsoft.com (3) support.microsoft.com (2) register.msn.com (2) 1997 Microsoft.Com Farm MOSWest DMZ Staging Servers Live SQL Servers SQL Consolidators Japan Data Center www.microsoft.com (3) premium.microsoft.com (1) HTTP Download Servers (2) Router search.microsoft.com (2) SQL SERVERS (2) msid.msn.com (1) FTP Download Server (1) Router Router Router Router Router Router Router Router Internal WWW SQL Reporting home.microsoft.com (4) home.microsoft.com (3) home.microsoft.com (2) register.microsoft.com (1) support.microsoft.com (1) Internet 13 DS3 (45 Mb/Sec Each) 2 OC3 (45Mb/Sec Each) 2 Ethernet (100 Mb/Sec Each) cdm.microsoft.com (1) FTP Servers Download Replication FTP.microsoft.com (3) 0 Switched Ethernet Switched Ethernet www.microsoft.com (3) Switched Ethernet Switched Ethernet www.microsoft.com (3) search.microsoft.com (1) premium.microsoft.com (1) European Data Center FTP Download Server (1) SQL SERVERS (2) Router msid.msn.com (1) MOSWest Admin LAN SQLNet Feeder LAN FDDI Ring (MIS4) Router www.microsoft.com (5) Building 11 Live SQL Server Router home.microsoft.com (5) FDDI Ring (MIS2) www.microsoft.com (4) activex.microsoft.com (2) search.microsoft.com (3) register.microsoft.com (2) msid.msn.com (1) search.microsoft.com (3) register.microsoft.com (2) msid.msn.com (1) FDDI Ring (MIS3) www.microsoft.com (3) premium.microsoft.com (1) msid.msn.com (1) FDDI Ring (MIS1) www.microsoft.com (4) premium.microsoft.com (2) FDDI Ring (MIS3) www.microsoft.com (3) premium.microsoft.com (1) msid.msn.com (1) FDDI Ring (MIS1) www.microsoft.com (4) premium.microsoft.com (2) register.microsoft.com (2) msid.msn.com (1) Primary Gigaswitch Secondary Gigaswitch register.microsoft.com (2) msid.msn.com (1) Primary Gigaswitch Secondary Gigaswitch Staging Servers (7) search.microsoft.com (3) support.microsoft.com (2) register.msn.com (2) 1997 Microsoft.Com Farm MOSWest DMZ Staging Servers Live SQL Servers SQL Consolidators Live SQL Servers SQL Consolidators Japan Data Center www.microsoft.com (3) premium.microsoft.com (1) HTTP Download Servers (2) Router search.microsoft.com (2) SQL SERVERS (2) msid.msn.com (1) premium.microsoft.com (1) HTTP Download Servers (2) Router search.microsoft.com (2) SQL SERVERS (2) msid.msn.com (1) FTP Download Server (1) Router Router Router Router Router Router Router Router FTP Download Server (1) Router Router Router Router Router Router Router Router Internal WWW SQL Reporting Internal WWW SQL Reporting home.microsoft.com (4) home.microsoft.com (3) home.microsoft.com (2) register.microsoft.com (1) support.microsoft.com (1) home.microsoft.com (4) home.microsoft.com (3) home.microsoft.com (2) register.microsoft.com (1) support.microsoft.com (1) Internet 13 DS3 (45 Mb/Sec Each) 2 OC3 (45Mb/Sec Each) 2 Ethernet (100 Mb/Sec Each) cdm.microsoft.com (1) FTP Servers Download Replication FTP.microsoft.com (3) 0
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 43 Scaleable Systems Scale UP and Scale OUT Everyone does both. Choice is • Size of a brick • Clones or partitions • Size of a pack
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 44 Everyone scales out What’s the Brick? 1M$/slice • IBM S390? • Sun E 10,000?  100 K$/slice • Wintel 8X  10 K$/slice • Wintel 4x 1 K$/slice • Wintel 1x
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 45 Clones: Availability+Scalability Some applications are • Read-mostly • Low consistency requirements • Modest storage requirement (less than 1TB) Examples: • HTTP web servers (IP sprayer/sieve + replication) • LDAP servers (replication via gossip) • App/compute servers or firewalls Replicate app at all nodes (clones) Spray requests across nodes. Grow by adding clones Fault tolerance: stop sending to dead clone. Growth: add a clone.
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 46 Facilities Clones Need Automatic replication • Applications (and system software) • Data Automatic request routing • Spray or sieve Management: • Who is up? • Update management & propagation • Application monitoring. Clones are very easy to manage: • Rule of thumb: 100’s of clones per admin
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 47 Partitions for Scalability Clones are not appropriate for some apps. • Statefull apps do not replicate well • high update rates do not replicate well • Huge DBs (disk to expensive to clone) Examples • Email / chat / … • Databases Partition state among servers Scalability (online): • Partition split/merge • Partitioning must be transparent to client.
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 48 Partitioned (aka. Clustered) Apps Mail servers • Perfectly partitionable  Business Object Servers • Partition by set of objects. Parallel Databases Transparent access to partitioned tables Parallel Query
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 49 Packs for Availability Each partition may fail (independent of others) Partitions migrate to new node via fail-over • Fail-over in seconds Pack: the nodes supporting a partition • VMS Cluster • Tandem Process Pair • SP2 HACMP • Sysplex™ • WinNT MSCS (wolfpack) Cluster In A Box now commodity Partitions grow in packs.
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 50 What Parts+Packs Need Automatic partitioning (in dbms, mail, files,…) • Location transparent • Partition split/merge • Grow without limits (100x10TB) Simple failover model • Partition migration is transparent • MSCS-like model for services Application-centric request routing Management: • Who is up? • Automatic partition management (split/merge) • Application monitoring.
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 51 Services on Clones & Partitions Application provides a set of services If cloned: • Services are on subset of clones If partitioned: • Services run at each partition System load balancing routes request to • Any clone • Correct partition. • Routes around failures.
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 52 Farm pairs: Always Up  Two farms  Changes from one sent to other  When one farm fails other provides service  Masks • Hardware/Software faults • Operations tasks (reorganize, upgrade move • Environmental faults (power fail)
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 53 Availabilty for a simple web site Web Clients Front End Web File Store SQL Temp State SQL Database Packs for availability Clones for availability Load Balance
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 54 Cloned Packed file servers Packed Partitions: Database Transparency Farm Scale Out Scenarios SQL Temp State Web File StoreA Cloned Front Ends (firewall, sprayer, web server) SQL Partition 3 The FARM: Clones and Packs of Partitions Web Clients Web File StoreB replication SQL Database SQL Partition 2 SQL Partition1 Load Balance
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 55 Clients IIS Web Server or other IP based services Data Servers SQL, Exchange, File Network Load Balancing Clones Cluster Service Pack COM+ Components Component Load Balancing (COM+) Clones Application Servers Reliable, Scalable, Modular 1 2 32 3 … 1 2 8 … 4 3
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 56 Talk 2 (if there is time) Terminology for scaleability Farms of servers: • Clones: identical  Scaleability + availability • Partitions:  Scaleability • Packs  Partition availability via fail-over Farm Clone Partition Pack
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 57 0 50 100 150 200 250 300 350 400 450 NT4 InProc W2K B1 inProc W2K RC1 InProc W2K RC2 InProc 1P 2P 4P 8P 0 50 100 150 200 250 300 350 400 450 NT4 InProc W2K B1 inProc W2K RC1 InProc W2K RC2 InProc NT4 OOP W2K B1 OOP W2K RC1 OOP W2K RC2 OOP 1P 2P 4P 8P Scalability: COM+ progress serving 1,000-statement ASP’s (servelets) Poor SMP Scaleability on IIS4 NT4 Big improvements from standard Transaction Processing tricks Out of Proc (safe execution) now much faster than In Proc was on IIS4 SPS: servelets per second (ASPs served per second by IIS, 1,000 statement VBscript) Shift from 4x200 Mhz to 8 450 Mhz
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 58 Scaleability: So, What about the death of NT/Alpha? Two simultaneous Compaq TPC-C numbers  Intel Profusion  NT/SQL/COM+  550 Mhz  8 Processors  4 GB memory  40,368 TPM-C @ 18.46$/tpmC  745 K$ 5-year cost  Avail: 12/31/99  Alpha  Unix/Sybase/Tuexdo  700 Mhz  8 Processors  16 GB memory  42,437 TPM-C @ 55.45 $/tpmC  $2.35 M$ 5-year cost  Avail: 10/18/99 200% more expense for 5% more performance?
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 59 Outline Summary of what you heard. (10 min) The database scene in general. (10 min) Scaleability: Farms, Clones,Parts & Packs (15 min) Microsoft DB research focus. (15 min) • TerraServer (design and ops). • RAGS. • Data Mining Q&A (10 min)
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 60 The TerraServer http://www.terraserver.microsoft.com/
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 61  Coverage: Range from 70ºN to 70ºS today: 35% U.S., 1% outside U.S.  Source Imagery: • 4 TB 1sq meter/pixel Aerial (USGS - 60,000 46Mb B&W- 151Mb Color IR files) • 1 TB 1.56 meter/pixel Satellite (Spin-2 - 2400 300 Mb B&W)  Display Imagery: 200x200 pixel images, subsample to build image pyramid  Store 5x compressed data  Nav Tools: • 1.5 m place names • “Click-on” Coverage map • Expedia & Virtual Globe • Pick of the week 1.6x 1.6 km “city view” .8 x .8 km 8m thumbnail .4 x.4 km browse 200x200 m tile Concept: User navigates an ‘almost seamless’ image of earth Database & application UI
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 62 browser HTML Java Viewer The Internet Web Client Image Delivery Application SQL Server 7 Microsoft Site Server EE Internet Information Server 4.0 Image Provider Site(s) TerraServer DB Terra-Server Stored Procedures Internet Information Server 5.0 Image Server Active Server Pages (ADO) MTS TerraServer Web Site Software: Classic 3 Tier Design SQL Server 7 24 20 (8/12) 46 46 Fire wall
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 63 SourceMeta Image Imagery Image Type Search Image Search Logical Schema Scale Job Load Job Load Mgmt External Link External Group External Geo Famous Category Famous Place TerraServer Terra Database TerraAdmin Admin Gazetteer Country Name State Name Place Name Place Type Small Place Name Pyramid
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 64 TerraServer File Group Layout Convert 324 disks to 28 RAID5 sets plus 28 spare drives Make 4 NT volumes (RAID 50) 595 GB per volume Build 30 20GB files on each volume DB is File Group of 120 files E: F: G: H:
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 65 Hardware 100 Mbps Ethernet Switch DS3 Site Servers Internet Map Server SPIN-2 Web Servers 2.9 TB Database Server AlphaServer 8400 8x400. 10 GB RAM 324 StorageWorks disks 10 drive tape library (STC Timber Wolf DLT7000 )
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 66 Backup and Recovery • STK 9710 Tape robot • SQL Server Backup & Restore + • Legato Networker • Fast, incremental, differential, online • Clocked at 80 MBps (peak) (~ 200 GB/hr) Restore • Fast, incremental (file oriented), not online. Load & Backup&Recovery Performance Data Bytes Backed Up 1.2 TB Total Time 7.25 Hours Number of Tapes Consumed 27 Total Tape Drives 10 Data ThroughPut 168 GB/Ho Average ThroughPut Per Device 16.8 GB/Hour Average Throughput Per Device 4.97 MB/Sec NTFS Logical Volumes 2
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 67 BAD OLD Load DLT Tape “tar” Drop’N’ DoJob Wait 4 Load LoadMgr DB 100mbit EtherSwitch 108 9.1 GB Drives Enterprise Storage Array Alpha Server 8400 108 9.1 GB Drives 108 9.1 GB Drives STC DLT Tape Library 60 4.3 GB Drives Alpha Server 4100 ESA Alpha Server 4100 LoadMgr DLT Tape NT Backup ImgCutter Drop’N’ Images 10: ImgCutter 20: Partition 30: ThumbImg 40: BrowseImg 45: JumpImg 50: TileImg 55: Meta Data 60: Tile Meta 70: Img Meta 80: Update Place ... LoadMgr
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 68 New Image Load and Update ODBC Tx TerraLoader ODBC TX TerraServer SQL DBMS DLT Tape “tar” Metadata Load DB Active Server Pages Cut & Load Scheduling System Image Cutter Merge ODBC Tx Dither Image Pyramid From base
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 70 TerraServer Daily Traffic Jun 22, 1998 thru June 22, 1999 0 10M 20M 30M Count Sessions Hit Page View DB Query Image After a Year: 1 TB of data 750 M records 2.3 billion Hits 2.0 billion DB Queries 1.7 billion Images sent 368 million Page Views 99.93% DB Availability 3rd design now Online Built and operated by team of 4 people TotalTime (Hours) 0 720 1440 2160 2880 3600 4320 5040 5760 6480 7200 7920 8640 Up Down Time (Hours:minutes) 0:00 0:30 1:00 1:30 2:00 2:30 3:00 3:30 4:00 4:30 5:00 5:30 6:00 Scheduled HW+Software Operations
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 74 What TerraServer Shows Can serve huge databases on Internet for about a penny a page view mostly phone bill (!). Advertising pays more than a penny a page. Commodity tools do scale fairly far. A few people (3 developers, 1 operator) using power tools can build an impressive web site
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 75 Outline Summary of what you heard. (20 min) The database scene in general. (10 min) Scaleability: Packs & Mobs (10 min) Microsoft DB research focus. (15 min) • TerraServer (design and ops). • RAGS. • Data Mining Q&A (15 min)
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 76 Automatic Testing 60% of Microsoft R&D is testing. What can research do to help? • beyond joining the 500,000 Win2K beta testers Test generation robot: • Make up SQL queries • Send them to SQL Server, Oracle, DB2, Informix,… • If answer is the same, great, if not there is a problem Also good for stress tests Found MANY bugs in our products (all fixed). Found MANY bugs in other’s products. Very valuable tool.  MSR-TR-98-21 Massive Stochastic Testing of SQL, Slutz, Don http://research.microsoft.com/scripts/pubDB/pubsasp.asp?RecordID=175 W X Y Z 1672 1672 1672 1672 232 234 241 31 1 1 1 1 31 15 12 28 1 12 5 116 0 29 32 4 18 18 19 25 45 19 18 113 Error All four agree 84% Problem with intermediate table. Case W,X, and agree 95%
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 77 Some Tera-Byte Databases Kilo Mega Giga Tera Peta Exa Zetta Yotta The Web: 1 TB of HTML TerraServer 1 TB of images Several other 1 TB (file) servers Hotmail: 20 TB of email Sloan Digital Sky Survey: 40 TB raw, 2 TB cooked EOS/DIS (picture of planet each week) • 15 PB by 2007 Federal Clearing house: images of checks • 15 PB by 2006 (7 year history) Nuclear Stockpile Stewardship Program • 10 Exabytes (???!!)
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 78 Data Mining Find interesting structure (patterns, relationships) in data • Prediction • Segmentation (clustering) • Dependency modeling (find distribution) • Summarization • Trend and change detection and modeling Allow user to state the query in terms of the business logic • User does not speak statistics or SQL Use data to build predictors • regression, classification, segmentation etc. Generate summaries and reports for insight • find “easy to describe” segments in data automatically • find segments not known to analyst
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 79 Data Mining: Microsoft SiteServer Commerce 3.0 Intelligent Cross-sell Based on: • Historical sales baskets in stores • Contents of current shopper basket • Browsing behavior of shopper Predict: ranking of products in store likely to be most interesting to shopper. Http://www.holtoutlet.com/outlet4
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 80 Mail to 25% and capture 40% 400% improved response! 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0.1% 0.2% 0.3% 0.6% 1.3% 5.3% 6.7% 25.5% 34.5% 43.8% 56.9% 68.5% 94.8% 98.5% 100.0% % mailed % Captured of true targets Real data drawn from a Microsoft marketing example
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 81 How do people use www.microsoft.com? Data Mining (Clustering) Engine User browsing data 100M hits per day 14M users/week X segments Cluster Visualizer Wizard
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 82
Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 83 Outline Summary of what you heard. (10 min) The database scene in general. (10 min) Scaleability: Farms, Clones,Parts & Packs (15 min) Microsoft DB research focus. (15 min) • TerraServer (design and ops). • RAGS. • Data Mining Q&A (10 min)

More Related Content

PPT
WebServices_Grid.ppt
EqinNiftalyev
 
PDF
Data Model for Mainframe in Splunk: The Newest Feature of Ironstream
Precisely
 
PDF
Secrets of Enterprise Data Mining 201310
Mark Tabladillo
 
PPTX
Architecting Cloudy Applications
David Chou
 
PPTX
2013 10-03-semantics-meetup-s buxton-mark_logic_pub
Stephen Buxton
 
PDF
Realtime Indexing for Fast Queries on Massive Semi-Structured Data
ScyllaDB
 
PDF
Secrets of Enterprise Data Mining 201305
Mark Tabladillo
 
PDF
Applied Semantic Search 201306
Mark Tabladillo
 
WebServices_Grid.ppt
EqinNiftalyev
 
Data Model for Mainframe in Splunk: The Newest Feature of Ironstream
Precisely
 
Secrets of Enterprise Data Mining 201310
Mark Tabladillo
 
Architecting Cloudy Applications
David Chou
 
2013 10-03-semantics-meetup-s buxton-mark_logic_pub
Stephen Buxton
 
Realtime Indexing for Fast Queries on Massive Semi-Structured Data
ScyllaDB
 
Secrets of Enterprise Data Mining 201305
Mark Tabladillo
 
Applied Semantic Search 201306
Mark Tabladillo
 

Similar to Gray_Compass99.ppt (20)

PDF
Simplifying Change Data Capture using Databricks Delta
Databricks
 
PDF
Off-Label Data Mesh: A Prescription for Healthier Data
HostedbyConfluent
 
PDF
Application Development and Data Modeling on Amazon DynamoDB
Amazon Web Services Japan
 
PDF
Graph Database Use Cases - StampedeCon 2015
StampedeCon
 
PDF
Graph database Use Cases
Max De Marzi
 
PDF
Sound cloud - User & Partner Conference - AT Internet
AT Internet
 
PPTX
Apache IOTDB: a Time Series Database for Industrial IoT
jixuan1989
 
PDF
What is a distributed data science pipeline. how with apache spark and friends.
Andy Petrella
 
PPTX
ISBG 2016 - XPages on IBM Bluemix
Oliver Busse
 
PDF
Towards a rebirth of data science (by Data Fellas)
Andy Petrella
 
PPTX
Large scale, interactive ad-hoc queries over different datastores with Apache...
jaxLondonConference
 
PPTX
Database 101 on IBM i
HelpSystems
 
PPTX
Understanding The Azure Platform November 09
DavidGristwood
 
PDF
ITCamp 2013 - Cristian Lefter - Transact-SQL from 0 to SQL Server 2012
ITCamp
 
PDF
Introduction to Text Mining and Visualization with Interactive Web Application
Olga Scrivner
 
PPTX
Big Data Analytics: Finding diamonds in the rough with Azure
Christos Charmatzis
 
PDF
Applied Enterprise Semantic Search 201305
Mark Tabladillo
 
PDF
The Google Hacking Database: A Key Resource to Exposing Vulnerabilities
TechWell
 
PPT
Synchronicity: Just-In-Time Discovery of Lost Web Pages
Michael Nelson
 
PDF
What You Need To Know About The Top Database Trends
Dell World
 
Simplifying Change Data Capture using Databricks Delta
Databricks
 
Off-Label Data Mesh: A Prescription for Healthier Data
HostedbyConfluent
 
Application Development and Data Modeling on Amazon DynamoDB
Amazon Web Services Japan
 
Graph Database Use Cases - StampedeCon 2015
StampedeCon
 
Graph database Use Cases
Max De Marzi
 
Sound cloud - User & Partner Conference - AT Internet
AT Internet
 
Apache IOTDB: a Time Series Database for Industrial IoT
jixuan1989
 
What is a distributed data science pipeline. how with apache spark and friends.
Andy Petrella
 
ISBG 2016 - XPages on IBM Bluemix
Oliver Busse
 
Towards a rebirth of data science (by Data Fellas)
Andy Petrella
 
Large scale, interactive ad-hoc queries over different datastores with Apache...
jaxLondonConference
 
Database 101 on IBM i
HelpSystems
 
Understanding The Azure Platform November 09
DavidGristwood
 
ITCamp 2013 - Cristian Lefter - Transact-SQL from 0 to SQL Server 2012
ITCamp
 
Introduction to Text Mining and Visualization with Interactive Web Application
Olga Scrivner
 
Big Data Analytics: Finding diamonds in the rough with Azure
Christos Charmatzis
 
Applied Enterprise Semantic Search 201305
Mark Tabladillo
 
The Google Hacking Database: A Key Resource to Exposing Vulnerabilities
TechWell
 
Synchronicity: Just-In-Time Discovery of Lost Web Pages
Michael Nelson
 
What You Need To Know About The Top Database Trends
Dell World
 
Ad

Recently uploaded (20)

PPTX
A Presentation on Touch Screen Technology
memembruh336
 
PDF
DASA ADMISSION 2024_FirstRound_FirstRank_LastRank.pdf
rameswartudu05
 
PDF
ENT215_Completing-a-large-scale-migration-and-modernization-with-AWS.pdf
ssuser28425f
 
PDF
Hybrid model detection and classification of lung cancer
IAESIJAI
 
PPTX
Group 1 Presentation -Planning and Decision Making .pptx
MatthewLewis227954
 
PPTX
A Presentation on Artificial Intelligence
memembruh336
 
PDF
Assigned Numbers - 2025 - Bluetooth® Document
Bloombase
 
PDF
NewMind AI Weekly Chronicles - August'25-Week II
NewMind AI
 
PDF
From MVP to Full-Scale Product A Startup’s Software Journey.pdf
KodekX
 
PDF
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
PPTX
TLE Review Electricity (Electricity).pptx
JayPolicarpio2
 
PDF
Getting Started with Data Integration: FME Form 101
Safe Software
 
PPTX
cloud_computing_Infrastucture_as_cloud_p
mainakbhattacharya20
 
PDF
A comparative study of natural language inference in Swahili using monolingua...
IAESIJAI
 
PDF
Enhancing emotion recognition model for a student engagement use case through...
IAESIJAI
 
PPTX
Chapter 5: Probability Theory and Statistics
RandyFin
 
PDF
A novel scalable deep ensemble learning framework for big data classification...
IAESIJAI
 
PDF
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
A Presentation on Touch Screen Technology
memembruh336
 
DASA ADMISSION 2024_FirstRound_FirstRank_LastRank.pdf
rameswartudu05
 
ENT215_Completing-a-large-scale-migration-and-modernization-with-AWS.pdf
ssuser28425f
 
Hybrid model detection and classification of lung cancer
IAESIJAI
 
Group 1 Presentation -Planning and Decision Making .pptx
MatthewLewis227954
 
A Presentation on Artificial Intelligence
memembruh336
 
Assigned Numbers - 2025 - Bluetooth® Document
Bloombase
 
NewMind AI Weekly Chronicles - August'25-Week II
NewMind AI
 
From MVP to Full-Scale Product A Startup’s Software Journey.pdf
KodekX
 
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
TLE Review Electricity (Electricity).pptx
JayPolicarpio2
 
Getting Started with Data Integration: FME Form 101
Safe Software
 
cloud_computing_Infrastucture_as_cloud_p
mainakbhattacharya20
 
A comparative study of natural language inference in Swahili using monolingua...
IAESIJAI
 
Enhancing emotion recognition model for a student engagement use case through...
IAESIJAI
 
Chapter 5: Probability Theory and Statistics
RandyFin
 
A novel scalable deep ensemble learning framework for big data classification...
IAESIJAI
 
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
Ad

Gray_Compass99.ppt

  • 1. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 1 Microsoft SQL Server, Scalability, & Database Research Jim Gray Researcher Microsoft Corporation
  • 2. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 2 Outline Summary of what you heard. (10 min) The database scene in general. (10 min) Scaleability: Farms, Clones,Parts & Packs (15 min) Microsoft DB research focus. (15 min) • TerraServer (design and ops). • RAGS. • Data Mining Q&A (10 min)
  • 3. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 3 Organizations Are Going Online Building a digital nervous system. Inexpensive hardware means huge databases are possible. But, we are drowning in data. Databases help organize information. Microsoft’s goal: • Information at your fingertips. • Make it easy to capture, manage, and analyze information.
  • 4. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 4 Microsoft SQL Server 7 Goals Easy  Dynamic self management  Multi-site management  Operation Scripting  Job scheduling and execution  Alert/response management  Scriptable Install+upgrade  DBA profiling/tuning tools  Unicode  English Language Query  Integrated with NT Security  Integrated with NT files Scalability Data Warehousing
  • 5. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 11 Scalability  Win9x/NTW version  Dynamic row-level locking  Improved query optimizer  Intra-query parallelism  VLDB improvements  Replication improvements  Distributed query  High Availability Clusters Easy Scalability Data Warehousing
  • 6. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 12 Scale Down to Windows 95-98 Full function (same as NTW) Integration with Access 97 MSDE in Office2000 and MSDN WinCE version demonstrated
  • 7. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 13 Replication Transactional and Merge Remote update ODBC and OLE DB subscribers Wizards Performance 2PC, RPC Subscriber DB2 CICS Subscriber Subscriber VSAM OS 390 DB2 Publisher Updating Subscriber (immediate updates) Distributor Subscriber
  • 8. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 14 Query Processor Enhancements Parallelism Improved scan, fetch, & sort Smart hash & merge join Large joins & grouping Better query optimization Multi-index operations Automatic statistics maintenance Distributed Query Heterogeneous Query Focus on Complex Queries
  • 9. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 15 •# of emp. per group •total inc. per group Local Agg. 4 x 50 rows + + + + Disks 50,000 rows Global Agg. Result 50 rows + Parallel Query SMP & Disk Parallelism Plus Distributed Plus Hash Join (fanciest on the planet) Plus Optimized Partitioned views
  • 10. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 16 Distributed Heterogeneous Queries Data Fusion / Integration Join spread sheets, databases, directories, Text DBs etc. Any source that exposes OLE DB interfaces SQL Server as gateway, even on the desktop Database (DB2, VSAM, Oracle, …) Spreadsheet Photos Mail Maps Documents and the Web Directory Service SQL 7.0 Query Processor
  • 11. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 17 Utilities The Key to LARGE Databases  Backup • Fuzzy • Parallel • Incremental • Restartable Recovery • Fast • File granularity Reorganize • shrinks file • reclusters file Auto-Repair Index creation ~2x faster than 6.5 DBCC • not required, • a good practice • 5x - 100x faster 0 10 20 30 40 50 60 1 2 3 4 SQL Server 6.5 SQL Server 7.0 # of indices Recovery time ( secs )
  • 12. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 20 Data Warehousing  Warehousing Framework  Visual data modeler  Microsoft repository  Data transformation services (DTS)  Plato & Dcube - Multi Dimensional Data Cubes  English query 2.0 Easy Scalability Data Warehousing
  • 13. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 22 Data Warehouse / Data Analysis Data Transformation Services to get data into the warehouse CUBE (OLE/DB OLAP) to analyze data Operational Data Extact & Load Data Warehouse Storage OLAP
  • 14. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 25 Source table Partition 1 ROLAP Partition 2 Partition 3 ROLAP Europe USA Asia MD SQL SQL Plato and Data Cube and HOLAP “Plato” server “Plato” Designer Dcube Client app User 1 Dcube Client app User 2 RED WHITE BLUE By Color By Make & Year By Color & Year By Make By Year Sum
  • 15. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 26 English Query
  • 16. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 27 Easy Scalable Data Warehousing
  • 17. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. “Shiloh” The Next SQL Server Shiloh (H1’00) - Strengthen Position • Data Warehousing leadership  Materialized Views  Cascading Referential Integrity (#1 requested user-group feature)  XML support • Scalability  WinCE support  W2K VLM (36 and 64 bit)  Multi-instance support Yukon – Next Big Step • Scalability (Clusters, Partitions) • Programmability • Ease of Use (Self Tuning, Auto Config)
  • 18. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 29 Outline Summary of what you heard. (10 min) The database scene in general. (10 min) Scaleability: Farms, Clones,Parts & Packs (15 min) Microsoft DB research focus. (15 min) • TerraServer (design and ops). • RAGS. • Data Mining Q&A (10 min)
  • 19. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 30 Info Capture You can record everything you see or hear or read. What will you do with it? How will you organize & analyze it? Most data will never be seen Analysis an summarization are key technologies Video 8 PB per lifetime (10GBph) Audio 30 TB (10KBps) Read or write: 8 GB (words) See: http://www.lesk.com/mlesk/ksg97/ksg.html Yotta Zetta Exa Peta Tera Giga Mega Kilo A Book .Movi e All books (words) All Books MultiMedia Everything ! Recorded A Photo
  • 20. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 31 Data Tidal Wave  Seagate 47GB drive @ 783$ (= 1.7 ¢/mb) • 100 GB penny per MB drive coming in 2000  10 $/GB = 10 k$/ Terabyte! • “Everyone” can afford one  What’s a terror bite? • If you sell ten billion items a year (e.g Wal-Mart) • And you record 100 bytes on each one • Then you get a TeraByte/year  Where will the terror bytes come from? • Multimedia (like the TerraServer) and...
  • 21. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 32 Reducing Data’s Cost-of-Ownership Self-Managing data Cost of ownership: One admin/TB (100K$ vs 10K$) Admin cost exceeds storage cost. SQL 7: Suggests indices Migrates data away from end of file Truncates file Someday: Automatic move files to balance disks Online defragmentation & restructuring Online physical redesign
  • 22. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 33 OBJECT RELATIONAL The Next Great DBMS Wave All DB vendors have added objects to DB Microsoft is adding DBs to Objects Integration with COM+ Gives user-defined types and objects Plug-ins will be Billion dollar industry • Blades for SQL Server razor
  • 23. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 34 Why Is XML Important? Self-describing data “ABC47-Z”, “100”, “STL”, “C”, “3”, “28” Data stream in a typical interface… <INVENTORY> <PART_NUM>ABC47-Z</PART_NUM> <QUANTITY>100</QUANTITY> <WAREHOUSE>STL</WAREHOUSE> <ZONE>C</ZONE> <AISLE>3</AISLE> <BIN>28</BIN> </INVENTORY> Same data stream in XML…
  • 24. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 35 table.xsl bar.xsl art.xsl
  • 25. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 36 XML Applications Exposing Software as a “Service” • Websites without UI’s • Exposed services with common scheme • Integration points at the enterprise, value- chain, workgroup, desktop and intelligent gizmo “levels” B2B value chains • Uses XML to transmit wide range of date to a broad set of stakeholders (regulatory agencies, suppliers, customers, etc.). • Leverage for prior efforts like EDI • BizTalk a key industry effort in this regard
  • 26. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 37 XML Message XML Document XML Message XML: BizTalk Framework Order Processing MVS CICS SAP R/3 XML XML Service Interface Browser Client Apps New Form Factors XML Document Another Service JD Edwards XML XML Library www.biztalk.org XML XML schema
  • 27. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 38 Outline Summary of what you heard. (20 min) The database scene in general. (10 min) Scaleability: Farms, Clones,Parts & Packs (10 min) Microsoft DB research focus. (15 min) • TerraServer (design and ops). • RAGS. • Data Mining Q&A (15 min)
  • 28. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 39 Terminology for scaleability Farms of servers: • Clones: identical  Scaleability + availability • Partitions:  Scaleability • Packs  Partition availability via fail-over Farm Clone Partition Pack
  • 29. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 40 Unpredictable Growth The TerraServer Story: • We expected 5 M hits per day • We got 50 M hits on day 1 • We peak at 15-20 M hpd on a “hot” day • Average 5 M hpd after 1 year Most of us cannot predict demand • Must be able to deal with NO demand • Must be able to deal with HUGE demand
  • 30. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 41 An Architecture for Internet Services? Need to be able to add capacity • New processing • New storage • New networking Need continuous service • Online change of all components (hardware and software) • Multiple service sites • Multiple network providers Need great development tools • Change the application several times per year. • Add new services several times per year.
  • 31. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 42 Premise: Each Site is a Farm Buy computing by the slice (brick): • Rack of servers + disks. Grow by adding slices • Spread data and computation to new slices Two growth styles: • Clones: anonymous servers • Parts+Packs: Partitions fail over within a pack In both cases, remote farm for disaster recovery Switched Ethernet Switched Ethernet www.microsoft.com (3) search.microsoft.com (1) premium.microsoft.com (1) European Data Center FTP Download Server (1) SQL SERVERS (2) Router msid.msn.com (1) MOSWest Admin LAN SQLNet Feeder LAN FDDI Ring (MIS4) Router www.microsoft.com (5) Building 11 Live SQL Server Router home.microsoft.com (5) FDDI Ring (MIS2) www.microsoft.com (4) activex.microsoft.com (2) search.microsoft.com (3) register.microsoft.com (2) msid.msn.com (1) FDDI Ring (MIS3) www.microsoft.com (3) premium.microsoft.com (1) msid.msn.com (1) FDDI Ring (MIS1) www.microsoft.com (4) premium.microsoft.com (2) register.microsoft.com (2) msid.msn.com (1) Primary Gigaswitch Secondary Gigaswitch Staging Servers (7) search.microsoft.com (3) support.microsoft.com (2) register.msn.com (2) 1997 Microsoft.Com Farm MOSWest DMZ Staging Servers Live SQL Servers SQL Consolidators Japan Data Center www.microsoft.com (3) premium.microsoft.com (1) HTTP Download Servers (2) Router search.microsoft.com (2) SQL SERVERS (2) msid.msn.com (1) FTP Download Server (1) Router Router Router Router Router Router Router Router Internal WWW SQL Reporting home.microsoft.com (4) home.microsoft.com (3) home.microsoft.com (2) register.microsoft.com (1) support.microsoft.com (1) Internet 13 DS3 (45 Mb/Sec Each) 2 OC3 (45Mb/Sec Each) 2 Ethernet (100 Mb/Sec Each) cdm.microsoft.com (1) FTP Servers Download Replication FTP.microsoft.com (3) 0 Switched Ethernet Switched Ethernet www.microsoft.com (3) Switched Ethernet Switched Ethernet www.microsoft.com (3) search.microsoft.com (1) premium.microsoft.com (1) European Data Center FTP Download Server (1) SQL SERVERS (2) Router msid.msn.com (1) MOSWest Admin LAN SQLNet Feeder LAN FDDI Ring (MIS4) Router www.microsoft.com (5) Building 11 Live SQL Server Router home.microsoft.com (5) FDDI Ring (MIS2) www.microsoft.com (4) activex.microsoft.com (2) search.microsoft.com (3) register.microsoft.com (2) msid.msn.com (1) search.microsoft.com (3) register.microsoft.com (2) msid.msn.com (1) FDDI Ring (MIS3) www.microsoft.com (3) premium.microsoft.com (1) msid.msn.com (1) FDDI Ring (MIS1) www.microsoft.com (4) premium.microsoft.com (2) FDDI Ring (MIS3) www.microsoft.com (3) premium.microsoft.com (1) msid.msn.com (1) FDDI Ring (MIS1) www.microsoft.com (4) premium.microsoft.com (2) register.microsoft.com (2) msid.msn.com (1) Primary Gigaswitch Secondary Gigaswitch register.microsoft.com (2) msid.msn.com (1) Primary Gigaswitch Secondary Gigaswitch Staging Servers (7) search.microsoft.com (3) support.microsoft.com (2) register.msn.com (2) 1997 Microsoft.Com Farm MOSWest DMZ Staging Servers Live SQL Servers SQL Consolidators Live SQL Servers SQL Consolidators Japan Data Center www.microsoft.com (3) premium.microsoft.com (1) HTTP Download Servers (2) Router search.microsoft.com (2) SQL SERVERS (2) msid.msn.com (1) premium.microsoft.com (1) HTTP Download Servers (2) Router search.microsoft.com (2) SQL SERVERS (2) msid.msn.com (1) FTP Download Server (1) Router Router Router Router Router Router Router Router FTP Download Server (1) Router Router Router Router Router Router Router Router Internal WWW SQL Reporting Internal WWW SQL Reporting home.microsoft.com (4) home.microsoft.com (3) home.microsoft.com (2) register.microsoft.com (1) support.microsoft.com (1) home.microsoft.com (4) home.microsoft.com (3) home.microsoft.com (2) register.microsoft.com (1) support.microsoft.com (1) Internet 13 DS3 (45 Mb/Sec Each) 2 OC3 (45Mb/Sec Each) 2 Ethernet (100 Mb/Sec Each) cdm.microsoft.com (1) FTP Servers Download Replication FTP.microsoft.com (3) 0
  • 32. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 43 Scaleable Systems Scale UP and Scale OUT Everyone does both. Choice is • Size of a brick • Clones or partitions • Size of a pack
  • 33. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 44 Everyone scales out What’s the Brick? 1M$/slice • IBM S390? • Sun E 10,000?  100 K$/slice • Wintel 8X  10 K$/slice • Wintel 4x 1 K$/slice • Wintel 1x
  • 34. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 45 Clones: Availability+Scalability Some applications are • Read-mostly • Low consistency requirements • Modest storage requirement (less than 1TB) Examples: • HTTP web servers (IP sprayer/sieve + replication) • LDAP servers (replication via gossip) • App/compute servers or firewalls Replicate app at all nodes (clones) Spray requests across nodes. Grow by adding clones Fault tolerance: stop sending to dead clone. Growth: add a clone.
  • 35. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 46 Facilities Clones Need Automatic replication • Applications (and system software) • Data Automatic request routing • Spray or sieve Management: • Who is up? • Update management & propagation • Application monitoring. Clones are very easy to manage: • Rule of thumb: 100’s of clones per admin
  • 36. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 47 Partitions for Scalability Clones are not appropriate for some apps. • Statefull apps do not replicate well • high update rates do not replicate well • Huge DBs (disk to expensive to clone) Examples • Email / chat / … • Databases Partition state among servers Scalability (online): • Partition split/merge • Partitioning must be transparent to client.
  • 37. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 48 Partitioned (aka. Clustered) Apps Mail servers • Perfectly partitionable  Business Object Servers • Partition by set of objects. Parallel Databases Transparent access to partitioned tables Parallel Query
  • 38. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 49 Packs for Availability Each partition may fail (independent of others) Partitions migrate to new node via fail-over • Fail-over in seconds Pack: the nodes supporting a partition • VMS Cluster • Tandem Process Pair • SP2 HACMP • Sysplex™ • WinNT MSCS (wolfpack) Cluster In A Box now commodity Partitions grow in packs.
  • 39. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 50 What Parts+Packs Need Automatic partitioning (in dbms, mail, files,…) • Location transparent • Partition split/merge • Grow without limits (100x10TB) Simple failover model • Partition migration is transparent • MSCS-like model for services Application-centric request routing Management: • Who is up? • Automatic partition management (split/merge) • Application monitoring.
  • 40. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 51 Services on Clones & Partitions Application provides a set of services If cloned: • Services are on subset of clones If partitioned: • Services run at each partition System load balancing routes request to • Any clone • Correct partition. • Routes around failures.
  • 41. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 52 Farm pairs: Always Up  Two farms  Changes from one sent to other  When one farm fails other provides service  Masks • Hardware/Software faults • Operations tasks (reorganize, upgrade move • Environmental faults (power fail)
  • 42. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 53 Availabilty for a simple web site Web Clients Front End Web File Store SQL Temp State SQL Database Packs for availability Clones for availability Load Balance
  • 43. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 54 Cloned Packed file servers Packed Partitions: Database Transparency Farm Scale Out Scenarios SQL Temp State Web File StoreA Cloned Front Ends (firewall, sprayer, web server) SQL Partition 3 The FARM: Clones and Packs of Partitions Web Clients Web File StoreB replication SQL Database SQL Partition 2 SQL Partition1 Load Balance
  • 44. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 55 Clients IIS Web Server or other IP based services Data Servers SQL, Exchange, File Network Load Balancing Clones Cluster Service Pack COM+ Components Component Load Balancing (COM+) Clones Application Servers Reliable, Scalable, Modular 1 2 32 3 … 1 2 8 … 4 3
  • 45. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 56 Talk 2 (if there is time) Terminology for scaleability Farms of servers: • Clones: identical  Scaleability + availability • Partitions:  Scaleability • Packs  Partition availability via fail-over Farm Clone Partition Pack
  • 46. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 57 0 50 100 150 200 250 300 350 400 450 NT4 InProc W2K B1 inProc W2K RC1 InProc W2K RC2 InProc 1P 2P 4P 8P 0 50 100 150 200 250 300 350 400 450 NT4 InProc W2K B1 inProc W2K RC1 InProc W2K RC2 InProc NT4 OOP W2K B1 OOP W2K RC1 OOP W2K RC2 OOP 1P 2P 4P 8P Scalability: COM+ progress serving 1,000-statement ASP’s (servelets) Poor SMP Scaleability on IIS4 NT4 Big improvements from standard Transaction Processing tricks Out of Proc (safe execution) now much faster than In Proc was on IIS4 SPS: servelets per second (ASPs served per second by IIS, 1,000 statement VBscript) Shift from 4x200 Mhz to 8 450 Mhz
  • 47. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 58 Scaleability: So, What about the death of NT/Alpha? Two simultaneous Compaq TPC-C numbers  Intel Profusion  NT/SQL/COM+  550 Mhz  8 Processors  4 GB memory  40,368 TPM-C @ 18.46$/tpmC  745 K$ 5-year cost  Avail: 12/31/99  Alpha  Unix/Sybase/Tuexdo  700 Mhz  8 Processors  16 GB memory  42,437 TPM-C @ 55.45 $/tpmC  $2.35 M$ 5-year cost  Avail: 10/18/99 200% more expense for 5% more performance?
  • 48. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 59 Outline Summary of what you heard. (10 min) The database scene in general. (10 min) Scaleability: Farms, Clones,Parts & Packs (15 min) Microsoft DB research focus. (15 min) • TerraServer (design and ops). • RAGS. • Data Mining Q&A (10 min)
  • 49. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 60 The TerraServer http://www.terraserver.microsoft.com/
  • 50. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 61  Coverage: Range from 70ºN to 70ºS today: 35% U.S., 1% outside U.S.  Source Imagery: • 4 TB 1sq meter/pixel Aerial (USGS - 60,000 46Mb B&W- 151Mb Color IR files) • 1 TB 1.56 meter/pixel Satellite (Spin-2 - 2400 300 Mb B&W)  Display Imagery: 200x200 pixel images, subsample to build image pyramid  Store 5x compressed data  Nav Tools: • 1.5 m place names • “Click-on” Coverage map • Expedia & Virtual Globe • Pick of the week 1.6x 1.6 km “city view” .8 x .8 km 8m thumbnail .4 x.4 km browse 200x200 m tile Concept: User navigates an ‘almost seamless’ image of earth Database & application UI
  • 51. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 62 browser HTML Java Viewer The Internet Web Client Image Delivery Application SQL Server 7 Microsoft Site Server EE Internet Information Server 4.0 Image Provider Site(s) TerraServer DB Terra-Server Stored Procedures Internet Information Server 5.0 Image Server Active Server Pages (ADO) MTS TerraServer Web Site Software: Classic 3 Tier Design SQL Server 7 24 20 (8/12) 46 46 Fire wall
  • 52. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 63 SourceMeta Image Imagery Image Type Search Image Search Logical Schema Scale Job Load Job Load Mgmt External Link External Group External Geo Famous Category Famous Place TerraServer Terra Database TerraAdmin Admin Gazetteer Country Name State Name Place Name Place Type Small Place Name Pyramid
  • 53. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 64 TerraServer File Group Layout Convert 324 disks to 28 RAID5 sets plus 28 spare drives Make 4 NT volumes (RAID 50) 595 GB per volume Build 30 20GB files on each volume DB is File Group of 120 files E: F: G: H:
  • 54. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 65 Hardware 100 Mbps Ethernet Switch DS3 Site Servers Internet Map Server SPIN-2 Web Servers 2.9 TB Database Server AlphaServer 8400 8x400. 10 GB RAM 324 StorageWorks disks 10 drive tape library (STC Timber Wolf DLT7000 )
  • 55. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 66 Backup and Recovery • STK 9710 Tape robot • SQL Server Backup & Restore + • Legato Networker • Fast, incremental, differential, online • Clocked at 80 MBps (peak) (~ 200 GB/hr) Restore • Fast, incremental (file oriented), not online. Load & Backup&Recovery Performance Data Bytes Backed Up 1.2 TB Total Time 7.25 Hours Number of Tapes Consumed 27 Total Tape Drives 10 Data ThroughPut 168 GB/Ho Average ThroughPut Per Device 16.8 GB/Hour Average Throughput Per Device 4.97 MB/Sec NTFS Logical Volumes 2
  • 56. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 67 BAD OLD Load DLT Tape “tar” Drop’N’ DoJob Wait 4 Load LoadMgr DB 100mbit EtherSwitch 108 9.1 GB Drives Enterprise Storage Array Alpha Server 8400 108 9.1 GB Drives 108 9.1 GB Drives STC DLT Tape Library 60 4.3 GB Drives Alpha Server 4100 ESA Alpha Server 4100 LoadMgr DLT Tape NT Backup ImgCutter Drop’N’ Images 10: ImgCutter 20: Partition 30: ThumbImg 40: BrowseImg 45: JumpImg 50: TileImg 55: Meta Data 60: Tile Meta 70: Img Meta 80: Update Place ... LoadMgr
  • 57. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 68 New Image Load and Update ODBC Tx TerraLoader ODBC TX TerraServer SQL DBMS DLT Tape “tar” Metadata Load DB Active Server Pages Cut & Load Scheduling System Image Cutter Merge ODBC Tx Dither Image Pyramid From base
  • 58. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 70 TerraServer Daily Traffic Jun 22, 1998 thru June 22, 1999 0 10M 20M 30M Count Sessions Hit Page View DB Query Image After a Year: 1 TB of data 750 M records 2.3 billion Hits 2.0 billion DB Queries 1.7 billion Images sent 368 million Page Views 99.93% DB Availability 3rd design now Online Built and operated by team of 4 people TotalTime (Hours) 0 720 1440 2160 2880 3600 4320 5040 5760 6480 7200 7920 8640 Up Down Time (Hours:minutes) 0:00 0:30 1:00 1:30 2:00 2:30 3:00 3:30 4:00 4:30 5:00 5:30 6:00 Scheduled HW+Software Operations
  • 59. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 74 What TerraServer Shows Can serve huge databases on Internet for about a penny a page view mostly phone bill (!). Advertising pays more than a penny a page. Commodity tools do scale fairly far. A few people (3 developers, 1 operator) using power tools can build an impressive web site
  • 60. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 75 Outline Summary of what you heard. (20 min) The database scene in general. (10 min) Scaleability: Packs & Mobs (10 min) Microsoft DB research focus. (15 min) • TerraServer (design and ops). • RAGS. • Data Mining Q&A (15 min)
  • 61. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 76 Automatic Testing 60% of Microsoft R&D is testing. What can research do to help? • beyond joining the 500,000 Win2K beta testers Test generation robot: • Make up SQL queries • Send them to SQL Server, Oracle, DB2, Informix,… • If answer is the same, great, if not there is a problem Also good for stress tests Found MANY bugs in our products (all fixed). Found MANY bugs in other’s products. Very valuable tool.  MSR-TR-98-21 Massive Stochastic Testing of SQL, Slutz, Don http://research.microsoft.com/scripts/pubDB/pubsasp.asp?RecordID=175 W X Y Z 1672 1672 1672 1672 232 234 241 31 1 1 1 1 31 15 12 28 1 12 5 116 0 29 32 4 18 18 19 25 45 19 18 113 Error All four agree 84% Problem with intermediate table. Case W,X, and agree 95%
  • 62. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 77 Some Tera-Byte Databases Kilo Mega Giga Tera Peta Exa Zetta Yotta The Web: 1 TB of HTML TerraServer 1 TB of images Several other 1 TB (file) servers Hotmail: 20 TB of email Sloan Digital Sky Survey: 40 TB raw, 2 TB cooked EOS/DIS (picture of planet each week) • 15 PB by 2007 Federal Clearing house: images of checks • 15 PB by 2006 (7 year history) Nuclear Stockpile Stewardship Program • 10 Exabytes (???!!)
  • 63. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 78 Data Mining Find interesting structure (patterns, relationships) in data • Prediction • Segmentation (clustering) • Dependency modeling (find distribution) • Summarization • Trend and change detection and modeling Allow user to state the query in terms of the business logic • User does not speak statistics or SQL Use data to build predictors • regression, classification, segmentation etc. Generate summaries and reports for insight • find “easy to describe” segments in data automatically • find segments not known to analyst
  • 64. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 79 Data Mining: Microsoft SiteServer Commerce 3.0 Intelligent Cross-sell Based on: • Historical sales baskets in stores • Contents of current shopper basket • Browsing behavior of shopper Predict: ranking of products in store likely to be most interesting to shopper. Http://www.holtoutlet.com/outlet4
  • 65. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 80 Mail to 25% and capture 40% 400% improved response! 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0.1% 0.2% 0.3% 0.6% 1.3% 5.3% 6.7% 25.5% 34.5% 43.8% 56.9% 68.5% 94.8% 98.5% 100.0% % mailed % Captured of true targets Real data drawn from a Microsoft marketing example
  • 66. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 81 How do people use www.microsoft.com? Data Mining (Clustering) Engine User browsing data 100M hits per day 14M users/week X segments Cluster Visualizer Wizard
  • 67. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 82
  • 68. Jim Gray, Research and Microsoft SQL Server Microsoft Research http://research.Microsoft.com/~gray/talks/ 3 October 1999 Chicago, Ill. 83 Outline Summary of what you heard. (10 min) The database scene in general. (10 min) Scaleability: Farms, Clones,Parts & Packs (15 min) Microsoft DB research focus. (15 min) • TerraServer (design and ops). • RAGS. • Data Mining Q&A (10 min)

Editor's Notes

  • #2: 3 October 1999
  • #3: 3 October 1999
  • #4: 3 October 1999
  • #5: 3 October 1999
  • #6: 3 October 1999
  • #7: 3 October 1999
  • #8: 3 October 1999
  • #9: 3 October 1999
  • #10: 3 October 1999
  • #11: 3 October 1999
  • #15: 3 October 1999
  • #22: 3 October 1999
  • #23: 3 October 1999
  • #25: 3 October 1999
  • #26: 3 October 1999
  • #29: 3 October 1999
  • #30: 3 October 1999
  • #31: 3 October 1999
  • #39: 3 October 1999
  • #44: 3 October 1999
  • #60: 3 October 1999
  • #62: 3 October 1999
  • #63: 3 October 1999
  • #66: 3 October 1999
  • #67: 3 October 1999
  • #76: 3 October 1999
  • #78: 3 October 1999
  • #84: 3 October 1999