SlideShare a Scribd company logo

common_schema 2.0: DBA's Framework for MySQL

Shlomi Noach, profile picture
Shlomi Noach

This document describes common_schema, an open source framework for MySQL that includes views, routines, QueryScript, and a debugger. The views provide aggregated metadata and recommendations. Routines extend MySQL's functionality with text/security functions. QueryScript allows SQL scripting. The debugger helps debug stored routines. Common_schema is installed via an SQL file and resides completely within MySQL.

Real Estate
1 of 42
Downloaded 15 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
common_schema2.0 D B A ' s f r a m e w o r k f o r M y S Q L S h l o m i N o a c h h t t p : / / o p e n a r k . o r g
Copyright © 2013, Shlomi Noach common_schema D B A ' s f r a m e w o r k f o r M y S Q L common_schema2.0 D B A ' s f r a m e w o r k f o r M y S Q L  About  Views & eval()  Routines  QueryScript  Introducing rdebug
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach About common_schema  common_schema is an open source project, licensed under the GPL License.  Authored by Shlomi Noach http://openark.org  Major contributions by Roland Bouman http://rpbouman.blogspot.com  Several contributions & suggestions by the community (Thanks!)
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach About common_schema  common_schema is a framework which includes: – Views: analytics, security, action-taking, ... – Routines: text, temporal, process, security, ... – QueryScript: an SQL oriented scripting language & interpreter – rdebug: debugger and debugging API for MySQL stored routines (alpha).
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach About common_schema  It is a schema that lies next to INFORMATION_SCHEMA.  It lies completely within the MySQL server, and does not require external packages or dependencies. No Perl scripts nor UDFs or plugins.
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Getting & Installing  common_schema distribution is a SQL file.  Currently hosted on Google Code: http://code.google.com/p/common-schema/  Install by importing SQL file into MySQL: bash$ mysql < /tmp/common_schema-1.3.1.sql complete - Base components: installed - InnoDB Plugin components: installed - Percona Server components: not installed Installation complete. Thank you for using common_schema!
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Views  Views providing non-trivial information about your table design, keys, grants, processes, transactions, locks, ...  While INFORMATION_SCHEMA provides with complete info, it is ofter difficult to aggregate. It is sometimes too normalized, and at other times too de-normalized.  Some metadata is simply not available in INFORMATION_SCHEMA.
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Views  common_schema's views will present with valuable knowledge  And will typically offer recommendations or possible actions based on that knowledge.  common_schema also provides the mechanism to apply those recommendations.
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Views: redundant_keys  Find duplicate/redundant keys in your schema. Recommend DROP statements. http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/redundant_keys.html – Similar to pt-duplicate-key-checker mysql> select * from redundant_keys where table_schema='sakila'G table_schema: sakila table_name: rental redundant_index_name: rental_date_2 redundant_index_columns: rental_date redundant_index_non_unique: 1 dominant_index_name: rental_date dominant_index_columns: rental_date, inventory_id, customer_id dominant_index_non_unique: 0 sql_drop_index: ALTER TABLE `sakila`.`rental` DROP INDEX `rental_date_2`
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Views: sql_range_partitions  Analyze your range partitioned tables. Offers the ADD/REORGANIZE PARTITION statements to roll partitions: mysql> create table test.report … partition by range (…) ; mysql> select * from sql_range_partitions where table_name='report' G table_schema: test table_name: report count_partitions: 7 sql_drop_first_partition: alter table `test`.`report` drop partition `p0` sql_add_next_partition: alter table `test`.`report` reorganize partition `p6` into ( partition `p_20090701000000` values less than (1246395600) /* 2009-07-01 00:00:00 */ , partition p_maxvalue values less than MAXVALUE )
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Views: sql grants  Provide SQL based access to user accounts and their grants.  Offers the relevant GRANT, REVOKE and DROP statements for such grants: GRANTEE: 'world_user'@'localhost' user: world_user host: localhost priv_level: `world`.* priv_level_name: schema current_privileges: INSERT, SELECT, UPDATE sql_grant: GRANT INSERT, SELECT, UPDATE ON `world`.* TO 'world_user'@'%' sql_revoke: REVOKE INSERT, SELECT, UPDATE ON `world`.* FROM 'world_user'@'%'
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach SQL generation & eval()  Views above present with “SQL columns”, offering a statement to execute.  This is at the heart of common_schema's views, and is part of the server-side mechanism the framework strongly supports.  The eval() routine accepts such SQL columns and executes (evaluates) them.
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach eval()  Accepts a query returning a text column. Column data is expected to be SQL statements to be executed. mysql> call eval(" SELECT sql_add_next_partition FROM sql_range_partitions WHERE table_schema='webprod'"); Query OK, 0 rows affected -- A new partition has just been added on -- all range-partitioned tables in `webprod`
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Views: innodb_transactions  Which transactions are running? – For how long? – How long are they being idle? Locked? – What queries are they issuing?  Recommend KILL, apply with eval() mysql> call eval(" SELECT sql_kill_query FROM innodb_transactions WHERE trx_idle_seconds >= 30 "); -- All idle transactions have just been killed
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Views: others  processlist_grantees: identifies the user connection with its associated account (missing info in MySQL)  auto_increment_columns: find “free space” or “usage” for AUTO_INCREMENT values  slave_status: get Seconds_behind_master via real SQL query  last_query_profiling :aggregated profile for last executed query
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Views: processlist_grantees mysql> select * from processlist_granteesG ID: 41165491 USER: dbuser HOST: web00.myapp:40049 DB: profile COMMAND: Query TIME: 0 STATE: updating INFO: DELETE FROM locks WHERE id = 'helper' AND dt < '2013-04-17 15:09:50' GRANTEE: 'dbuser'@'%.myapp' grantee_user: dbuser grantee_host: %.myapp is_super: 0 is_repl: 0 is_current: 0 sql_kill_query: KILL QUERY 41165491 sql_kill_connection: KILL 41165491
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Views: others  sql_accounts: block/release accounts  similar_grants: detect security roles  global_status_diff_nonzero: detect status change  innodb_locked_transactions: who locks who, on which query? Recommend KILL  more... http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/process_views.html http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/innodb_plugin_views.html http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/security_views.html http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/schema_analysis_views.html
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Routines  common_schema offers more than 60 useful stored routines, part of the framework's function library.  From text parsing & manipulation, through process diagnostics, query analysis & dynamic execution, to security routines, the function library extends and complements MySQL's own functions. http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/execution_routines.html http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/text_routines.html http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/security_routines.html http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/process_routines.html
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Routines: security  killall() kills connections by matching text with user, host or grantee.  security_audit() audits server's privileges tables and configuration to detect such threats as empty or duplicate passwords, excessive privileges, permissive hosts etc.  duplicate_grantee(): duplicates an account mysql> call killall('analytics'); mysql> call killall('localhost');
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Routines: text  extract_json_value(): Extract value from JSON notation via Xpath.  replace_sections(): Search and replace text appearing between section.  get_num_tokens(): Return number of tokens in delimited text. mysql> select common_schema.get_num_tokens( 'the quick brown fox', ' '); => 4 mysql> select common_schema.split_token( 'the quick brown fox', ' ', 3); => 'brown'
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Routines: text mysql> set @json := '{ "menu": { "id": "file", "value": "File", "popup": { "menuitem": [ {"value": "New", "onclick": "create()"}, {"value": "Open", "onclick": "open()"}, {"value": "Close", "onclick": "close()"} ] } } }'; mysql> select extract_json_value(@json, '//id') AS result; +--------+ | result | +--------+ | file | +--------+
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Routines: general  query_checksum(): Checksum the result set of a query.  crc64(): Return a 64 bit CRC of given input, as unsigned big integer.  random_hash(): Return a 64 bit CRC of given input, as unsigned big integer. mysql> call query_checksum( 'select code, name from world.Country order by code'); +----------------------------------+ | checksum | +----------------------------------+ | 79221e8d040e33cd3262f1680b4c8e54 | +----------------------------------+
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Routines: SQL & execution  eval() evaluates the queries generated by a given query.  exec(), exec_file() dynamically executes a given query or semicolon delimited list of queries.  run(), run_file() execute QueryScript code. mysql> call exec(' CREATE TABLE test.t(id INT); INSERT INTO test.t VALUES (2),(3),(5); ');
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach QueryScript  A SQL oriented scripting language, offering tight integration with SQL commands, easy and familiar control flow syntax and high level abstraction of complex tasks.  common_schema implements QueryScript via interpreter, based on stored routines.  This makes QueryScript suitable for administration and bulk tasks, not for OLTP tasks. http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/query_script.html
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Why QueryScript?  Stored routine programming is a pain: – Requires one to actually store the routine within the schema: can't just run something. – Syntax is cumbersome (ANSI:SQL). – Does not offer deeper insight into MySQL's limitations and bottlenecks. – Does not provide with syntax for oh-so- common tasks – Verbose. Can't see the forest for the trees.
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Compare: stored routine DELIMITER $$ DROP PROCEDURE IF EXISTS some_proc $$ CREATE PROCEDURE some_proc() READS SQL DATA SQL SECURITY INVOKER begin declare some_id bigint unsigned default null; declare done tinyint default 0; declare my_cursor cursor for SELECT some_id FROM some_table; declare continue handler for NOT FOUND set done = 1; open my_cursor; read_loop: loop fetch my_cursor into some_id; if done then leave read_loop; end if; -- do something with some_id end loop; close my_cursor; end $$ DELIMITER ;
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Compare: QueryScript set @script := ' foreach ($some_id: select some_id from some_table) { -- do something with $some_id } '; call run(@script);  Significantly less overhead  Familiar C-family syntax  No need to store the code in schema  Can execute script directly from file call run('/path/to/script.qs');
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach QueryScript: variables  Create variables which are known to initialize as NULL and are known to clean up as they exit scope.  Can be expanded and used where MySQL does not allow variables. var $count := 20; var $pop := 1000000; var $tbl := 'City'; select * from world.:${tbl} where population >= $pop limit :${count};
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach QueryScript: conditions  Familiar if, while or loop-while statements.  But conditions are also tightly integrated with SQL, and so queries make for valid conditions. if (@val > 3) { pass; } while (delete from world.Country where Continent = 'Asia' limit 10) { throttle 2; }
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach QueryScript: foreach  A sophisticated looping device, allowing iteration over queries, sets, numbers range, databases, tables...  But also acknowledges MySQL limitations and provides with a safer access method.  Table iteration uses INFORMATION_SCHEMA optimizations to avoid excessive locks: foreach($tbl, $scm: table like wp_posts) alter table :${scm}.:${tbl} add column post_geo_location VARCHAR(128);
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach QueryScript: foreach foreach($year: 2001:2009) delete from sakila.rental where rental_date >= CONCAT($year, '-07-01') and rental_date < CONCAT($year, '-09-01'); foreach($shard: {US, GB, Japan, FRA}) create database dbshard_:${shard}; foreach($scm: schema like wp%) { create table :$scm.wp_likes( id int, data VARCHAR(128)); } foreach ($some_id: select some_id from some_table) {... }
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach QueryScript: split  Automagically breaks a bulk operation into smaller chunks.  Supports DELETE, UPDATE, INSERT...SELECT, REPLACE...SELECT  Supports single and multi table statements split (delete from sakila.rental where rental_date < NOW() - interval 5 year); split (insert into world.City_duplicate select * from world.City) { throttle 1; }
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach QueryScript: split split(update sakila.film_actor set last_update=now() where actor_id != 4) { select $split_columns as columns, $split_range_start as range_start, $split_range_end as range_end, $split_total_rowcount as total; } +----------------------+-------------+------------+-------+ | columns | range_start | range_end | total | +----------------------+-------------+------------+-------+ | `actor_id`,`film_id` | '1','1' | '39','293' | 978 | +----------------------+-------------+------------+-------+ +----------------------+-------------+------------+-------+ | columns | range_start | range_end | total | +----------------------+-------------+------------+-------+ | `actor_id`,`film_id` | '39','293' | '76','234' | 1978 | +----------------------+-------------+------------+-------+ +----------------------+-------------+-------------+-------+ | columns | range_start | range_end | total | +----------------------+-------------+-------------+-------+ | `actor_id`,`film_id` | '76','234' | '110','513' | 2978 | +----------------------+-------------+-------------+-------+
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach QueryScript: example  Create shards & tables, and copy row data: -- iterate shards foreach ($shard: {US, GB, Japan, FRA}) { create database dbshard_:${shard}; -- iterate tables foreach ($tbl: table in original_shard) { create table dbshard_:${shard}.:${tbl} like original_shard.:$tbl; -- copy table data split (insert into dbshard_:${shard}.:${tbl} select * from original_shard.:$tbl) { -- show progress select $shard, $tbl, $split_total_rowcount; throttle 1; } } }
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach QueryScript: more goodies  throttle statement controls script execution time, reduces server load  Local variables auto-cleanup, can be used (expanded) in statements where variables not allowed (table names, LIMIT value etc.)  try-catch statement is available: easy error handling mechanism  echo, eval, throw statements make for easy development
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach rdebug  common_schema introduces rdebug: debugger and debugging API for MySQL stored routines.  A server side solution, based in itself on stored routines.  Provides routine API calls to manage debugging sessions of other routines.  At this time (April 2013) in alpha stage http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/rdebug.html
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach rdebug  Uses code injection; one must “compile” her routine with/out debug code. – Does not emulate code; retains logic of one's routines, with some limitations.  Supports: – Step into/over/out – Watch variables, modify variables – Set breakpoint, run to breakpoint – Watch stack state, statements
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach rdebug  common_schema provides with stored routine API. – One may debug on command line using mysql client – Or one may wrap with GUI tool to call upon routine API  Awaiting GUI client implementations!
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach rdebug mysql [debugger]> call rdebug_step_into(); +-------------+----------------+---------------+--------------+---------------------+ | stack_level | routine_schema | routine_name | statement_id | entry_time | +-------------+----------------+---------------+--------------+---------------------+ | 1 | test | review_colors | 145 | 2013-04-08 15:41:28 | +-------------+----------------+---------------+--------------+---------------------+ +----------------+---------------+---------------+---------------+----------------+ | routine_schema | routine_name | variable_name | variable_type | variable_value | +----------------+---------------+---------------+---------------+----------------+ | test | review_colors | current_color | local | white | | test | review_colors | current_count | local | 10 | | test | review_colors | done | local | 0 | +----------------+---------------+---------------+---------------+----------------+ +----------------+---------------+--------------+-----------------------------------------+ | routine_schema | routine_name | statement_id | statement | +----------------+---------------+--------------+-----------------------------------------+ | test | review_colors | 145 | call review_single_color(current_color) | +----------------+---------------+--------------+-----------------------------------------+ mysql [debugger]> call rdebug_set_variable('current_color', 'orange'); mysql [debugger]> call rdebug_step_over();
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach call for help()  common_schema documentation is available in these formats: – Online documentation (directly in code repository) – Bundled HTML download – Inline: Just call for help(): call help('split'); +--------------------------------------------------------------------------------+ | help | +--------------------------------------------------------------------------------+ | QueryScript Flow Control: split statement | | | | SYNOPSIS | | | | Single table operations, autodetect mode:
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Support common_schema  Download and try it out  Report issues http://code.google.com/p/common-schema/issues/list  Happy to receive ideas and contributions http://bit.ly/UPC3vh http://code.google.com/p/common-schema/issues/entry? template=Request%20for%20new%20component  Above all else: spread the word!
D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Thank you!  Visit http://openark.org for news & updates.  Other open source projects: – openark kit http://openarkkit.googlecode.com/svn/trunk/openarkkit/doc/html/introduction.html – mycheckpoint http://code.openark.org/forge/mycheckpoint

More Related Content

PDF
openark-kit: MySQL utilities for everyday use
Shlomi Noach
 
PDF
Programmatic queries: things you can code with sql
Shlomi Noach
 
PDF
MySQL JSON Functions
Sveta Smirnova
 
PDF
MySQL Utilities -- PyTexas 2015
Dave Stokes
 
PDF
PNWPHP -- What are Databases so &#%-ing Difficult
Dave Stokes
 
PDF
MySQL's NoSQL -- Texas Linuxfest August 22nd 2015
Dave Stokes
 
PDF
Ohio Linux Fest -- MySQL's NoSQL
Dave Stokes
 
PDF
OpenWorld 2014 - Schema Management: versioning and automation with Puppet and...
Frederic Descamps
 
openark-kit: MySQL utilities for everyday use
Shlomi Noach
 
Programmatic queries: things you can code with sql
Shlomi Noach
 
MySQL JSON Functions
Sveta Smirnova
 
MySQL Utilities -- PyTexas 2015
Dave Stokes
 
PNWPHP -- What are Databases so &#%-ing Difficult
Dave Stokes
 
MySQL's NoSQL -- Texas Linuxfest August 22nd 2015
Dave Stokes
 
Ohio Linux Fest -- MySQL's NoSQL
Dave Stokes
 
OpenWorld 2014 - Schema Management: versioning and automation with Puppet and...
Frederic Descamps
 

What's hot (20)

PDF
Second Step to the NoSQL Side: MySQL JSON Functions
Sveta Smirnova
 
PDF
Php &amp; my sql - how do pdo, mysq-li, and x devapi do what they do
Dave Stokes
 
PDF
Undelete (and more) rows from the binary log
Frederic Descamps
 
PPTX
Database 12c is ready for you... Are you ready for 12c?
Performance Tuning Corporation
 
PDF
MySQL Webinar 2/4 Performance tuning, hardware, optimisation
Mark Swarbrick
 
PPTX
PHP, The X DevAPI, and the MySQL Document Store Presented January 23rd, 20...
Dave Stokes
 
PPTX
Flex Cluster e Flex ASM - GUOB Tech Day - OTN TOUR LA Brazil 2014
Alex Zaballa
 
PDF
PHP Oracle Web Applications by Kuassi Mensah
PHP Barcelona Conference
 
PDF
Summary tables with flexviews
Justin Swanhart
 
PDF
Open Source World June '21 -- JSON Within a Relational Database
Dave Stokes
 
PDF
Mysql tech day_paris_ps_and_sys
Mark Leith
 
PDF
MySQL Group Replication
Kenny Gryp
 
PDF
MySQL Group Replication - an Overview
Matt Lord
 
PPTX
Oracle Unified Directory. Lessons learnt. Is it ready for a move from OID? (O...
Andrejs Prokopjevs
 
PDF
MySQL Replication Update - DEbconf 2020 presentation
Dave Stokes
 
PPSX
Oracle Cloud As Services
Özgür Umut Vurgun
 
PDF
My sql 5.6&MySQL Cluster 7.3
Oleksii(Alexey) Porytskyi
 
PDF
Scaling MySQl 1 to N Servers -- Los Angelese MySQL User Group Feb 2014
Dave Stokes
 
PDF
Manual Tecnico OGG Oracle to MySQL
Erick Vidbaz
 
ODP
MySQL Monitoring Mechanisms
Mark Leith
 
Second Step to the NoSQL Side: MySQL JSON Functions
Sveta Smirnova
 
Php &amp; my sql - how do pdo, mysq-li, and x devapi do what they do
Dave Stokes
 
Undelete (and more) rows from the binary log
Frederic Descamps
 
Database 12c is ready for you... Are you ready for 12c?
Performance Tuning Corporation
 
MySQL Webinar 2/4 Performance tuning, hardware, optimisation
Mark Swarbrick
 
PHP, The X DevAPI, and the MySQL Document Store Presented January 23rd, 20...
Dave Stokes
 
Flex Cluster e Flex ASM - GUOB Tech Day - OTN TOUR LA Brazil 2014
Alex Zaballa
 
PHP Oracle Web Applications by Kuassi Mensah
PHP Barcelona Conference
 
Summary tables with flexviews
Justin Swanhart
 
Open Source World June '21 -- JSON Within a Relational Database
Dave Stokes
 
Mysql tech day_paris_ps_and_sys
Mark Leith
 
MySQL Group Replication
Kenny Gryp
 
MySQL Group Replication - an Overview
Matt Lord
 
Oracle Unified Directory. Lessons learnt. Is it ready for a move from OID? (O...
Andrejs Prokopjevs
 
MySQL Replication Update - DEbconf 2020 presentation
Dave Stokes
 
Oracle Cloud As Services
Özgür Umut Vurgun
 
My sql 5.6&MySQL Cluster 7.3
Oleksii(Alexey) Porytskyi
 
Scaling MySQl 1 to N Servers -- Los Angelese MySQL User Group Feb 2014
Dave Stokes
 
Manual Tecnico OGG Oracle to MySQL
Erick Vidbaz
 
MySQL Monitoring Mechanisms
Mark Leith
 
Ad

Similar to common_schema 2.0: DBA's Framework for MySQL (20)

PDF
common_schema 2.2: DBA's framework for MySQL (April 2014)
Shlomi Noach
 
PDF
common_schema, DBA's framework for MySQL
Shlomi Noach
 
ODP
Common schema my sql uc 2012
Roland Bouman
 
ODP
Common schema my sql uc 2012
Roland Bouman
 
PDF
RivieraJUG - MySQL 8.0 - What's new for developers.pdf
Frederic Descamps
 
PDF
MySQL 8.0 New Features -- September 27th presentation for Open Source Summit
Dave Stokes
 
PPTX
Linuxfest Northwest 2022 - MySQL 8.0 Nre Features
Dave Stokes
 
PDF
MySQL Goes to 8! FOSDEM 2020 Database Track, January 2nd, 2020
Geir Høydalsvik
 
PDF
The MySQL SYS Schema
Mark Leith
 
PPTX
Case study on mysql in rdbms
RajalakshmiK19
 
PDF
Mysql Datadictionary
rootuser
 
PDF
Copy Of Mysql Datadictionary
Golak Sarangi
 
PPTX
DATA BASE || INTRODUCTION OF DATABASE \\ SQL 2018
teachersduniya.com
 
PPTX
MySQL 8.0 Released Update
Keith Hollman
 
PDF
My sql 5.7-upcoming-changes-v2
Morgan Tocker
 
PPT
Mysql Fun
SHC
 
PPTX
cPanel now supports MySQL 8.0 - My Top Seven Features
Dave Stokes
 
PDF
UAE MySQL Users Group Meet-up : MySQL Shell Document Store & more...
Frederic Descamps
 
PDF
How MySQL can boost (or kill) your application
Federico Razzoli
 
PDF
What's New in MariaDB Server 10.2 and MariaDB MaxScale 2.1
MariaDB plc
 
common_schema 2.2: DBA's framework for MySQL (April 2014)
Shlomi Noach
 
common_schema, DBA's framework for MySQL
Shlomi Noach
 
Common schema my sql uc 2012
Roland Bouman
 
Common schema my sql uc 2012
Roland Bouman
 
RivieraJUG - MySQL 8.0 - What's new for developers.pdf
Frederic Descamps
 
MySQL 8.0 New Features -- September 27th presentation for Open Source Summit
Dave Stokes
 
Linuxfest Northwest 2022 - MySQL 8.0 Nre Features
Dave Stokes
 
MySQL Goes to 8! FOSDEM 2020 Database Track, January 2nd, 2020
Geir Høydalsvik
 
The MySQL SYS Schema
Mark Leith
 
Case study on mysql in rdbms
RajalakshmiK19
 
Mysql Datadictionary
rootuser
 
Copy Of Mysql Datadictionary
Golak Sarangi
 
DATA BASE || INTRODUCTION OF DATABASE \\ SQL 2018
teachersduniya.com
 
MySQL 8.0 Released Update
Keith Hollman
 
My sql 5.7-upcoming-changes-v2
Morgan Tocker
 
Mysql Fun
SHC
 
cPanel now supports MySQL 8.0 - My Top Seven Features
Dave Stokes
 
UAE MySQL Users Group Meet-up : MySQL Shell Document Store & more...
Frederic Descamps
 
How MySQL can boost (or kill) your application
Federico Razzoli
 
What's New in MariaDB Server 10.2 and MariaDB MaxScale 2.1
MariaDB plc
 
Ad

More from Shlomi Noach (6)

PDF
Un-split brain MySQL
Shlomi Noach
 
PDF
Managing and Visualizing your Replication Topologies with Orchestrator
Shlomi Noach
 
PDF
Pseudo GTID and Easy MySQL Replication Topology Management
Shlomi Noach
 
PDF
Pseudo gtid & easy replication topology management
Shlomi Noach
 
PDF
MySQL DevOps at Outbrain
Shlomi Noach
 
PDF
What's new in MySQL 5.6
Shlomi Noach
 
Un-split brain MySQL
Shlomi Noach
 
Managing and Visualizing your Replication Topologies with Orchestrator
Shlomi Noach
 
Pseudo GTID and Easy MySQL Replication Topology Management
Shlomi Noach
 
Pseudo gtid & easy replication topology management
Shlomi Noach
 
MySQL DevOps at Outbrain
Shlomi Noach
 
What's new in MySQL 5.6
Shlomi Noach
 

Recently uploaded (20)

PPTX
Celebrate Independence Day in Premium Apartments in Kerala
Kent Constructions Pvt Ltd
 
PDF
Under Construction Projects in Mumbai: A Smart Investment for Future Living
bhoomisharma21147
 
PPTX
Introduction_to_Property_Management.pptx
iamysenpai
 
PDF
Chelsea Residences 1 at Dubai Maritime City – DAMAC Properties
Miva Realestate
 
PDF
401 Contracting Company for Construction
401 Contracting
 
PDF
BAYZ 102 at Business Bay, Dubai – Danube Properties
Miva Realestate
 
PDF
Interstellar Tower at JVT, Dubai – Mr Eight Development
Miva Realestate
 
PPTX
Tangled Up in Green Luxury Developments
vaswanigroup18
 
PPTX
Pennsylvania Association of REALTORS Standard Forms Update - August 2025
Tom Blefko
 
PDF
Why DJ Thielen is the #1 Choice for Real Estate Investing
DJ Thielen
 
PDF
Top 5 Gemba Walks Apps with eAuditor Audits & Inspections
eAuditor Audits & Inspections
 
PDF
Luxera Bahceport Project - Listing Turkey
Listing Turkey
 
PDF
Binghatti Haven at Dubai Sports City -
Miva Realestate
 
PDF
Evolving Illinois Rental Market: Trends and Opportunities for Landlords
Hirsh Mohindra
 
PDF
Lesson 9 _ the role of plantrdetrtbtvrcterxwctvyrbutynibvrtcewrxctv and anim...
glennpauloani
 
PPTX
Estate_Agent_Power_Presentation_Branding.pptx
Estate Agent Power
 
PDF
Real Estate in Pune – A Hotspot for Residential & Commercial Investment in India
bhoomisharma21147
 
PDF
M3M Infinum - Prime Business Hub in Noida
harshu23227
 
PDF
Expert Advice on Property Development Finance for 2025.pdf
Silver Oak Capital
 
PDF
Bhartiya City Nikoo Homes 6 & 7 - Iconic Luxury Apartments in the Heart of No...
JagadishKR1
 
Celebrate Independence Day in Premium Apartments in Kerala
Kent Constructions Pvt Ltd
 
Under Construction Projects in Mumbai: A Smart Investment for Future Living
bhoomisharma21147
 
Introduction_to_Property_Management.pptx
iamysenpai
 
Chelsea Residences 1 at Dubai Maritime City – DAMAC Properties
Miva Realestate
 
401 Contracting Company for Construction
401 Contracting
 
BAYZ 102 at Business Bay, Dubai – Danube Properties
Miva Realestate
 
Interstellar Tower at JVT, Dubai – Mr Eight Development
Miva Realestate
 
Tangled Up in Green Luxury Developments
vaswanigroup18
 
Pennsylvania Association of REALTORS Standard Forms Update - August 2025
Tom Blefko
 
Why DJ Thielen is the #1 Choice for Real Estate Investing
DJ Thielen
 
Top 5 Gemba Walks Apps with eAuditor Audits & Inspections
eAuditor Audits & Inspections
 
Luxera Bahceport Project - Listing Turkey
Listing Turkey
 
Binghatti Haven at Dubai Sports City -
Miva Realestate
 
Evolving Illinois Rental Market: Trends and Opportunities for Landlords
Hirsh Mohindra
 
Lesson 9 _ the role of plantrdetrtbtvrcterxwctvyrbutynibvrtcewrxctv and anim...
glennpauloani
 
Estate_Agent_Power_Presentation_Branding.pptx
Estate Agent Power
 
Real Estate in Pune – A Hotspot for Residential & Commercial Investment in India
bhoomisharma21147
 
M3M Infinum - Prime Business Hub in Noida
harshu23227
 
Expert Advice on Property Development Finance for 2025.pdf
Silver Oak Capital
 
Bhartiya City Nikoo Homes 6 & 7 - Iconic Luxury Apartments in the Heart of No...
JagadishKR1
 

common_schema 2.0: DBA's Framework for MySQL

  • 1. common_schema2.0 D B A ' s f r a m e w o r k f o r M y S Q L S h l o m i N o a c h h t t p : / / o p e n a r k . o r g
  • 2. Copyright © 2013, Shlomi Noach common_schema D B A ' s f r a m e w o r k f o r M y S Q L common_schema2.0 D B A ' s f r a m e w o r k f o r M y S Q L  About  Views & eval()  Routines  QueryScript  Introducing rdebug
  • 3. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach About common_schema  common_schema is an open source project, licensed under the GPL License.  Authored by Shlomi Noach http://openark.org  Major contributions by Roland Bouman http://rpbouman.blogspot.com  Several contributions & suggestions by the community (Thanks!)
  • 4. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach About common_schema  common_schema is a framework which includes: – Views: analytics, security, action-taking, ... – Routines: text, temporal, process, security, ... – QueryScript: an SQL oriented scripting language & interpreter – rdebug: debugger and debugging API for MySQL stored routines (alpha).
  • 5. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach About common_schema  It is a schema that lies next to INFORMATION_SCHEMA.  It lies completely within the MySQL server, and does not require external packages or dependencies. No Perl scripts nor UDFs or plugins.
  • 6. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Getting & Installing  common_schema distribution is a SQL file.  Currently hosted on Google Code: http://code.google.com/p/common-schema/  Install by importing SQL file into MySQL: bash$ mysql < /tmp/common_schema-1.3.1.sql complete - Base components: installed - InnoDB Plugin components: installed - Percona Server components: not installed Installation complete. Thank you for using common_schema!
  • 7. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Views  Views providing non-trivial information about your table design, keys, grants, processes, transactions, locks, ...  While INFORMATION_SCHEMA provides with complete info, it is ofter difficult to aggregate. It is sometimes too normalized, and at other times too de-normalized.  Some metadata is simply not available in INFORMATION_SCHEMA.
  • 8. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Views  common_schema's views will present with valuable knowledge  And will typically offer recommendations or possible actions based on that knowledge.  common_schema also provides the mechanism to apply those recommendations.
  • 9. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Views: redundant_keys  Find duplicate/redundant keys in your schema. Recommend DROP statements. http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/redundant_keys.html – Similar to pt-duplicate-key-checker mysql> select * from redundant_keys where table_schema='sakila'G table_schema: sakila table_name: rental redundant_index_name: rental_date_2 redundant_index_columns: rental_date redundant_index_non_unique: 1 dominant_index_name: rental_date dominant_index_columns: rental_date, inventory_id, customer_id dominant_index_non_unique: 0 sql_drop_index: ALTER TABLE `sakila`.`rental` DROP INDEX `rental_date_2`
  • 10. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Views: sql_range_partitions  Analyze your range partitioned tables. Offers the ADD/REORGANIZE PARTITION statements to roll partitions: mysql> create table test.report … partition by range (…) ; mysql> select * from sql_range_partitions where table_name='report' G table_schema: test table_name: report count_partitions: 7 sql_drop_first_partition: alter table `test`.`report` drop partition `p0` sql_add_next_partition: alter table `test`.`report` reorganize partition `p6` into ( partition `p_20090701000000` values less than (1246395600) /* 2009-07-01 00:00:00 */ , partition p_maxvalue values less than MAXVALUE )
  • 11. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Views: sql grants  Provide SQL based access to user accounts and their grants.  Offers the relevant GRANT, REVOKE and DROP statements for such grants: GRANTEE: 'world_user'@'localhost' user: world_user host: localhost priv_level: `world`.* priv_level_name: schema current_privileges: INSERT, SELECT, UPDATE sql_grant: GRANT INSERT, SELECT, UPDATE ON `world`.* TO 'world_user'@'%' sql_revoke: REVOKE INSERT, SELECT, UPDATE ON `world`.* FROM 'world_user'@'%'
  • 12. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach SQL generation & eval()  Views above present with “SQL columns”, offering a statement to execute.  This is at the heart of common_schema's views, and is part of the server-side mechanism the framework strongly supports.  The eval() routine accepts such SQL columns and executes (evaluates) them.
  • 13. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach eval()  Accepts a query returning a text column. Column data is expected to be SQL statements to be executed. mysql> call eval(" SELECT sql_add_next_partition FROM sql_range_partitions WHERE table_schema='webprod'"); Query OK, 0 rows affected -- A new partition has just been added on -- all range-partitioned tables in `webprod`
  • 14. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Views: innodb_transactions  Which transactions are running? – For how long? – How long are they being idle? Locked? – What queries are they issuing?  Recommend KILL, apply with eval() mysql> call eval(" SELECT sql_kill_query FROM innodb_transactions WHERE trx_idle_seconds >= 30 "); -- All idle transactions have just been killed
  • 15. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Views: others  processlist_grantees: identifies the user connection with its associated account (missing info in MySQL)  auto_increment_columns: find “free space” or “usage” for AUTO_INCREMENT values  slave_status: get Seconds_behind_master via real SQL query  last_query_profiling :aggregated profile for last executed query
  • 16. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Views: processlist_grantees mysql> select * from processlist_granteesG ID: 41165491 USER: dbuser HOST: web00.myapp:40049 DB: profile COMMAND: Query TIME: 0 STATE: updating INFO: DELETE FROM locks WHERE id = 'helper' AND dt < '2013-04-17 15:09:50' GRANTEE: 'dbuser'@'%.myapp' grantee_user: dbuser grantee_host: %.myapp is_super: 0 is_repl: 0 is_current: 0 sql_kill_query: KILL QUERY 41165491 sql_kill_connection: KILL 41165491
  • 17. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Views: others  sql_accounts: block/release accounts  similar_grants: detect security roles  global_status_diff_nonzero: detect status change  innodb_locked_transactions: who locks who, on which query? Recommend KILL  more... http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/process_views.html http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/innodb_plugin_views.html http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/security_views.html http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/schema_analysis_views.html
  • 18. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Routines  common_schema offers more than 60 useful stored routines, part of the framework's function library.  From text parsing & manipulation, through process diagnostics, query analysis & dynamic execution, to security routines, the function library extends and complements MySQL's own functions. http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/execution_routines.html http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/text_routines.html http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/security_routines.html http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/process_routines.html
  • 19. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Routines: security  killall() kills connections by matching text with user, host or grantee.  security_audit() audits server's privileges tables and configuration to detect such threats as empty or duplicate passwords, excessive privileges, permissive hosts etc.  duplicate_grantee(): duplicates an account mysql> call killall('analytics'); mysql> call killall('localhost');
  • 20. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Routines: text  extract_json_value(): Extract value from JSON notation via Xpath.  replace_sections(): Search and replace text appearing between section.  get_num_tokens(): Return number of tokens in delimited text. mysql> select common_schema.get_num_tokens( 'the quick brown fox', ' '); => 4 mysql> select common_schema.split_token( 'the quick brown fox', ' ', 3); => 'brown'
  • 21. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Routines: text mysql> set @json := '{ "menu": { "id": "file", "value": "File", "popup": { "menuitem": [ {"value": "New", "onclick": "create()"}, {"value": "Open", "onclick": "open()"}, {"value": "Close", "onclick": "close()"} ] } } }'; mysql> select extract_json_value(@json, '//id') AS result; +--------+ | result | +--------+ | file | +--------+
  • 22. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Routines: general  query_checksum(): Checksum the result set of a query.  crc64(): Return a 64 bit CRC of given input, as unsigned big integer.  random_hash(): Return a 64 bit CRC of given input, as unsigned big integer. mysql> call query_checksum( 'select code, name from world.Country order by code'); +----------------------------------+ | checksum | +----------------------------------+ | 79221e8d040e33cd3262f1680b4c8e54 | +----------------------------------+
  • 23. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Routines: SQL & execution  eval() evaluates the queries generated by a given query.  exec(), exec_file() dynamically executes a given query or semicolon delimited list of queries.  run(), run_file() execute QueryScript code. mysql> call exec(' CREATE TABLE test.t(id INT); INSERT INTO test.t VALUES (2),(3),(5); ');
  • 24. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach QueryScript  A SQL oriented scripting language, offering tight integration with SQL commands, easy and familiar control flow syntax and high level abstraction of complex tasks.  common_schema implements QueryScript via interpreter, based on stored routines.  This makes QueryScript suitable for administration and bulk tasks, not for OLTP tasks. http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/query_script.html
  • 25. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Why QueryScript?  Stored routine programming is a pain: – Requires one to actually store the routine within the schema: can't just run something. – Syntax is cumbersome (ANSI:SQL). – Does not offer deeper insight into MySQL's limitations and bottlenecks. – Does not provide with syntax for oh-so- common tasks – Verbose. Can't see the forest for the trees.
  • 26. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Compare: stored routine DELIMITER $$ DROP PROCEDURE IF EXISTS some_proc $$ CREATE PROCEDURE some_proc() READS SQL DATA SQL SECURITY INVOKER begin declare some_id bigint unsigned default null; declare done tinyint default 0; declare my_cursor cursor for SELECT some_id FROM some_table; declare continue handler for NOT FOUND set done = 1; open my_cursor; read_loop: loop fetch my_cursor into some_id; if done then leave read_loop; end if; -- do something with some_id end loop; close my_cursor; end $$ DELIMITER ;
  • 27. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Compare: QueryScript set @script := ' foreach ($some_id: select some_id from some_table) { -- do something with $some_id } '; call run(@script);  Significantly less overhead  Familiar C-family syntax  No need to store the code in schema  Can execute script directly from file call run('/path/to/script.qs');
  • 28. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach QueryScript: variables  Create variables which are known to initialize as NULL and are known to clean up as they exit scope.  Can be expanded and used where MySQL does not allow variables. var $count := 20; var $pop := 1000000; var $tbl := 'City'; select * from world.:${tbl} where population >= $pop limit :${count};
  • 29. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach QueryScript: conditions  Familiar if, while or loop-while statements.  But conditions are also tightly integrated with SQL, and so queries make for valid conditions. if (@val > 3) { pass; } while (delete from world.Country where Continent = 'Asia' limit 10) { throttle 2; }
  • 30. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach QueryScript: foreach  A sophisticated looping device, allowing iteration over queries, sets, numbers range, databases, tables...  But also acknowledges MySQL limitations and provides with a safer access method.  Table iteration uses INFORMATION_SCHEMA optimizations to avoid excessive locks: foreach($tbl, $scm: table like wp_posts) alter table :${scm}.:${tbl} add column post_geo_location VARCHAR(128);
  • 31. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach QueryScript: foreach foreach($year: 2001:2009) delete from sakila.rental where rental_date >= CONCAT($year, '-07-01') and rental_date < CONCAT($year, '-09-01'); foreach($shard: {US, GB, Japan, FRA}) create database dbshard_:${shard}; foreach($scm: schema like wp%) { create table :$scm.wp_likes( id int, data VARCHAR(128)); } foreach ($some_id: select some_id from some_table) {... }
  • 32. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach QueryScript: split  Automagically breaks a bulk operation into smaller chunks.  Supports DELETE, UPDATE, INSERT...SELECT, REPLACE...SELECT  Supports single and multi table statements split (delete from sakila.rental where rental_date < NOW() - interval 5 year); split (insert into world.City_duplicate select * from world.City) { throttle 1; }
  • 33. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach QueryScript: split split(update sakila.film_actor set last_update=now() where actor_id != 4) { select $split_columns as columns, $split_range_start as range_start, $split_range_end as range_end, $split_total_rowcount as total; } +----------------------+-------------+------------+-------+ | columns | range_start | range_end | total | +----------------------+-------------+------------+-------+ | `actor_id`,`film_id` | '1','1' | '39','293' | 978 | +----------------------+-------------+------------+-------+ +----------------------+-------------+------------+-------+ | columns | range_start | range_end | total | +----------------------+-------------+------------+-------+ | `actor_id`,`film_id` | '39','293' | '76','234' | 1978 | +----------------------+-------------+------------+-------+ +----------------------+-------------+-------------+-------+ | columns | range_start | range_end | total | +----------------------+-------------+-------------+-------+ | `actor_id`,`film_id` | '76','234' | '110','513' | 2978 | +----------------------+-------------+-------------+-------+
  • 34. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach QueryScript: example  Create shards & tables, and copy row data: -- iterate shards foreach ($shard: {US, GB, Japan, FRA}) { create database dbshard_:${shard}; -- iterate tables foreach ($tbl: table in original_shard) { create table dbshard_:${shard}.:${tbl} like original_shard.:$tbl; -- copy table data split (insert into dbshard_:${shard}.:${tbl} select * from original_shard.:$tbl) { -- show progress select $shard, $tbl, $split_total_rowcount; throttle 1; } } }
  • 35. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach QueryScript: more goodies  throttle statement controls script execution time, reduces server load  Local variables auto-cleanup, can be used (expanded) in statements where variables not allowed (table names, LIMIT value etc.)  try-catch statement is available: easy error handling mechanism  echo, eval, throw statements make for easy development
  • 36. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach rdebug  common_schema introduces rdebug: debugger and debugging API for MySQL stored routines.  A server side solution, based in itself on stored routines.  Provides routine API calls to manage debugging sessions of other routines.  At this time (April 2013) in alpha stage http://common-schema.googlecode.com/svn/trunk/common_schema/doc/html/rdebug.html
  • 37. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach rdebug  Uses code injection; one must “compile” her routine with/out debug code. – Does not emulate code; retains logic of one's routines, with some limitations.  Supports: – Step into/over/out – Watch variables, modify variables – Set breakpoint, run to breakpoint – Watch stack state, statements
  • 38. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach rdebug  common_schema provides with stored routine API. – One may debug on command line using mysql client – Or one may wrap with GUI tool to call upon routine API  Awaiting GUI client implementations!
  • 39. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach rdebug mysql [debugger]> call rdebug_step_into(); +-------------+----------------+---------------+--------------+---------------------+ | stack_level | routine_schema | routine_name | statement_id | entry_time | +-------------+----------------+---------------+--------------+---------------------+ | 1 | test | review_colors | 145 | 2013-04-08 15:41:28 | +-------------+----------------+---------------+--------------+---------------------+ +----------------+---------------+---------------+---------------+----------------+ | routine_schema | routine_name | variable_name | variable_type | variable_value | +----------------+---------------+---------------+---------------+----------------+ | test | review_colors | current_color | local | white | | test | review_colors | current_count | local | 10 | | test | review_colors | done | local | 0 | +----------------+---------------+---------------+---------------+----------------+ +----------------+---------------+--------------+-----------------------------------------+ | routine_schema | routine_name | statement_id | statement | +----------------+---------------+--------------+-----------------------------------------+ | test | review_colors | 145 | call review_single_color(current_color) | +----------------+---------------+--------------+-----------------------------------------+ mysql [debugger]> call rdebug_set_variable('current_color', 'orange'); mysql [debugger]> call rdebug_step_over();
  • 40. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach call for help()  common_schema documentation is available in these formats: – Online documentation (directly in code repository) – Bundled HTML download – Inline: Just call for help(): call help('split'); +--------------------------------------------------------------------------------+ | help | +--------------------------------------------------------------------------------+ | QueryScript Flow Control: split statement | | | | SYNOPSIS | | | | Single table operations, autodetect mode:
  • 41. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Support common_schema  Download and try it out  Report issues http://code.google.com/p/common-schema/issues/list  Happy to receive ideas and contributions http://bit.ly/UPC3vh http://code.google.com/p/common-schema/issues/entry? template=Request%20for%20new%20component  Above all else: spread the word!
  • 42. D B A ' s f r a m e w o r k f o r M y S Q L common_schema Copyright © 2012, Shlomi Noach Thank you!  Visit http://openark.org for news & updates.  Other open source projects: – openark kit http://openarkkit.googlecode.com/svn/trunk/openarkkit/doc/html/introduction.html – mycheckpoint http://code.openark.org/forge/mycheckpoint