SlideShare a Scribd company logo
#IDUG
DB2 Scripting Hacks
Roland Schock
ARS Computer und Consulting GmbH
Session Code: ED02
2014-09-12 | Platform: DB2 for LUW
#IDUG
This session
• Get started with DB2 command line, CLPplus and the system
commands
• Shells, platform differences, shell variables, proper quoting and
calling db2 properly
• Pipes and Redirection
• Exploit GNU text utilities to get things done easily.
• Samples and Goodies
2
#IDUG
DB2 LUW Panel, 16.10.2013
•Quote:
"The removal of Control Center
improved our command line skills"
3
#IDUG
Motivation
• Why do we develop scripts
• To make our life a bit easier.
• We are lazy! [And that‘s good in this case!]
• It is a sanity check for the DBA to incorporate scripts for daily checks and
regular tasks.
• We don‘t like to write tedious documentation.
• Graphical user interfaces are hard to automate.
• If we write our commands in a script file, we can reuse
(automation) and we also have documented our steps.
• Can also be used as training materials for a new team member.
4
#IDUG
Motivation (cont.)
• Perils of a consultant
• When you are called to customers to help; you don‘t want to tell them,
we need a particular tools suite that costs $$$ to be able to help
• We have to find ways work with the tools provided by db2
• Perhaps, you can convince the customer to use some free tools,
we can also leave behind without license fee issues
• All roads lead to Rome. ()
• TIMTOWTDI = "There is more than one way to do it"
So our examples might not be perfect… ;-)
• We can review some samples a bit further in the presentation
5
#IDUG
Motivation (cont.)
• Advantages of Command line tools
• Keep the results in text files, plus filter unwanted/unneeded parts from
the output
• System commands
• GNU text utilities
• We have three categories of command line utilities
• DB2 command line
• 'db2' as an interactive shell or call-by-call, e.g. db2 list applications
• DB2 system commands
• Like db2ilist, db2diag or db2pd
• DB2 graphical utilities
• db2top
6
#IDUG
•Part 1 – DB2 Tools
7
#IDUG
DB2 Command Line Processor (db2clp)
DB2 offers two ways to use the command line
• 'db2' as an interactive shell with a command line prompt
• Does not need the repetition of 'db2 ' on each line
• No interpretation of OS shell specific characters, like * or brackets
within the interactive shell
• End the DB2 shell with
'quit' or 'terminate'
• Options can be used
('-t' for termination
char ';')
8
db2inst1@opmdemo:~> db2
(c) Copyright IBM Corporation 1993,2007
Command Line Processor for DB2 Client 10.5.2
...
To exit db2 interactive mode, type QUIT at the command prompt. Outside
interactive mode, all commands must be prefixed with 'db2'.
To list the current command option settings, type LIST COMMAND OPTIONS.
For more detailed help, refer to the Online Reference Manual.
db2 =>
#IDUG
DB2 Command Line Processor (db2clp)
• Using 'db2 ' as a prefix to commands from the operating system
shell
• Starts a separate instance of the db2 process for each command
• Utilizes background process db2bp to keep the connection
to DB2 between invocations
• Allows I/O redirection of operating system shell with <, >, >>, |
• Returns error codes, if something goes wrong (see later slide)
9
#IDUG
DB2 back-end process (db2bp) and the UNIX shell
• db2bp is started in the background:
$ . /home/db2inst1/sqllib/db2profile No db2bp process has been created yet
$ db2 list db directory db2bp process now exists.
$ db2 connect to sample same db2bp process is re-used
$ db2 "select count(*) from staff" …and re-used again
$ db2 connect reset db2bp still exists after disconnecting.
$ db2 terminate Now db2bp is gone
• Every distinct shell or script that interacts with DB2 is assigned a
unique instance of db2bp.
• Multiple shells cannot share the same DB2 back-end process.
10
#IDUG
Did you notice the quotation marks?
• Watch out for shell specific characters!
The first two lines work, the last line breaks. Do You know why?
• $ db2 "select count(*) from staff"
• $ db2 select "count(*)" from staff
• $ db2 select count(*) from staff
11
#IDUG
Useful tricks with db2 command
• Trick: start 'db2 -t' to copy and paste db2 commands interactively
from other scripts
• Hint: Check for other useful parameters via 'db2 list command options'
• Put your favorites in an environment variable:
export DB2OPTIONS='+a -c +ec -o -p'
• Use ! to spawn any shell command from inside CLP
• Within the DB2 shell, you can start logging with
UPDATE COMMAND OPTIONS USING Z ON outfile.txt V ON
12
#IDUG
Useful tricks with db2 command (cont.)
• To diagnose code page issues:
• db2 -a connect to dbname
• The SQLCA contains useful information for code page problems
(See speaker notes for details)
• Use the return code of db2 for error handling in your scripts
• Code Description
0 DB2 command or SQL statement executed successfully
1 SELECT or FETCH statement returned no rows
2 DB2 command or SQL statement warning
4 DB2 command or SQL statement error
8 Command line processor system error
13
#IDUG
CLPPlus
• New since DB2 9.7
• Optional installation component of IBM Data Server Client
• It is not included in
• DB2 Runtime Client
• Data Server Driver Package
• DB2 Express-C
• Requires at least Java JRE 1.5.x
• Can connect to DB2 LUW, DB2 z/OS and Informix via JDBC
14
#IDUG
CLPPlus Start
• From Windows Startmenu: Start → IBM DB2 →
Command line tools → Command line Processor Plus
• On the command line: clpplus user@server:port/database
• Missing arguments will be
requested for connect
• Database does not need
to be cataloged
• Continuous improvements
in upcoming DB2 fixpacks
and versions
15
#IDUG
CLPPlus Commands
• Editor-Buffer can be used for most recently used commands
• Commands for formating the data and pagination
• Simple configuration commands
• …
16
#IDUG
CLPPlus Example
17
#IDUG
DB2 System Commands
• DB2 system commands to query the system
• Did you have a look in the manuals lately? ;-)
• Sometimes there‘s no other way to RTFM and it is usually
interesting/enlightening.
• I think it is like a cross word. Just look over it from time to time and
discover new features.
• Have you heard about?
• db2_local_ps, db2caem, db2ckbkp
• db2flsn, db2logsforrfwd, db2osconf
• db2tdbmgr, db2xprt, db2relocatedb
18
#IDUG
DB2 System Commands: db2pd
• db2pd
• Direct access to DB2 system memory (i.e. dirty uncommitted read ;-)
• Low level details, but very helpful for analysis
• Samples:
• db2pd -db SAMPLE -locks // check for locks
• db2pd -help | more
• db2pd -db SAMPLE -tcbstats // lists table control block statistics
• db2pd -edus interval=10 top=5 // top five CPU consumers of last 10 secs
• db2pd -alldbs -hadr | grep -e onnected -e artition -e "andby " -e "rimary "
// multiple dbs
19
#IDUG
DB2 System Commands: db2diag
• db2diag
• db2diag –H // displays last 30 minutes of the db2diag.log
• db2diag –A // archives the db2diag.log and creates a new one
• db2diag –g db=TEST // shows the entries only for the Database TEST
• db2diag -l severe -H 2d // last 2 days of severe errors
• db2diag -gvi msg:=DB2_MAX_INACT_STMTS | more
• db2diag -gi data:="password validation" -fmt "%{tsmonth}-%{tsday}-
%{tshour}.%{tsmin}.%{tssec}t%{database}t%{data}" | sed
"s/DATA.*bytes//;N;s/n//"
2013-07-22-15.51.33 SAMPLE Password validation for user janedoo failed with rc = -2146500507
2013-07-22-15.51.59 SAMPLE Password validation for user james007 failed with rc = -2146500502
20
#IDUG
DB2 System Commands: db2top
• Can be run interactive or in batch mode
• db2top [−d dbname] [−n nodename] [−u username] [−p password]
[−V schema] [−i interval] [−P <partition> ] [−b option] [−a] [−B] [−k] [−R] [−x] [−f file
</HH:MM:SS> <+offset> ] [−D delimiter] [−C <option>] [−m duration] [−o outfile]
• −b option, db2top will display information in CSV format.
• Sorry for Windows users, only for Linux and Unix
21
#IDUG
•Part 2 – The Shell
22
#IDUG
Different operating systems – different shells
• Platform differences
• Windows CMD, PowerShell, (Posix-Shell with R2)
• Linux: bash, csh, sh, pdksh, …
• AIX: ksh, bash, csh, …
• Solaris, HP-UX and other Unix dialects… ;-)
• Different shells, mostly similar
• See also http://en.wikipedia.org/wiki/Comparison_of_command_shells
23
#IDUG
Environment variables in Windows
• set DB2INSTANCE=db2
• echo %DB2Instance%
• Variables are not case sensitive
• In batch files use %% instead of %
because single % is for variables on console
• Example:
• for %i in (1 2 3) do echo %i
24
#IDUG
Environment variables in Unix
• export DB2INSTANCE=db2
• echo $DB2INSTANCE
• Shell is case sensitive about variable names
• Works in shell scripts and on command line
• Might need curly brackets to delimit the variable name:
• echo ${LOGDIR}/mylogfile.txt
• Example:
• for i in 1 2 3 ; do echo ${i} ; done
25
#IDUG
Shell command line expansion
• Be careful with your shell command line expansion!
If you do not quote/escape properly you will get strange results.
• db2 "SELECT * FROM employee"
• db2 SELECT * FROM employee
• An * in Unix gets replaced by all filenames in the current directory
if it isn't quoted or escaped.
• A tilde '~' will be replaced by your user home directory
• Alias is just plain text replacement without parameters
• alias ll='ls –alF'
26
#IDUG
Quoting in the command line
• Some characters are interpreted in the shell and may need quotes
• Watch out with *, ?, %, &, |, <, >, , ", !, ^, =, ~, $, , , {, }, (, ), ', `
• DB2 uses single quotes for Text ' '
• db2 UPDATE employee SET lastname='Smith' where ID='00210'
• Shell uses double quotes " ", but can also use single quotes
• db2 'select count(*) from employee'
• Brackets have different meaning in a Unix shell:
• tar cf - * | (mkdir /tmp/demo ; cd /tmp/demo ; tar xvf - )
27
#IDUG
Quoting in the command line
• Nesting can be tricky! Here we add quotes around our reorg command:
• Windows:
db2 "select 'db2 ""reorg table '||rtrim(tabschema)||'.'||rtrim(tabname)||' "" '
from syscat.tables where stats_time < current timestamp - 5 days"
• Unix:
db2 "select 'db2 " ' " ' "reorg table '||rtrim(tabschema)||'.'||rtrim(tabname)||'
" ' " ' " ' from syscat.tables where stats_time < current timestamp - 5 days"
• Or if you know the ASCII codes… ;-)
db2 "select 'db2 '||chr(34)||'reorg table
'||rtrim(tabschema)||'.'||rtrim(tabname)||chr(34) from syscat.tables where
stats_time < current timestamp - 5 days"
• All of the above commands can be directly used in your shell or in a script
28
#IDUG
Calling batch files
• Starting a batch file from a command line will pass an
environment to the batch
• Windows
• All environment variables are passed on
• The batch process can re-use the database connection
• Changes to variables remain. Otherwise use setlocal/endlocal
• Unix
• Exported variables are passed on. Variables defined with 'set' are left out.
• The batch file is started as a sub-process (fork) and needs it's own connection to DB2
• Remedy: Use sourcing and prepend the command with '. ' (dot and space) to let the
batch file run in the same shell instead of forking a new shell interpreter for the batch.
29
#IDUG
Calling batch files (cont.)
• Batch files can get input parameters from the command line
• Windows use them as %1, %2, etc.
• Windows Shell Extensions can use e.g. %~x1 to split filenames in
parameters
• Unix gets $1, $2, etc.; $* stands for all parameters
$? Is the return code of the previous command
• Some shells can split strings in parameters ${1#db2}
30
#IDUG
Pipes and I/O redirections
• Concatenation of commands
• Sending output from one command to some other tool via pipes
• db2 list applications | more
• Multiple pipes
• All commands in the pipe can run at the same time
• Check for DB2 msgs in loads:
cat LOAD*.msg | grep "^DB2" | sort | uniq –c
• Conditional execution
• db2start && db2 connect to SAMPLE
• Using return codes for error handling
31
#IDUG
Standard streams
• Standard streams stdin, stdout und stderr
• Well known by C/C++-Programmers ;-)
• The shell and command tools takes input from stdin (stream 0)
and sends output to stdout (stream 1) and error messages to
stderr (stream 2) .
• Combining stderr in stdout
• To redirect both stdout and stderr to the same file, use 2>&1
• E.g. useful for cronjobs, which send the output as mail
• stderr exists for clever programmers, who want to show error messages
even if the output is sent to a file
32
#IDUG
•Part 3 – GNU Utils
33
#IDUG
GNU Core Utilities – General Introduction
• What they are?
• Gnu's Not Unix, is the name for a complete Unix-compatible software
system, which is free
• Richard Stallman is the Guru/Father of the GNU Project.
• Where did they come from?
• By the 1980s, almost all software was proprietary. This made the GNU
Project necessary.
• By 1990 they either found or written all the major components except
one—the kernel. Then Linux, a Unix-like kernel, was developed by Linus
Torvalds in 1991 and made free software in 1992.
34
#IDUG
GNU Core Utilities – Where to get them?
• Typically part of a Linux distribution or part of development tools
• Install via YaST, yum, zypper or some other tool from your distribution
• Look for software development tools, if you can't find them directly
• GNU Core Utilities, available via http://www.gnu.org
• Direct link: http://www.gnu.org/software/coreutils/
• Typically as sources to compile on your own
• Also available as binaries for Windows and other platforms.
35
#IDUG
GNU Text Utilities
• Overview of GNU Core Utilities
• Utilities that are valuable when trying to match, replace and advance line
processing (grep, sed, awk, …)
• Which commands are useful
• cat, head, tail, more|less, cut
• wc, uniq, sort
• fgrep, grep
• diff, patch
• sed, awk, xargs
• Less used tools
• split, csplit, reverse
36
#IDUG
GNU Text Utilities – cat
• Copies file to console
• cat db2diag.log // show db2diag.log
• Can be used to put files in a pipe
• cat myfile.tgz | gzip -d | tar xvf - // unpack tgz file
• Can combine multiple files
• cat *.msg | grep "^DB2" | sort | uniq -c // look for messages
• Waits for input from console if no file given
• Copy SSH keys to other machine
• cat ~/.ssh/id_rsa.pub | ssh user@machine “mkdir ~/.ssh; cat >> ~/.ssh/authorized_keys”
37
#IDUG
GNU Text Utilities – head, tail
• Fetch first lines of a file or stream
• db2diag | head -17
• Returns the first or last lines from input
• tail -20 db2diag.log
• Find newest three files in directory
• ls -1tr | tail -3
• Follow the file
• tail -f ~/sqllib/db2dump/db2diag.log
• You can even "space" a bit by pressing Enter several times ;-)
38
#IDUG
GNU Text Utilities – more | less
• Simple pagination with Space (screen) and Return (single lines)
• On some systems navigation commands possible (Pos1, End, G, …)
• less is a powerful alternative to more (pun intended)
• Can go backwards (b), allows searching (/) and highlights found string
• Jump to end of current file (g)
• …
• Check out the man pages on your system
• man less
39
#IDUG
GNU Text Utilities – cut
• Returns parts of a text line (by char or field)
• Quick'n dirty way for number of connections to databases in instance
db2 list applications | cut -c67-75 | sort | uniq –c
• Used very often to cut off pieces in a sequence of chars
• Can work with a field separator (default=TAB)
• Get second and third column from CSV file:
cut -f2,3 -d, myfile.csv
40
#IDUG
GNU Text Utilities – wc
• Very useful to count output lines (-l) or maximum line lenght (-L)
number of chars (-c) or number of words (-w)
• Findout how many dbs are running as HADR Primary on this machine
db2pd -hadr -alldbs | grep -e 'rimary ' | wc –l
• Number of connections from certain IP range
db2 list applications | fgrep 172.25. | wc -l
41
#IDUG
GNU Text Utilities – sort
• Does not only sort text files line by line
• Can sort numerically
• Find top 10 largest files or directory in /var/log
du -sk /var/log/* | sort -r -n | head -10
• Can work on fields, field separator is transition between
whitespace and non-whitespace
• Some implementations have problems with multiple blanks
gawk "/----/{s=1;getline} //{if(s==1) print}" 2013-04-12-prof-db01.txt |
sed "s/ */ /" | gsort -k 2,2 -u | more
42
#IDUG
GNU Text Utilities – uniq
• Remove duplicate lines (may need a sort first ;-)
• Check for DB2 msgs in loads:
cat LOAD*.msg | grep "^DB2" | sort | uniq –c -d
• Sort is also capable to remove duplicates (sort –u), but uniq can
also count the occurences, print only the duplicates and work on
fields for comparison
• Check out uniq --help
43
#IDUG
GNU Text Utilities – grep, fgrep, egrep
• grep stands for Global Regular Expression Print
• fgrep = fast grep with simpler expressions, egrep = extended grep
• Regular expressions base on following principles
• Boolean 'or' with | Pickup|Ute
• Grouping with ( ) grey|gray is equal to gr(a|e)y
• Quantification with ?, * and +
? stands for zero or one colou?r matches color and colour
* stands for zero or more ab*c matches ac, abc, abbc, abbbc, …
+ stands for one and more ab+c matches abc, abbc, abbbc but not ac
• Metachars ., [], [^ ], ^, $, n, {m,n}, [:print:]
44
#IDUG
GNU Text Utilities – sed
• sed stands for stream editor
• cat myselectoutput.txt | sed "s/ *$//g" >removed_trailing_spaces.txt
• Complex sample: Format XML based logfile
grep -e "Time Millis" -e LogText trace.log | sed "s/<Time Millis.*">//;
s/</Time>//;N;s/n/ /; s/<LogText>//; s/<![CDATA[//;
s/]]></LogText>//"
45
<Trace Level="MIN">
<Time Millis="1387783880929"> 2013-12-23 08:31:20.929+01:00</Time>
<Server Format="IP">localhost</Server>
<ProductId>COD</ProductId>
<Component>Admin Server</Component>
<ProductInstance></ProductInstance>
<LogText><![CDATA[[LOG]: There is no agent installed on any of the host virtual machines. Skipping processing of the
host 335aba37-2a44-3dcd-9069-bca2fab11a79]]></LogText>
<Source FileName="com.ibm.license.mgmt.core.actions.vmman.algorithm.MergeHostStrategy" Method="process"/>
<Thread>MergeDataTask-Thread:0</Thread>
<Principal></Principal>
</Trace>
#IDUG
GNU Text Utilities – awk
• awk – Alfred Aho, Peter Weinberger and Brian Kernighan
• awk is a data-driven scripting language consisting of a set of
conditions and actions to be taken against streams of textual data
• /condition/ {actions}
• Special words BEGIN, END, FS, RS, …
• Can be used from 1liners to complex programs to manipulate text
• Nowadays sometimes superseeded by perl
• http://www.thegeekstuff.com/tag/awk-tutorial-examples/
46
#IDUG
GNU Text Utilities
• diff: Find differences in two text files
• patch: Insert changes in text files generated by diff
• xargs: builds and executes commands from standard input
• Less used tools
• split, csplit, reverse, …
47
#IDUG
•Part 4 – Samples
48
#IDUG
Utilities and db2 commands
• Samples of all things you can play around and customize to your
needs
• Quick way find out what DDL was executed on your DB
• db2 list history all for db MYPRODDB | grep DDL
• Check if your instance is running
• ps -ef | grep $DB2INSTANCE | grep db2sysc | grep -v grep | wc –l
• db2pd -agents | grep -e Partition -e Unable
• Quick and dirty number of connections to all databases in instance:
• db2 list applications | cut -c 99-106 | sort | uniq –c
49
#IDUG
Utilities and db2 commands (cont.)
• Find the busiest tables via command line or ksh
• db2pd -db dbname -tcbstats| awk '/TCB Table Stats/ { x =1} x==1 { print}'
| awk '/^0x/ { print $9, $2}' | sort –rn | head -15
• Get reorg recommendations and change them to inplace reorg
instead of offline reorg
• db2 get recommendations for health indicator db.tb_reorg_req for
database on d3t | grep "^REORG" |awk ' /REORG TABLE /{gsub(/;/, "
inplace ;" )}; 1'|awk '{gsub(/"RUNSTATS/,";nRUNSTATS"); print}' |
tee -a reorg.sql
50
#IDUG
Utilities and db2 commands (cont.)
• Generate script to re-catalog your databases
• db2 list db directory | 
egrep "Database alias|Node name" | 
sed -e 's/Database alias//g;s/Node name//g;s/ //g;s/=//g' | 
sed 'N;s/n/ : /' | 
awk '{print "db2 catalog db "$1" at node "$3;}'
51
#IDUG
Use SQL to generate stuff
• Extract statements from package cache as input for db2advis
• db2 -x "select '--#SET FREQUENCY
'||ltrim(char(num_executions))||‘@@‘||rtrim(STMT_TEXT)||';' from
table(snapshot_dyn_sql('PK2I11C1', -1)) as SNAP_STMT where
stmt_text is not null and num_executions > 100 order by
num_executions desc" | fgrep -iv "SYSIBM" | sed „s/ *$//“
>MostCommonQueries.txt
• Now replace @@ by CRLF and feed into db2advis or db2batch
• Checking integrity after foreign keys if needed
• db2 -xtd! "select 'SET INTEGRITY FOR
'||chr(34)||rtrim(tabschema)||chr(34)||'.'||chr(34)||rtrim(tabname
)||chr(34)||' IMMEDIATE CHECKED FORCE GENERATED' from
syscat.tables where tabschema not in ('SYSIBM','SYSCAT','SYSSTAT')
and STATUS='C' order by parents,children!" >checkref.sql
db2 -v -f checkref.sql
echo Checking integrity for constraints done.
52
#IDUG
How to get started
• Save output of DB2 utility to file, maybe shorten it as useful
• cat textfile and pipe through filters
• Start filtering out unwanted parts/lines
• Modify/cut resulting parts until satisfied
• Recombine pieces for the big result
• Always think about the complexity of multiple executions of the
filter programs
• Maybe save intermediate results to files
53
#IDUG
•Part 5 – Other goodies
54
#IDUG
KEY-BASED AUTHENTICATION USING SSH
• To get information in scripts from other machines you would need to
access them without explicitly typing in passwords.
• Key based authentication is situation of authentication takes place based
on the public/private information, rather than with the more usual
method of prompting for a password.
• This is very convenient if a non-interactive process is trying to
authenticate with a remote machine
• SSH is the preferred way of setting up such connections now as it uses
public and private key pairs to establish a secure connection.
• It allows you securely to start commands on another machine without
providing passwords!
55
#IDUG
Key-based authentication for SSH
• Arranging key-based authentication for SSH
• Machine "A" is the machine you want to connect from
• Machine "B" is the machine you want to connect to
• Setup – Part 1:
• On Machine "A", generate your key pair using
ssh-keygen -t dsa
• This will create two files:
• ~/.ssh/id_dsa
• ~/.ssh/id_dsa.pub
56
#IDUG
Key-based authentication for SSH (cont.)
• Setup – Part 2:
• Log onto Machine "B"
• Secure copy the id_dsa.pub key across to the local machine
• Now look at the file /~.ssh/id_dsa.pub, if it starts with "ssh-dss", ie
• ssh-dss
AAAAB3NzaC1kc3MAAAC+CLO2M9OfcIjEaFBJ+cNAubJeCw8dtlH
• Then append it to the file ./ssh/authorized_keys2 and ./ssh2/authorized_keys2
• cat id_dsa.pub >> ~/.ssh/authorized_keys2
• cat id_dsa.pub >> ~/.ssh2/authorized_keys2
• change permissions on files to user read/write only
• Test – now you will be able to ssh from Machine A to Machine B
(no password req.)
57
#IDUG
Backup/Restore via pipe
• Pipes are special file objects under Unix operating systems
• Create with mkfifo
• Can be used for a quick backup/restore from one machine to another or
between instances
• Machine 1: Machine 2:
mkfifo /tmp/mydest mkfifo /tmp/mysource
db2 restore db test_1 as test_new
from /tmp/mysource replace existing
without prompting
rsh host2 'cat > /tmp/mysource' < /tmp/mydest &
db2 backup db test_1 to /tmp/mydest
58
#IDUG
Useful links
• Introduction to text manipulation on UNIX-based systems
http://www.ibm.com/developerworks/aix/library/au-unixtext/
• Bash 1liners
http://www.bashoneliners.com/
• AWK Tutorial
http://www.thegeekstuff.com/tag/awk-tutorial-examples/
• Twitter: ARSDB2
59
#IDUG
Roland Schock
ARS Computer und Consulting GmbH
schock@ars.de
Session Code: ED02
DB2 Scripting Hacks
#IDUG
•Appendix
61
#IDUG
Windows 7 User Access Control
• With User Access Control in Windows 7
or Windows Server 2008, you have
to confirm the execution of
administrative commands.
•
• To start the instance via db2start you need admin rights
or you have to be instance owner. If the instance owner
is your domain user, it is getting tricky, if you are on the
road with your laptop. Runas doesn't help here!
62
#IDUG
Windows 7 User Access Control (cont.)
• If you use your domain account for DB2 installation and
administration, you get admin rights only as long as your laptop
is in the domain.
• You should better use a local user for install and administration
• db2set DB2_GRP_LOOKUP and SYSADM_GROUP could help
• Much easier is following trick:
• Start → IBM DB2 → Commandline tools →
Command window → Shift+Right-Click on the
needed entry and login with
MachineNameInstanceOwner
(if the instance owner is a local userid)
63

More Related Content

PPTX
Herd your chickens: Ansible for DB2 configuration management
PPTX
Hidden Gems of Performance Tuning: Hierarchical Profiler and DML Trigger Opti...
PDF
Hotsos Advanced Linux Tools
PPT
db2dart and inspect
PDF
DB2 and PHP in Depth on IBM i
PDF
3 boyd direct3_d12 (1)
PDF
Top Tips Every Notes Developer Needs To Know
Herd your chickens: Ansible for DB2 configuration management
Hidden Gems of Performance Tuning: Hierarchical Profiler and DML Trigger Opti...
Hotsos Advanced Linux Tools
db2dart and inspect
DB2 and PHP in Depth on IBM i
3 boyd direct3_d12 (1)
Top Tips Every Notes Developer Needs To Know

Similar to 2022-Scripting_Hacks.pdf (20)

PDF
BUD17-310: Introducing LLDB for linux on Arm and AArch64
PPT
DB2UDB_the_Basics Day 7
PDF
1049: Best and Worst Practices for Deploying IBM Connections - IBM Connect 2016
PDF
Toad tipstricksexpertinsight
PDF
Building DSLs On CLR and DLR (Microsoft.NET)
PPT
嵌入式Linux課程-GNU Toolchain
PPTX
Reversing with gdb
PPT
Mis4200notes8 2
ODP
Asian Spirit 3 Day Dba On Ubl
PDF
MySQL 5.7: Core Server Changes
PDF
Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics
KEY
Exciting JavaScript - Part II
PDF
Comandos linux bash, f2 linux pesquisa, http://f2linux.wordpress.com
PDF
Brad Wood - CommandBox CLI
PDF
Firebird
PDF
Docker HK Meetup - 201707
PDF
TechEvent 2019: Oracle to PostgreSQL - a Travel Guide from Practice; Roland S...
PPT
2 db2 instance creation
PPT
IBM DB2 LUW UDB DBA Training by www.etraining.guru
BUD17-310: Introducing LLDB for linux on Arm and AArch64
DB2UDB_the_Basics Day 7
1049: Best and Worst Practices for Deploying IBM Connections - IBM Connect 2016
Toad tipstricksexpertinsight
Building DSLs On CLR and DLR (Microsoft.NET)
嵌入式Linux課程-GNU Toolchain
Reversing with gdb
Mis4200notes8 2
Asian Spirit 3 Day Dba On Ubl
MySQL 5.7: Core Server Changes
Replicate from Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics
Exciting JavaScript - Part II
Comandos linux bash, f2 linux pesquisa, http://f2linux.wordpress.com
Brad Wood - CommandBox CLI
Firebird
Docker HK Meetup - 201707
TechEvent 2019: Oracle to PostgreSQL - a Travel Guide from Practice; Roland S...
2 db2 instance creation
IBM DB2 LUW UDB DBA Training by www.etraining.guru

Recently uploaded (20)

PPTX
Introduction to machine learning and Linear Models
PPTX
Market Analysis -202507- Wind-Solar+Hybrid+Street+Lights+for+the+North+Amer...
PPTX
DISORDERS OF THE LIVER, GALLBLADDER AND PANCREASE (1).pptx
PPTX
Introduction to Basics of Ethical Hacking and Penetration Testing -Unit No. 1...
PPTX
Data_Analytics_and_PowerBI_Presentation.pptx
PPTX
iec ppt-1 pptx icmr ppt on rehabilitation.pptx
PPT
ISS -ESG Data flows What is ESG and HowHow
PDF
BF and FI - Blockchain, fintech and Financial Innovation Lesson 2.pdf
PDF
Introduction to Data Science and Data Analysis
PPTX
oil_refinery_comprehensive_20250804084928 (1).pptx
PPTX
STUDY DESIGN details- Lt Col Maksud (21).pptx
PPTX
Database Infoormation System (DBIS).pptx
PPTX
IB Computer Science - Internal Assessment.pptx
PPTX
AI Strategy room jwfjksfksfjsjsjsjsjfsjfsj
PPT
Miokarditis (Inflamasi pada Otot Jantung)
PPTX
Computer network topology notes for revision
PPTX
Microsoft-Fabric-Unifying-Analytics-for-the-Modern-Enterprise Solution.pptx
PPTX
SAP 2 completion done . PRESENTATION.pptx
PDF
22.Patil - Early prediction of Alzheimer’s disease using convolutional neural...
Introduction to machine learning and Linear Models
Market Analysis -202507- Wind-Solar+Hybrid+Street+Lights+for+the+North+Amer...
DISORDERS OF THE LIVER, GALLBLADDER AND PANCREASE (1).pptx
Introduction to Basics of Ethical Hacking and Penetration Testing -Unit No. 1...
Data_Analytics_and_PowerBI_Presentation.pptx
iec ppt-1 pptx icmr ppt on rehabilitation.pptx
ISS -ESG Data flows What is ESG and HowHow
BF and FI - Blockchain, fintech and Financial Innovation Lesson 2.pdf
Introduction to Data Science and Data Analysis
oil_refinery_comprehensive_20250804084928 (1).pptx
STUDY DESIGN details- Lt Col Maksud (21).pptx
Database Infoormation System (DBIS).pptx
IB Computer Science - Internal Assessment.pptx
AI Strategy room jwfjksfksfjsjsjsjsjfsjfsj
Miokarditis (Inflamasi pada Otot Jantung)
Computer network topology notes for revision
Microsoft-Fabric-Unifying-Analytics-for-the-Modern-Enterprise Solution.pptx
SAP 2 completion done . PRESENTATION.pptx
22.Patil - Early prediction of Alzheimer’s disease using convolutional neural...

2022-Scripting_Hacks.pdf

  • 1. #IDUG DB2 Scripting Hacks Roland Schock ARS Computer und Consulting GmbH Session Code: ED02 2014-09-12 | Platform: DB2 for LUW
  • 2. #IDUG This session • Get started with DB2 command line, CLPplus and the system commands • Shells, platform differences, shell variables, proper quoting and calling db2 properly • Pipes and Redirection • Exploit GNU text utilities to get things done easily. • Samples and Goodies 2
  • 3. #IDUG DB2 LUW Panel, 16.10.2013 •Quote: "The removal of Control Center improved our command line skills" 3
  • 4. #IDUG Motivation • Why do we develop scripts • To make our life a bit easier. • We are lazy! [And that‘s good in this case!] • It is a sanity check for the DBA to incorporate scripts for daily checks and regular tasks. • We don‘t like to write tedious documentation. • Graphical user interfaces are hard to automate. • If we write our commands in a script file, we can reuse (automation) and we also have documented our steps. • Can also be used as training materials for a new team member. 4
  • 5. #IDUG Motivation (cont.) • Perils of a consultant • When you are called to customers to help; you don‘t want to tell them, we need a particular tools suite that costs $$$ to be able to help • We have to find ways work with the tools provided by db2 • Perhaps, you can convince the customer to use some free tools, we can also leave behind without license fee issues • All roads lead to Rome. () • TIMTOWTDI = "There is more than one way to do it" So our examples might not be perfect… ;-) • We can review some samples a bit further in the presentation 5
  • 6. #IDUG Motivation (cont.) • Advantages of Command line tools • Keep the results in text files, plus filter unwanted/unneeded parts from the output • System commands • GNU text utilities • We have three categories of command line utilities • DB2 command line • 'db2' as an interactive shell or call-by-call, e.g. db2 list applications • DB2 system commands • Like db2ilist, db2diag or db2pd • DB2 graphical utilities • db2top 6
  • 7. #IDUG •Part 1 – DB2 Tools 7
  • 8. #IDUG DB2 Command Line Processor (db2clp) DB2 offers two ways to use the command line • 'db2' as an interactive shell with a command line prompt • Does not need the repetition of 'db2 ' on each line • No interpretation of OS shell specific characters, like * or brackets within the interactive shell • End the DB2 shell with 'quit' or 'terminate' • Options can be used ('-t' for termination char ';') 8 db2inst1@opmdemo:~> db2 (c) Copyright IBM Corporation 1993,2007 Command Line Processor for DB2 Client 10.5.2 ... To exit db2 interactive mode, type QUIT at the command prompt. Outside interactive mode, all commands must be prefixed with 'db2'. To list the current command option settings, type LIST COMMAND OPTIONS. For more detailed help, refer to the Online Reference Manual. db2 =>
  • 9. #IDUG DB2 Command Line Processor (db2clp) • Using 'db2 ' as a prefix to commands from the operating system shell • Starts a separate instance of the db2 process for each command • Utilizes background process db2bp to keep the connection to DB2 between invocations • Allows I/O redirection of operating system shell with <, >, >>, | • Returns error codes, if something goes wrong (see later slide) 9
  • 10. #IDUG DB2 back-end process (db2bp) and the UNIX shell • db2bp is started in the background: $ . /home/db2inst1/sqllib/db2profile No db2bp process has been created yet $ db2 list db directory db2bp process now exists. $ db2 connect to sample same db2bp process is re-used $ db2 "select count(*) from staff" …and re-used again $ db2 connect reset db2bp still exists after disconnecting. $ db2 terminate Now db2bp is gone • Every distinct shell or script that interacts with DB2 is assigned a unique instance of db2bp. • Multiple shells cannot share the same DB2 back-end process. 10
  • 11. #IDUG Did you notice the quotation marks? • Watch out for shell specific characters! The first two lines work, the last line breaks. Do You know why? • $ db2 "select count(*) from staff" • $ db2 select "count(*)" from staff • $ db2 select count(*) from staff 11
  • 12. #IDUG Useful tricks with db2 command • Trick: start 'db2 -t' to copy and paste db2 commands interactively from other scripts • Hint: Check for other useful parameters via 'db2 list command options' • Put your favorites in an environment variable: export DB2OPTIONS='+a -c +ec -o -p' • Use ! to spawn any shell command from inside CLP • Within the DB2 shell, you can start logging with UPDATE COMMAND OPTIONS USING Z ON outfile.txt V ON 12
  • 13. #IDUG Useful tricks with db2 command (cont.) • To diagnose code page issues: • db2 -a connect to dbname • The SQLCA contains useful information for code page problems (See speaker notes for details) • Use the return code of db2 for error handling in your scripts • Code Description 0 DB2 command or SQL statement executed successfully 1 SELECT or FETCH statement returned no rows 2 DB2 command or SQL statement warning 4 DB2 command or SQL statement error 8 Command line processor system error 13
  • 14. #IDUG CLPPlus • New since DB2 9.7 • Optional installation component of IBM Data Server Client • It is not included in • DB2 Runtime Client • Data Server Driver Package • DB2 Express-C • Requires at least Java JRE 1.5.x • Can connect to DB2 LUW, DB2 z/OS and Informix via JDBC 14
  • 15. #IDUG CLPPlus Start • From Windows Startmenu: Start → IBM DB2 → Command line tools → Command line Processor Plus • On the command line: clpplus user@server:port/database • Missing arguments will be requested for connect • Database does not need to be cataloged • Continuous improvements in upcoming DB2 fixpacks and versions 15
  • 16. #IDUG CLPPlus Commands • Editor-Buffer can be used for most recently used commands • Commands for formating the data and pagination • Simple configuration commands • … 16
  • 18. #IDUG DB2 System Commands • DB2 system commands to query the system • Did you have a look in the manuals lately? ;-) • Sometimes there‘s no other way to RTFM and it is usually interesting/enlightening. • I think it is like a cross word. Just look over it from time to time and discover new features. • Have you heard about? • db2_local_ps, db2caem, db2ckbkp • db2flsn, db2logsforrfwd, db2osconf • db2tdbmgr, db2xprt, db2relocatedb 18
  • 19. #IDUG DB2 System Commands: db2pd • db2pd • Direct access to DB2 system memory (i.e. dirty uncommitted read ;-) • Low level details, but very helpful for analysis • Samples: • db2pd -db SAMPLE -locks // check for locks • db2pd -help | more • db2pd -db SAMPLE -tcbstats // lists table control block statistics • db2pd -edus interval=10 top=5 // top five CPU consumers of last 10 secs • db2pd -alldbs -hadr | grep -e onnected -e artition -e "andby " -e "rimary " // multiple dbs 19
  • 20. #IDUG DB2 System Commands: db2diag • db2diag • db2diag –H // displays last 30 minutes of the db2diag.log • db2diag –A // archives the db2diag.log and creates a new one • db2diag –g db=TEST // shows the entries only for the Database TEST • db2diag -l severe -H 2d // last 2 days of severe errors • db2diag -gvi msg:=DB2_MAX_INACT_STMTS | more • db2diag -gi data:="password validation" -fmt "%{tsmonth}-%{tsday}- %{tshour}.%{tsmin}.%{tssec}t%{database}t%{data}" | sed "s/DATA.*bytes//;N;s/n//" 2013-07-22-15.51.33 SAMPLE Password validation for user janedoo failed with rc = -2146500507 2013-07-22-15.51.59 SAMPLE Password validation for user james007 failed with rc = -2146500502 20
  • 21. #IDUG DB2 System Commands: db2top • Can be run interactive or in batch mode • db2top [−d dbname] [−n nodename] [−u username] [−p password] [−V schema] [−i interval] [−P <partition> ] [−b option] [−a] [−B] [−k] [−R] [−x] [−f file </HH:MM:SS> <+offset> ] [−D delimiter] [−C <option>] [−m duration] [−o outfile] • −b option, db2top will display information in CSV format. • Sorry for Windows users, only for Linux and Unix 21
  • 22. #IDUG •Part 2 – The Shell 22
  • 23. #IDUG Different operating systems – different shells • Platform differences • Windows CMD, PowerShell, (Posix-Shell with R2) • Linux: bash, csh, sh, pdksh, … • AIX: ksh, bash, csh, … • Solaris, HP-UX and other Unix dialects… ;-) • Different shells, mostly similar • See also http://en.wikipedia.org/wiki/Comparison_of_command_shells 23
  • 24. #IDUG Environment variables in Windows • set DB2INSTANCE=db2 • echo %DB2Instance% • Variables are not case sensitive • In batch files use %% instead of % because single % is for variables on console • Example: • for %i in (1 2 3) do echo %i 24
  • 25. #IDUG Environment variables in Unix • export DB2INSTANCE=db2 • echo $DB2INSTANCE • Shell is case sensitive about variable names • Works in shell scripts and on command line • Might need curly brackets to delimit the variable name: • echo ${LOGDIR}/mylogfile.txt • Example: • for i in 1 2 3 ; do echo ${i} ; done 25
  • 26. #IDUG Shell command line expansion • Be careful with your shell command line expansion! If you do not quote/escape properly you will get strange results. • db2 "SELECT * FROM employee" • db2 SELECT * FROM employee • An * in Unix gets replaced by all filenames in the current directory if it isn't quoted or escaped. • A tilde '~' will be replaced by your user home directory • Alias is just plain text replacement without parameters • alias ll='ls –alF' 26
  • 27. #IDUG Quoting in the command line • Some characters are interpreted in the shell and may need quotes • Watch out with *, ?, %, &, |, <, >, , ", !, ^, =, ~, $, , , {, }, (, ), ', ` • DB2 uses single quotes for Text ' ' • db2 UPDATE employee SET lastname='Smith' where ID='00210' • Shell uses double quotes " ", but can also use single quotes • db2 'select count(*) from employee' • Brackets have different meaning in a Unix shell: • tar cf - * | (mkdir /tmp/demo ; cd /tmp/demo ; tar xvf - ) 27
  • 28. #IDUG Quoting in the command line • Nesting can be tricky! Here we add quotes around our reorg command: • Windows: db2 "select 'db2 ""reorg table '||rtrim(tabschema)||'.'||rtrim(tabname)||' "" ' from syscat.tables where stats_time < current timestamp - 5 days" • Unix: db2 "select 'db2 " ' " ' "reorg table '||rtrim(tabschema)||'.'||rtrim(tabname)||' " ' " ' " ' from syscat.tables where stats_time < current timestamp - 5 days" • Or if you know the ASCII codes… ;-) db2 "select 'db2 '||chr(34)||'reorg table '||rtrim(tabschema)||'.'||rtrim(tabname)||chr(34) from syscat.tables where stats_time < current timestamp - 5 days" • All of the above commands can be directly used in your shell or in a script 28
  • 29. #IDUG Calling batch files • Starting a batch file from a command line will pass an environment to the batch • Windows • All environment variables are passed on • The batch process can re-use the database connection • Changes to variables remain. Otherwise use setlocal/endlocal • Unix • Exported variables are passed on. Variables defined with 'set' are left out. • The batch file is started as a sub-process (fork) and needs it's own connection to DB2 • Remedy: Use sourcing and prepend the command with '. ' (dot and space) to let the batch file run in the same shell instead of forking a new shell interpreter for the batch. 29
  • 30. #IDUG Calling batch files (cont.) • Batch files can get input parameters from the command line • Windows use them as %1, %2, etc. • Windows Shell Extensions can use e.g. %~x1 to split filenames in parameters • Unix gets $1, $2, etc.; $* stands for all parameters $? Is the return code of the previous command • Some shells can split strings in parameters ${1#db2} 30
  • 31. #IDUG Pipes and I/O redirections • Concatenation of commands • Sending output from one command to some other tool via pipes • db2 list applications | more • Multiple pipes • All commands in the pipe can run at the same time • Check for DB2 msgs in loads: cat LOAD*.msg | grep "^DB2" | sort | uniq –c • Conditional execution • db2start && db2 connect to SAMPLE • Using return codes for error handling 31
  • 32. #IDUG Standard streams • Standard streams stdin, stdout und stderr • Well known by C/C++-Programmers ;-) • The shell and command tools takes input from stdin (stream 0) and sends output to stdout (stream 1) and error messages to stderr (stream 2) . • Combining stderr in stdout • To redirect both stdout and stderr to the same file, use 2>&1 • E.g. useful for cronjobs, which send the output as mail • stderr exists for clever programmers, who want to show error messages even if the output is sent to a file 32
  • 33. #IDUG •Part 3 – GNU Utils 33
  • 34. #IDUG GNU Core Utilities – General Introduction • What they are? • Gnu's Not Unix, is the name for a complete Unix-compatible software system, which is free • Richard Stallman is the Guru/Father of the GNU Project. • Where did they come from? • By the 1980s, almost all software was proprietary. This made the GNU Project necessary. • By 1990 they either found or written all the major components except one—the kernel. Then Linux, a Unix-like kernel, was developed by Linus Torvalds in 1991 and made free software in 1992. 34
  • 35. #IDUG GNU Core Utilities – Where to get them? • Typically part of a Linux distribution or part of development tools • Install via YaST, yum, zypper or some other tool from your distribution • Look for software development tools, if you can't find them directly • GNU Core Utilities, available via http://www.gnu.org • Direct link: http://www.gnu.org/software/coreutils/ • Typically as sources to compile on your own • Also available as binaries for Windows and other platforms. 35
  • 36. #IDUG GNU Text Utilities • Overview of GNU Core Utilities • Utilities that are valuable when trying to match, replace and advance line processing (grep, sed, awk, …) • Which commands are useful • cat, head, tail, more|less, cut • wc, uniq, sort • fgrep, grep • diff, patch • sed, awk, xargs • Less used tools • split, csplit, reverse 36
  • 37. #IDUG GNU Text Utilities – cat • Copies file to console • cat db2diag.log // show db2diag.log • Can be used to put files in a pipe • cat myfile.tgz | gzip -d | tar xvf - // unpack tgz file • Can combine multiple files • cat *.msg | grep "^DB2" | sort | uniq -c // look for messages • Waits for input from console if no file given • Copy SSH keys to other machine • cat ~/.ssh/id_rsa.pub | ssh user@machine “mkdir ~/.ssh; cat >> ~/.ssh/authorized_keys” 37
  • 38. #IDUG GNU Text Utilities – head, tail • Fetch first lines of a file or stream • db2diag | head -17 • Returns the first or last lines from input • tail -20 db2diag.log • Find newest three files in directory • ls -1tr | tail -3 • Follow the file • tail -f ~/sqllib/db2dump/db2diag.log • You can even "space" a bit by pressing Enter several times ;-) 38
  • 39. #IDUG GNU Text Utilities – more | less • Simple pagination with Space (screen) and Return (single lines) • On some systems navigation commands possible (Pos1, End, G, …) • less is a powerful alternative to more (pun intended) • Can go backwards (b), allows searching (/) and highlights found string • Jump to end of current file (g) • … • Check out the man pages on your system • man less 39
  • 40. #IDUG GNU Text Utilities – cut • Returns parts of a text line (by char or field) • Quick'n dirty way for number of connections to databases in instance db2 list applications | cut -c67-75 | sort | uniq –c • Used very often to cut off pieces in a sequence of chars • Can work with a field separator (default=TAB) • Get second and third column from CSV file: cut -f2,3 -d, myfile.csv 40
  • 41. #IDUG GNU Text Utilities – wc • Very useful to count output lines (-l) or maximum line lenght (-L) number of chars (-c) or number of words (-w) • Findout how many dbs are running as HADR Primary on this machine db2pd -hadr -alldbs | grep -e 'rimary ' | wc –l • Number of connections from certain IP range db2 list applications | fgrep 172.25. | wc -l 41
  • 42. #IDUG GNU Text Utilities – sort • Does not only sort text files line by line • Can sort numerically • Find top 10 largest files or directory in /var/log du -sk /var/log/* | sort -r -n | head -10 • Can work on fields, field separator is transition between whitespace and non-whitespace • Some implementations have problems with multiple blanks gawk "/----/{s=1;getline} //{if(s==1) print}" 2013-04-12-prof-db01.txt | sed "s/ */ /" | gsort -k 2,2 -u | more 42
  • 43. #IDUG GNU Text Utilities – uniq • Remove duplicate lines (may need a sort first ;-) • Check for DB2 msgs in loads: cat LOAD*.msg | grep "^DB2" | sort | uniq –c -d • Sort is also capable to remove duplicates (sort –u), but uniq can also count the occurences, print only the duplicates and work on fields for comparison • Check out uniq --help 43
  • 44. #IDUG GNU Text Utilities – grep, fgrep, egrep • grep stands for Global Regular Expression Print • fgrep = fast grep with simpler expressions, egrep = extended grep • Regular expressions base on following principles • Boolean 'or' with | Pickup|Ute • Grouping with ( ) grey|gray is equal to gr(a|e)y • Quantification with ?, * and + ? stands for zero or one colou?r matches color and colour * stands for zero or more ab*c matches ac, abc, abbc, abbbc, … + stands for one and more ab+c matches abc, abbc, abbbc but not ac • Metachars ., [], [^ ], ^, $, n, {m,n}, [:print:] 44
  • 45. #IDUG GNU Text Utilities – sed • sed stands for stream editor • cat myselectoutput.txt | sed "s/ *$//g" >removed_trailing_spaces.txt • Complex sample: Format XML based logfile grep -e "Time Millis" -e LogText trace.log | sed "s/<Time Millis.*">//; s/</Time>//;N;s/n/ /; s/<LogText>//; s/<![CDATA[//; s/]]></LogText>//" 45 <Trace Level="MIN"> <Time Millis="1387783880929"> 2013-12-23 08:31:20.929+01:00</Time> <Server Format="IP">localhost</Server> <ProductId>COD</ProductId> <Component>Admin Server</Component> <ProductInstance></ProductInstance> <LogText><![CDATA[[LOG]: There is no agent installed on any of the host virtual machines. Skipping processing of the host 335aba37-2a44-3dcd-9069-bca2fab11a79]]></LogText> <Source FileName="com.ibm.license.mgmt.core.actions.vmman.algorithm.MergeHostStrategy" Method="process"/> <Thread>MergeDataTask-Thread:0</Thread> <Principal></Principal> </Trace>
  • 46. #IDUG GNU Text Utilities – awk • awk – Alfred Aho, Peter Weinberger and Brian Kernighan • awk is a data-driven scripting language consisting of a set of conditions and actions to be taken against streams of textual data • /condition/ {actions} • Special words BEGIN, END, FS, RS, … • Can be used from 1liners to complex programs to manipulate text • Nowadays sometimes superseeded by perl • http://www.thegeekstuff.com/tag/awk-tutorial-examples/ 46
  • 47. #IDUG GNU Text Utilities • diff: Find differences in two text files • patch: Insert changes in text files generated by diff • xargs: builds and executes commands from standard input • Less used tools • split, csplit, reverse, … 47
  • 48. #IDUG •Part 4 – Samples 48
  • 49. #IDUG Utilities and db2 commands • Samples of all things you can play around and customize to your needs • Quick way find out what DDL was executed on your DB • db2 list history all for db MYPRODDB | grep DDL • Check if your instance is running • ps -ef | grep $DB2INSTANCE | grep db2sysc | grep -v grep | wc –l • db2pd -agents | grep -e Partition -e Unable • Quick and dirty number of connections to all databases in instance: • db2 list applications | cut -c 99-106 | sort | uniq –c 49
  • 50. #IDUG Utilities and db2 commands (cont.) • Find the busiest tables via command line or ksh • db2pd -db dbname -tcbstats| awk '/TCB Table Stats/ { x =1} x==1 { print}' | awk '/^0x/ { print $9, $2}' | sort –rn | head -15 • Get reorg recommendations and change them to inplace reorg instead of offline reorg • db2 get recommendations for health indicator db.tb_reorg_req for database on d3t | grep "^REORG" |awk ' /REORG TABLE /{gsub(/;/, " inplace ;" )}; 1'|awk '{gsub(/"RUNSTATS/,";nRUNSTATS"); print}' | tee -a reorg.sql 50
  • 51. #IDUG Utilities and db2 commands (cont.) • Generate script to re-catalog your databases • db2 list db directory | egrep "Database alias|Node name" | sed -e 's/Database alias//g;s/Node name//g;s/ //g;s/=//g' | sed 'N;s/n/ : /' | awk '{print "db2 catalog db "$1" at node "$3;}' 51
  • 52. #IDUG Use SQL to generate stuff • Extract statements from package cache as input for db2advis • db2 -x "select '--#SET FREQUENCY '||ltrim(char(num_executions))||‘@@‘||rtrim(STMT_TEXT)||';' from table(snapshot_dyn_sql('PK2I11C1', -1)) as SNAP_STMT where stmt_text is not null and num_executions > 100 order by num_executions desc" | fgrep -iv "SYSIBM" | sed „s/ *$//“ >MostCommonQueries.txt • Now replace @@ by CRLF and feed into db2advis or db2batch • Checking integrity after foreign keys if needed • db2 -xtd! "select 'SET INTEGRITY FOR '||chr(34)||rtrim(tabschema)||chr(34)||'.'||chr(34)||rtrim(tabname )||chr(34)||' IMMEDIATE CHECKED FORCE GENERATED' from syscat.tables where tabschema not in ('SYSIBM','SYSCAT','SYSSTAT') and STATUS='C' order by parents,children!" >checkref.sql db2 -v -f checkref.sql echo Checking integrity for constraints done. 52
  • 53. #IDUG How to get started • Save output of DB2 utility to file, maybe shorten it as useful • cat textfile and pipe through filters • Start filtering out unwanted parts/lines • Modify/cut resulting parts until satisfied • Recombine pieces for the big result • Always think about the complexity of multiple executions of the filter programs • Maybe save intermediate results to files 53
  • 54. #IDUG •Part 5 – Other goodies 54
  • 55. #IDUG KEY-BASED AUTHENTICATION USING SSH • To get information in scripts from other machines you would need to access them without explicitly typing in passwords. • Key based authentication is situation of authentication takes place based on the public/private information, rather than with the more usual method of prompting for a password. • This is very convenient if a non-interactive process is trying to authenticate with a remote machine • SSH is the preferred way of setting up such connections now as it uses public and private key pairs to establish a secure connection. • It allows you securely to start commands on another machine without providing passwords! 55
  • 56. #IDUG Key-based authentication for SSH • Arranging key-based authentication for SSH • Machine "A" is the machine you want to connect from • Machine "B" is the machine you want to connect to • Setup – Part 1: • On Machine "A", generate your key pair using ssh-keygen -t dsa • This will create two files: • ~/.ssh/id_dsa • ~/.ssh/id_dsa.pub 56
  • 57. #IDUG Key-based authentication for SSH (cont.) • Setup – Part 2: • Log onto Machine "B" • Secure copy the id_dsa.pub key across to the local machine • Now look at the file /~.ssh/id_dsa.pub, if it starts with "ssh-dss", ie • ssh-dss AAAAB3NzaC1kc3MAAAC+CLO2M9OfcIjEaFBJ+cNAubJeCw8dtlH • Then append it to the file ./ssh/authorized_keys2 and ./ssh2/authorized_keys2 • cat id_dsa.pub >> ~/.ssh/authorized_keys2 • cat id_dsa.pub >> ~/.ssh2/authorized_keys2 • change permissions on files to user read/write only • Test – now you will be able to ssh from Machine A to Machine B (no password req.) 57
  • 58. #IDUG Backup/Restore via pipe • Pipes are special file objects under Unix operating systems • Create with mkfifo • Can be used for a quick backup/restore from one machine to another or between instances • Machine 1: Machine 2: mkfifo /tmp/mydest mkfifo /tmp/mysource db2 restore db test_1 as test_new from /tmp/mysource replace existing without prompting rsh host2 'cat > /tmp/mysource' < /tmp/mydest & db2 backup db test_1 to /tmp/mydest 58
  • 59. #IDUG Useful links • Introduction to text manipulation on UNIX-based systems http://www.ibm.com/developerworks/aix/library/au-unixtext/ • Bash 1liners http://www.bashoneliners.com/ • AWK Tutorial http://www.thegeekstuff.com/tag/awk-tutorial-examples/ • Twitter: ARSDB2 59
  • 60. #IDUG Roland Schock ARS Computer und Consulting GmbH schock@ars.de Session Code: ED02 DB2 Scripting Hacks
  • 62. #IDUG Windows 7 User Access Control • With User Access Control in Windows 7 or Windows Server 2008, you have to confirm the execution of administrative commands. • • To start the instance via db2start you need admin rights or you have to be instance owner. If the instance owner is your domain user, it is getting tricky, if you are on the road with your laptop. Runas doesn't help here! 62
  • 63. #IDUG Windows 7 User Access Control (cont.) • If you use your domain account for DB2 installation and administration, you get admin rights only as long as your laptop is in the domain. • You should better use a local user for install and administration • db2set DB2_GRP_LOOKUP and SYSADM_GROUP could help • Much easier is following trick: • Start → IBM DB2 → Commandline tools → Command window → Shift+Right-Click on the needed entry and login with MachineNameInstanceOwner (if the instance owner is a local userid) 63