Admin Tech Clash: Discussing Best (and Worst) Administration Practices from hundreds of customers

Admin Tech Clash: Discussing Best (and Worst)
Administration Practices from
hundreds of customers
Presented by: Christoph Adler
Ben Menesi

Christoph Adler
 Senior Consultant at panagenda
 15 years of IBM (Lotus) solutions
experience
 Since 2007:
 IBM Notes Client Management
 ICS Infrastructure Analysis and
Optimization
 Good to know:
 Lives in Germany, hence the funny
accent
 Frequent traveler and speaker at
international conferences
 No energy drinks – squirrel alert!
 Beer or wine? Beer!

 Head of Product at Ytria
 IBM Notes Administrator and Developer
wearing both hats
 Working with Ytria since 2011
 C API Development
 Server side Administration
 Enterprise IT Security
 Good to know:
 Born Hungarian, raised in Hungary &
Switzerland, living in Canada for the
past 5 years
 Frequent traveler and speaker at
international conferences
 Whiskey or Vodka? Beer with
Whiskey!
Ben Menesi

Your collaboration infrastructure contains a TON
of valuable data
In order to extract this data you need to know
WHERE to look
Visualization helps to see trends and patterns and
understand relationships
IBM can help with IDD (IBM Domino Doublecheck)
We (Ytria / panagenda) can help with our software
solutions, services and expertise BUT

Scout‘s honor: We won‘t talk about our
products in this session!

User Activity
Server-side Logging
Client-side Logs & Performance

Usage Activity
 Why should we care?
Understanding usage data allows us to find
unused / heavily used applications
In the light of this data we can calculate resource
requirements and optimize performance
We can profile our high impact users and
processes and optimize licensing

Usage Activity
 What data source should we use?
Activity
Trends
DB User
Activity
Log &
Catalog
3rd Party
Solutions
 Valuable high level
reports
 Not very
customizable
 Lack of
customization leads
to inaccuracies
 Very detailed logs
 Really crucial info
buried inside one
text field
 Unless through 3rd
party tools: NONO
 Great amount of
detail
 Only available per
database
 To be discussed later
 Amazing products
out there
 Some installed on
server side
 Some client side
 To be discussed later

Activity Trends
 How does it work??
Log.nsf Catalog.nsf Names.nsf Statistics
Activity Trend
Collector
Activity.nsf

Activity Trends
 Activity data – advantages
 DB Usage (active / inactive), space growth, etc…
 User details (active / inactive), connection times, dbs and
network used etc...)
 Server tasks (HTTP, IMAP, LDAP, Replication etc...)
 Some visual reports
 Activity data – disadvantages
 What you get is what you get – no digging deeper
 Raw data isn’t always used properly
 Think about a DB marked “Active” with only servers and
agents using it
 Doesn’t properly handle space used percentages

Log & Catalog
 Activity data – advantages
 It’s only two databases to analyze NOT hundreds or thousands
 You’ll get the full picture instead of only application usage
 Think about usage of system databases or client overhead
 Activity data – disadvantages
 Only contains data of the last 7 days
 Actually: There is an undocumented parameter for the Log=
notes.ini line:
 LOG=logfilename, log_option, cant_touch_this, days, size,
[activitydays]
 Data structure (Later)

Log & Catalog
 How to get valueable data out of log.nsf
• Create an additional view
• View selection formula: SELECT FORM = "Activity“
• Add columns that are interesting in your scenario

Log & Catalog
 Details: Database usage document
• Note there is a 1400 activity entry maximum per database (FIFO)
• There is also a 64K size limit for the user activity
• More details in IBM Technote #1086245

Log & Catalog
 Database activity: catalog.nsf
• related information, but
different focus
• Full text index details
• Replication information
• ACL overview
Note:
Domino does not distinguish
between user, server or maintenance
tasks activity at this level

Log & Catalog
 Example visualizations: Domino Environment Overview

Log & Catalog
 Example visualizations: Network Bandwith Demand

Log & Catalog
 Example visualizations: User Impact

Log & Catalog
 Example visualizations: Database Demand Characteristics

Log & Catalog
 Example visualizations: Network demand

Log & Catalog
 Real customer example:
 15 European Locations
 8000 Users
 1200 Domino Apps
 Logistics
„We want to mobilize the 20 most heavily used applications that
sales users throughout Europe access in read only mode. How can
we find them?“

Log & Catalog
Domino Data
 Collect session logs from all servers  log.nsf
 Collect nsf inventory from all servers  catalog.nsf
 Collect person information from all users  names.nsf
Non Domino Data
 Merge organizational info to person info
 Analyze and Visualize

Log & Catalog
 Applications used from Sales in read only mode

DB User Activity
 Recorded User Activity
 User Activity can be a *VERY* rich source of data (incl. HTTP!)
 Tip: No_Force_Activity_Logging=0 (Default value) allows on ALL
dbs
 Tip: Set confidential to prevent users with lower than Designer
access

DB User Activity
 Prior to ODS 48
 Entry size allocated 44 bytes, object size 61600 bytes (max 1400)
 ODS 48+ (undocumented improvement)
 Entry size 92 bytes. Object size 128800 bytes (same max 1400)
 Curious: only two additional counters added, yet ODS48+ entry size
is over 200% the size of ODS48 – why?
Date &
Time
Reads Writes User
Date &
Time
Reads Writes Updates Deletes User

DB User Activity
 Does Notes only log what it displays?
 Nope
 Read more: http://techlab.ytria.com/5448/lotus-notes-
articles/interpret-user-activity-traces-portrait-global-usage
Reads Adds Updates Deletes
Data
Non
Data
Σ
Data
Non
Data
Σ
Data
Non
Data
Σ
Data
Non
Data
Σ

DB User Activity
 Limitations
 Only available per database
 Free tools available to facilitate working with this data
across multiple databases:
http://www.agecom.com.au/useractivity
 Or implement via W32_NSFDbGetUserActivity
 Makes no difference between servers & users
 Filter user names using free or paid tools
 Only maintains 1400 entries
 Isn’t this the same for the log file activity documents?
 Wait, this doesn’t need to be an issue

DB User Activity
 Customer Example: calculating usage patterns from usage activity
Entry Date
Entry Date#1400
#1
5 days
Entry Date
Entry Date
120 days
#1400
#1
Database A Database B

3rd Party Solutions
 panagenda iDNA…

 3rd Party products for usage analysis
 IDD (IBM Domino Doublecheck)  free offer
http://goo.gl/maL0fo
 panagenda iDNA
https://www.panagenda.com/products/idna
 Ytria databaseEZ & consoleEZ
http://ytria.com/databaseez
http://ytria.com/consoleEZ
 Tableau
http://www.tableau.com
3rd Party Solutions

Mining logs
Domino Server
The Admin

Event lifecycle
1.
• Events take place
2.
• Events are logged
3.
• Some events captured
4
• Few events addressed
5
• Tons of events missed, angry users on the phone

Dissecting Log.nsf
 Server tasks can log events to multiple types of documents!
 But no duplicate logs: different information!
 Example: Server 1 replicator opens session to Server 2 as per
Connection Document settings
Misc.
Replication

Dissecting Log.nsf
 Miscellaneous events: what do we see?
 Some high level info (Server, Start & Finish time) and
 UI shows one text type item: Date + Time & Description
 What is an event?
Time Status Code Severity Type
Addin Name Target Server Target DB Target User

Dissecting Log.nsf
 Miscellaneous events – event attributes logged in multi-value Text
items
EventList
Time &
Description
EventSeverity
EventStatus
EventType
Event 1
Severity
HEX Status
Type
Time &
Description
Event 2
Severity
HEX Status
Type
Time &
Description
Event 3
Severity
HEX Status
Type

Dissecting Log.nsf
 Can we get more data?
 We can use formulas and the list of Severities, Events & Status
Codes to get a better idea of each Misc. log document
 Total # of events / document [Eventlist item]
– @Elements(Eventlist)
 Total # of unique events [EventStatus item]
– @Elements(@Unique(EventStatus))

Dissecting Log.nsf
 Highest severity / document [EventSeverity item]
– _sev:=@If(@Sort(@Unique(@Text(EventSeverity));[Ascendin
g])[1]="0";@Sort(@Unique(@Text(EventSeverity));[Ascendin
g])[2];@Sort(@Unique(@Text(EventSeverity));[Ascending])[1
]);
– Once we have the _sev variable, use icons to display (from DDM
resources)
– @If(_sev="0";"iconNormal";_sev="1";"iconFatal";_sev="2";"i
conFailure";_sev="3";"iconHigh";_sev="4";"iconLow";_sev="
5";"iconNormal";"")+".gif“
– What you get:

Dissecting Log.nsf
 A view displaying the highest severity encountered, # of events,
unique events and all levels of severities per document:
 More info here: http://www.slideshare.net/BenedekMenesi/ibm-
connected-2015

Client-side Logs & Performance

ClientClocking
NOTE:
All of this should only be done on “ONE” client – slows down client,
creates huge log files!
 Use the following to see and measure client performance
 CLIENT_CLOCK=1
 CONSOLE_LOG_ENABLED=1
 Debug_Outfile=c:debug.txt
 Debug_Console=1
 DEBUG_TCP_ALL=1 (use only if needed)
 Fileserver or local disk Performance debugging is virtually
impossible
 If on >= 8.5 try adding DEBUGGINGWCTENABLED=16386 in Notes

ClientClocking
 Client_Clock=1 leads to a debug file of the following format:

ClientClocking
OpenNTF – NRPC Parser
Link:
http://www.openntf.org/internal/home.nsf/project.xsp?action=openD
ocument&name=Notes%20RPC%20Parser
Developed by:
 Jamie Magee, MartinScott Consulting LLC, US
 Andrew Magerman, Magerman Collaborative Software GmbH,
Switzerland

ClientClocking
OpenNTF – NRPC Parser

Cache.ndk
We've all been “advised to”/guilty of deleting the cache.ndk in an
effort to cure Notes performance problems/symptoms
• Some companies even delete cache.ndk on a scheduled basis

Cache.ndk
HOWEVER, let us explain the impact of that on Notes performance ...
 Increases traffic between client and server by 4,000% (forty times
more traffic, yes)
(10,000+% if the data directory is on a network drive)
 The more databases end users use, the worse it gets
 There is always a root cause – deleting cache.ndk only fixes
symptoms
 Let's look at some detailed traffic analysis for just ONE application
(the mail file) of ONE user ...

Cache.ndk
100 users =
150 MB vs. 3,7 MB
1000 users =
1.5 GB vs. 37 MB
Excluding Fileservers!
After deleting cache.ndk Keeping / NOT deleting cache.ndk

On Disk Structure
ODS = On Disk Structure
 ODS 16 = Notes 2
 ODS 20 = Notes 4 (or Templates)
 ODS 43 = Notes 6 & 7
 ODS 51 = Notes 8.5 & 9
 ODS 52 = Notes 9.0.1
The difference between ODS 43 and 52 = up to 80% LESS FILE I/O
This helps with ANY client AND server hardware

On Disk Structure
CAUTION:
The ODS is not automatically updated when upgrading clients
 Except for names.nsf, bookmark.nsf, and desktop*.dsk/ndk – IF –
you upgrade properly (think ini:Setup/TemplateSetup=)
 ODS 51 requires Notes >= 8.5 and CREATE_R85_DATABASES=1 in
notes.ini for NEW db's
 ODS 52 requires Notes >= 9.0.1 and CREATE_R9_DATABASES=1 in
notes.ini for NEW db's

On Disk Structure
Good News for Notes > 8.5
 Use NSF_UpdateODS=1 with CREATE_R85/9_DATABASES=1 for a
one-time upgrade of most local databases
 NOTE: USE WITH EXTREME CARE IF YOUR DATA DIRECTORIES
ARE ON A NETWORK DRIVE!
 NOTE: End users can't access databases during compact
 For databases on server: no more downtime required, use:
 Compact [dbpath] –Replica to create replica in the
background (new in domino 9)
 More info: https://www-
01.ibm.com/support/docview.wss?uid=swg27039379&ai
d=1

On Disk Structure
Impact examples
 Startup time of a Notes 8.5.3 client with 3 ODS 20 apps in Notes
data = 10 seconds
 After ODS upgrade: 2 seconds
 Reduced File I/O of your disks/SAN/NAS after ODS 41 to 52 upgrade
by 60%
 Removing 70% of all old files in Data directories on SAN/NAS = 45%
less managed storage (backup)
The bad news: There is no way to tell what is “out there”
 3rd party tools or random inspections are highly recommended
 News for Notes > 8.5

Slow clients
Port settings are included in the notes.ini
 Type / Active / Encryption / Compression / Timeout
Bad idea
 Remove LAN0 / COMx / DisabledPorts
Good idea
Use TCPIP according to the client type
 PORTS = TCPIP
 TCPIP = TCP,0,15,0,,12320,  Desktops
 TCPIP = TCP,0,15,0,,45088,  Laptops
 TCPIP = TCP,0,15,0,,12288,  Citrix / VDI
Riverbed !?
Don‘t forget your Servers!

Admin Tech Clash: Discussing Best (and Worst) Administration Practices from hundreds of customers

More Related Content

Viewers also liked (13)

Similar to Admin Tech Clash: Discussing Best (and Worst) Administration Practices from hundreds of customers (20)

More from Christoph Adler (20)

Recently uploaded (20)

Admin Tech Clash: Discussing Best (and Worst) Administration Practices from hundreds of customers