OTN tour 2015 Experience in implementing SSL between oracle db and oracle clients
2 likes892 views
The TCP buffer size was too small for the SSL handshake. Increasing the TCP send/receive buffer sizes on the listeners resolved the issue.
OTN tour 2015 Experience in implementing SSL between oracle db and oracle clients
- 1. Experience in implementing SSL between Oracle DB and Oracle Clients
- 2. Agenda ● Project overview ● SSL session (1 way/2 ways) ● Certificate Authorities & Certificates ● 3 Different Oracle Listeners configurations (insecure/secure/ssl) ● SSL & Oracle Clients setup (tns configuration parameters) ● Issues faced & troubleshooting tools ● Conclusion ● Q & A
- 3. RAC Cluster Project overview SSL Listener SEC Listener SCAN Listeners Developers & End users / tools directly connecting to DB Main application technology stack OBIEE, WLS, Forms, Reports Integrations Other DBs, Essbase, Ora Net Clients SSL Listener SEC Listener SSL Listener SEC Listener
- 4. How does SSL/TLS work? How does SSL/TLS work? from http://security.stackexchange.com/ => here How does SSL work? What is an SSL handshake? from http://www.symantec.com/ => here Transport Layer Security from https://en.wikipedia.org/ => here
- 5. SSL session overview (1 way) ServerInitial request (to https://domainname.com) Hello .... SSLv2, SSLv3, TLSv1, .... ciphers, hash function ● Server Certificate ● Server Private Key Server Certificate (signed by Certificate Authority) Public Key + Domain Name (subject) + Org name ....● List of Trusted CA certificates Establish trust ... Agree on the symmetric encryption key ... premaster secret master secret & generate session keys Use session key and agreed encryption cipher to send data ...
- 6. SSL session overview (2 way) Server ● Server Certificate ● Server Private Key● List of Trusted CA certificates Establish trust ... Agree on the symmetric encryption key ... master secret & generate session keys Use session key and agreed encryption cipher to send data ... premaster secret ● Client Certificate ● Client Private Key ● List of Trusted CA certificates
- 7. ● There are 3 options to obtain an SSL certificate a. Generate your own self-signed certificate (from https://www.linux.com here) b. Request certificate from your organisation CA c. Request certificate from public CA ■ VeriSign ■ GeoTrust ■ ... https://en.wikipedia.org/wiki/Certificate_authority ● In A & B you or your organisation controls hashing and encryption protocols ● However no clients will recognize certificate by default ● In option C you will need to make sure your client versions supports protocols dictated by external CA a. Recent example => Many CAs announced that they will not support SHA-1 forcing SHA256/SHA-2 implementation Certificate Authorities & Certificates
- 8. ● Intermediate certificate authorities ○ https://en.wikipedia.org/wiki/Intermediate_certificate_authorities Certificate Authorities & Certificates
- 9. ● Certificate Authorities & Certificates
- 10. ● Certificate Authorities & Certificates
- 11. ● Certificate Authorities & Certificates
- 12. Oracle
- 13. Listener Config
- 14. 3 Different Oracle Listeners configurations ● Default Listener (insecure) ○ Username/password are encrypted ● Secure Listener ○ Use encryption the same way as SSL Listener ○ No authentication phase (SSL handshake) ● SSL Listener ○ Use encryption ○ Could be configured with 1 way or 2 ways authentication
- 15. Default Listener (insecure) listener.ora LISTENER = (DESCRIPTION = (ADDRESS = (PROTOCOL=TCP)(HOST = db_host)(PORT = 1521)) (ADDRESS = (PROTOCOL=IPC)(KEY = REGLSNR_LISTENER)) )
- 16. Secure Listener listener.ora LISTENER = (DESCRIPTION = (ADDRESS = (PROTOCOL=TCP)(HOST = db_host)(PORT = 1522)) (ADDRESS = (PROTOCOL=IPC)(KEY = REGLSNR_LISTENER)) ) sqlnet.ora sqlnet.crypto_checksum_server = REQUIRED sqlnet.encryption_server = REQUIRED sqlnet.crypto_checksum_types_server = (SHA1) sqlnet.encryption_types_server = (AES256)
- 17. listener.ora LISTENER = (DESCRIPTION = (ADDRESS = (PROTOCOL=TCPS)(HOST = db_host)(PORT = 1523)) (ADDRESS = (PROTOCOL=IPC)(KEY = REGLSNR_LISTENER)) ) WALLET_LOCATION = (SOURCE = (METHOD = FILE) (METHOD_DATA = (DIRECTORY = /u01/app/oracle/SSL/certs) ) ) SSL_CLIENT_AUTHENTICATION = FALSE SSL_CIPHER_SUITES = (SSL_RSA_WITH_AES_256_CBC_SHA) SSL Listener
- 18. oracle@host > openssl s_client -connect db_host:1523 CONNECTED(00000003) depth=0 CN = PROD ... verify return:1 Certificate chain 0 s:/CN=PROD i:/CN=PROD --- Server certificate -----BEGIN CERTIFICATE----- MIIBkDCB+gIBADANBgkqhkiG9w0BAQQFADARMQ8wDQYDVQQDEwZFQlNMTlAwHhcN MTQwNzAyMDQzNTI4WhcNNDExMTE2MDUzNTI4WjARMQ8wDQYDVQQDEwZFQlNMTlAw gZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAJGbN1hxmtFKx1SpUU8JPvRyjMzW ... -----END CERTIFICATE----- subject=/CN=PROD issuer=/CN=PROD --- Acceptable client certificate CA names /C=US/O=VeriSign, Inc./OU=Class 1 Public Primary Certification Authority /C=US/O=GTE Corporation/OU=GTE CyberTrust Solutions, Inc./CN=GTE CyberTrust Global Root /C=US/O=RSA Data Security, Inc./OU=Secure Server Certification Authority /C=US/O=VeriSign, Inc./OU=Class 2 Public Primary Certification Authority /C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority /C=US/O=GeoTrust Inc./CN=GeoTrust Global CA /C=US/O=Equifax/OU=Equifax Secure Certificate Authority --- SSL handshake has read 1269 bytes and written 210 bytes --- New, TLSv1/SSLv3, Cipher is AES256-SHA ... SSL & Oracle Clients setup
- 19. SSL & Oracle Clients setup ... --- SSL handshake has read 1269 bytes and written 210 bytes --- New, TLSv1/SSLv3, Cipher is AES256-SHA Server public key is 1024 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1 Cipher : AES256-SHA Session-ID: Session-ID-ctx: Master-Key: 68E402B5CD161618522350EE... Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1447632207 Timeout : 300 (sec) Verify return code: 18 (self signed certificate) --- oracle@host >
- 20. Client Config
- 21. == TCP ========================================================================== tnsnames.ora PROD = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = prod-db-scan.corp.myorg.lv)(PORT = 1523)) ) (CONNECT_DATA = (SERVICE_NAME = PROD_ETL01) ) ) == TCPS ========================================================================= tnsnames.ora PROD_SSL = (DESCRIPTION = (SECURITY=(SSL_SERVER_CERT_DN="CN=prod-db.corp.myorg.lv,O=MyOrg Inc,L=Riga,C=Latvia")) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCPS)(HOST = prod-db-scan.corp.myorg.lv)(PORT = 7048)) ) (CONNECT_DATA = (SERVICE_NAME = PROD_ETL01) ) ) SSL & Oracle Clients setup (OCI)
- 22. == TCPS ========================================================================== sqlnet.ora SSL_SERVER_DN_MATCH = yes SSL_CLIENT_AUTHENTICATION = FALSE SSL_CIPHER_SUITES = (SSL_RSA_WITH_AES_256_CBC_SHA) WALLET_LOCATION = (SOURCE = (METHOD = FILE) (METHOD_DATA = (DIRECTORY = /u01/app/oracle/SSL/cert) ) ) SSL & Oracle Clients setup (OCI)
- 23. ● java … -Doracle.net.ssl_server_dn_match=true … ● jdbc:oracle:thin:@ (DESCRIPTION =( SECURITY=(SSL_SERVER_CERT_DN="CN=prod-db.corp.myorg.lv,O=MyOrg Inc,L=Riga,C=Latvia")) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCPS)(HOST = prod-db-scan.corp.myorg.lv)(PORT = 7048) ))(CONNECT_DATA = (SERVICE_NAME = PROD_ETL01))) ● Java has it own certificate store SSL & Oracle Clients setup (JDBC)
- 25. SSL & Oracle Clients setup (OCI) Using the orapki Utility to Manage PKI Elements https://docs.oracle.com/database/121/DBSEG/asoappf.htm#DBSEG610
- 26. $ ls -l /u01/app/oracle/SSL/cert/* total 16 -rw-r--r--@ 1 yvel 5000 1365 Aug 21 2014 cwallet.sso -rw-r--r--@ 1 yvel 5000 1288 Aug 21 2014 ewallet.p12 oracle@host:/home/oracle> orapki wallet display -wallet /u01/app/oracle/SSL/cert ; date Oracle PKI Tool : Version 11.2.0.4.0 - Production Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved. Requested Certificates: User Certificates: Subject: CN=PROD Trusted Certificates: Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign, Inc.,C=US Subject: OU=Equifax Secure Certificate Authority,O=Equifax,C=US Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign, Inc.,C=US Subject: CN=GeoTrust Global CA,O=GeoTrust Inc.,C=US Subject: OU=Secure Server Certification Authority,O=RSA Data Security, Inc.,C=US Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign, Inc.,C=US Wed Sep 9 16:27:36 PDT 2015 oracle@kpfp2:/home/oracle> orapki wallet create -wallet $WALLET -auto_login -pwd $v_pwd orapki wallet remove -wallet $WALLET -trusted_cert_all -pwd $v_pwd orapki wallet display -wallet $WALLET -pwd $v_pwd orapki wallet jks_to_pkcs12 -wallet $WALLET -keystore /full/path/to/certificate.jks -jkspwd <pwd> orapki wallet add -wallet $WALLET -trusted_cert -cert GIAG2.crt -pwd $v_pwd SSL & Oracle Clients setup (OCI)
- 27. SSL & Oracle Clients setup (OCI) $ openssl pkcs12 -info -in /u01/app/oracle/SSL/cert/ewallet.p12 Enter Import Password: MAC Iteration 1024 MAC verified OK PKCS7 Encrypted data: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 1024 Certificate bag Bag Attributes localKeyID: E6 B6 52 DD 00 00 00 04 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00 04 subject=/C=US/O=MyOrg Inc/CN=MyOrg Internet Authority G2 issuer=/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA -----BEGIN CERTIFICATE----- MIIEBDCCAuygAwIBAgIDAjppMA0GCSqGSIb3DQEBBQUAMEIxCzAJBgNVBAYTAlVT MRYwFAYDVQQKEw1HZW9UcnVzdCBJbmMuMRswGQYDVQQDExJHZW9UcnVzdCBHbG9i YWwgQ0EwHhcNMTMwNDA1MTUxNTU1WhcNMTUwNDA0MTUxNTU1WjBJMQswCQYDVQQG EwJVUzETMBEGA1UEChMKR29vZ2xlIEluYzElMCMGA1UEAxMcR29vZ2xlIEludGVy bmV0IEF1dGhvcml0eSBHMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB AJwqBHdc2FCROgajguDYUEi8iT/xGXAaiEZ+4I/F8YnOIe5a/mENtzJEiaB0C1NP VaTOgmKV7utZX8bhBYASxF6UP7xbSDj0U/ck5vuR6RXEz/RTDfRK/J9U3n2+oGtv h8DQUB8oMANA2ghzUWx//zo8pzcGjr1LEQTrfSTe5vn8MXH7lNVg8y5Kr0LSy+rE ... PKCS 12 => https://en.wikipedia.org/wiki/PKCS_12 It defines an archive file format for storing many cryptography objects as a single file. It is commonly used to bundle a private key with its X.509 certificate.
- 28. ● JDBC Clients ○ Have it own default certificate store with preloaded trusted CA list $ v_java_cert=/Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home/jre/lib/security/cacerts $ keytool -list -keystore $v_java_cert -storepass changeit Keystore type: JKS Keystore provider: SUN Your keystore contains 93 entries digicertassuredidrootca, Apr 16, 2008, trustedCertEntry, Certificate fingerprint (SHA1): 05:63:B8:63:0D:62:D7:5A:BB:C8:AB:1E:4B:DF:B5:A8:99:B2:4D:43 comodorsaca, May 12, 2015, trustedCertEntry, Certificate fingerprint (SHA1): AF:E5:D2:44:A8:D1:19:42:30:FF:47:9F:E2:F8:97:BB:CD:7A:8C:B4 thawtepremiumserverca, May 26, 2015, trustedCertEntry, Certificate fingerprint (SHA1): E0:AB:05:94:20:72:54:93:05:60:62:02:36:70:F7:CD:2E:FC:66:66 ... $ keytool -exportcert -alias digicertassuredidrootca -keystore $v_java_cert -storepass changeit - file test.crt -rfc $ keytool -printcert -file test.crt Owner: CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US Issuer: CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US Serial number: ce7e0e517d846fe8fe560fc1bf03039 Valid from: Thu Nov 09 16:00:00 PST 2006 until: Sun Nov 09 16:00:00 PST 2031 Certificate fingerprints: MD5: 87:CE:0B:7B:2A:0E:49:00:E1:58:71:9B:37:A8:93:72 SSL & Oracle Clients setup (JBDC)
- 29. SSL & Oracle Clients setup (JDBC) ● A JRE client must be updated with the JCE to enable TLS_RSA_WITH_AES_256_CBC_SHA cipher. ○ Download appropriate JCE archive with 2 jar files ○ Copy the jar files to $JAVA_HOME/jre/lib/security/ directory ● The JCE could be obtained from the URL below depending on a JRE version ○ 1.6 JCE ○ 1.7 JCE ○ 1.8 JCE
- 30. Issues faced
- 31. RAC Cluster Variety of clients and versions to cover ... SSL Listener SEC Listener SCAN Listeners Developers & End users using tools directly connecting to DB Main application technology stack OBIEE, WLS, Forms, Reports Integrations Other DBs, Essbase, Ora Net Clients SSL Listener SEC Listener SSL Listener SEC Listener 1. 2.3.
- 32. Some components used old Oracle Clients ● Essbase ○ Blend in 11.1.0.7 Oracle Client ○ Challenging to update ● OBIEE ○ Presentation layer use 11.1.0.7 Oracle Client
- 33. SSL Listener & TCP Buffer Problem definition Initial: ● DB Instances doesn't register DB Services with a remote SCAN listeners, after a new certificate has been added to server wallet Current: ● All SSL connections from one node hangs connecting to any SSL enabled listeners that runs on a remote node if a listener is able to send data out quickly enough
- 34. SSL Listener & TCP Buffer SSL Listener works if …. a) strace-ing the listener process b) turning debug logging up to a high level in the listener c) Running the listener on a non-bonded interface d) Having the listener increase its send buffer size from 16k (default) to 32k or 64k on the socket d1) We can also do this at the OS level via `echo "4096 32768 4194304" > /proc/sys/net/ipv4/tcp_wmem'
- 35. SSL Listener & TCP Buffer Normal / Expected processing sPORT:sIP - cIP:cPORT TCP socketTCP socket buffer Process owner of the socket O_NONBLOCK W3 W2 W1 DATA Data writes O_NONBLOCK sPORT:sIP - cIP:cPORTProcess owner of the socket DATA Kernel waits a bit for more data and sends it out Buffer is full. Kernel sends EAGAIN back 5k 1k 3k 3k W3b 2k W3bEAGAIN sPORT:sIP - cIP:cPORTProcess owner of the socket The process code handles error and keep sending the rest of the data until successful 3k W3b 3k Kernel refuses writes until buffer is free
- 36. Oracle Listener - fails to process EAGAIN TCP socketTCP socket buffer O_NONBLOCK Data writes O_NONBLOCK sPORT:sIP - cIP:cPORTSSL Listener DATA Buffer is full. Kernel sends EAGAIN back 3k W3b 2k W3bEAGAIN sPORT:sIP - cIP:cPORTSSL Listener Listener never re-sends the W3b part Kernel ready to process writes after some time Oracle Listener ssl handshake function fails to process EAGAIN errors sPORT:sIP - cIP:cPORTSSL Listener DATA Buffer is full. Kernel sends EAGAIN back 2k W3b 3k W3b The W3b part is dropped Where is client's response? W3b The W3b part is dropped SSL Listener & TCP Buffer
- 37. Conclusions
- 38. RAC Cluster Variety of clients and versions to cover ... SSL Listener SEC Listener SCAN Listeners Developers & End users using tools directly connecting to DB Main application technology stack OBIEE, WLS, Forms, Reports Integrations Other DBs, Essbase, Ora Net Clients SSL Listener SEC Listener SSL Listener SEC Listener
- 39. Q & A