SlideShare a Scribd company logo

SSO 5.1 start-up issues

F
fbuechsel

The document discusses troubleshooting SSO startup issues. It provides information on SSO architecture, common symptoms of failures like vCenter Server being unable to start or login issues, and a suggested troubleshooting procedure. This includes checking logs, services, ports, and database configuration. Case studies demonstrate normal startup logs and an example where the database host is not reachable.

Software
1 of 31
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
© 2014 VMware Inc. All rights reserved. Troubleshooting SSO 5.1 Start-Up issues Frank Buechsel
Agenda – SSO architecture – Symptoms of SSO not running or malfunctioning – Suggested troubleshooting procedure – Case studies – Useful command line options
SSO architecture
SSO 5.1 architecture • Status for the actual java wrapper • Does not reflect the status of the 4 sub systems • IMS (https://localhost:7444/ims/STSService?wsdl) • Lookupservice (https://localhost:7444/lookupservice/sdk) • ROOT (vFabric) (https://localhost:7444) • SSO-Adminserver (https://localhost:7444/sso-adminserver/sdk)
Symptoms of SSO not running or malfunctioning
Symptoms of SSO not running or malfunctioning 1/3 • vCenter Server unable to start vpxd.log: 2013-05-19T02:45:27.312-07:00 [04628 info '[SSO][CreateSsoFacade]'] [CreateUserDirectory] STS URI set to: https://vCenter.newhire.local:7444/ims/STSService?wsdl 2013-05-19T02:45:27.312-07:00 [04628 info '[SSO][CreateSsoFacade]'] [CreateUserDirectory] Admin URI set to: https://vCenter.newhire.local:7444/sso-adminserver/sdk 2013-05-19T02:45:27.312-07:00 [04628 info '[SSO][CreateSsoFacade]'] [CreateUserDirectory] Groupcheck URI set to: https://vCenter.newhire.local:7444/sso-adminserver/sdk 2013-05-19T02:45:27.555-07:00 [04628 error 'Default'] Found dangling SSL error: [0] error:00000001:lib(0):func(0):reason(1) 2013-05-19T02:45:27.555-07:00 [04628 error 'Default'] Found dangling SSL error: [1] error:00000001:lib(0):func(0):reason(1) 2013-05-19T02:45:27.555-07:00 [04628 error '[SSO][SsoFactory_CreateFacade]'] Unable to create SSO facade: vmodl.fault.SystemError. 2013-05-19T02:45:27.555-07:00 [04628 error 'vpxdvpxdMain'] [Vpxd::ServerApp::Init] Init failed: Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr) --> Backtrace: […] 2013-05-19T02:45:27.556-07:00 [04628 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...
Symptoms of SSO not running or malfunctioning 2/3 • No login possible in the Web Client / vSphere Client
Symptoms of SSO not running or malfunctioning 3/3 • Installation of dependent services failing (Inventory Service, vCenter Server, Web Client)
Suggested troubleshooting procedure
Suggested troubleshooting procedure 1/3 • Check in services.msc if the service is running • Remember that this only referes to the wrapper and tomcat • Check for used ports • Look at the most recent catalina.log • Look at the most recent localhost.log • Look at the imsSystem.log • Look at the config.txt • Look at the database
Suggested troubleshooting procedure 2/3 Java exceptions are hard to read without being filtered for useful information • Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Cannot open database "RSA" requested by the login. The login failed.) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:286) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:993) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:897) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291) at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:626) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:407) at com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext.access$001(SecurityAwareClassPathXmlApplicationContext.java:27) at com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext$1.run(SecurityAwareClassPathXmlApplicationContext.java:164) at com.rsa.ims.security.spi.SimpleSecurityContextImpl.doAs(SimpleSecurityContextImpl.java:82) at com.rsa.security.SecurityContext.doAs(SecurityContext.java:419) at com.rsa.security.SecurityContext.doAsSystem(SecurityContext.java:461) at com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext.refresh(SecurityAwareClassPathXmlApplicationContext.java:161) at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139) at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:105) at com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext.<init>(SecurityAwareClassPathXmlApplicationContext.java:90) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:126) ... 29 more Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Cannot open database "RSA" requested by the login. The login failed.) at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:141) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:105) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:278) ... 57 more
Suggested troubleshooting procedure 3/3 • Log bundle analysis the manual way (will always read in the latest log) • $ ls -tr catalina* | tail -n 1 | xargs awk -F "INFO " '{print $2}' | tail -n 4 | grep Starting | wc -l Expected output would be “4” • $ ls -tr localhost.* | tail -n 1 | xargs cat | egrep '(Caused)' Expected output would be nothing • $ egrep '(Caused)' imsSystem.log Expected output would be nothing • $ cat config.txt | grep -F db. | tail -n 11 | grep com.rsa (IMS component configuration) • $ cat config.txt | grep -F db. | tail -n 11 | grep " db" (lookupservice db configuration) Expected output would be matching information
Case studies
Case studies – Normal start-up 1/3 No exceptions in catalina.log, localhost.log and imsSystem.log catalina.log
Case studies – Normal start-up 2/3 localhost.log imsSystem.log
Case studies – Normal start-up 3/3 IMS db configuration $ less config.txt | grep -F db. | tail -n 11 | grep " db" [2013-05-18 10:14:21,956 INFO StaticDataReporter com.vmware.vim.ssoconfig] db.url=jdbc:jtds:sqlserver://; serverName=;instance=VCSQL;databaseName=RSA [2013-05-18 10:14:21,956 INFO StaticDataReporter com.vmware.vim.ssoconfig] db.user=RSA_USER [2013-05-18 10:14:21,957 INFO StaticDataReporter com.vmware.vim.ssoconfig] db.type=Mssql [2013-05-18 10:14:21,985 INFO StaticDataReporter com.vmware.vim.ssoconfig] db.host=VCENTER Lookupservice db configuration $ less config.txt | grep -F db. | tail -n 11 | grep com.rsa [2013-05-18 10:14:21,953 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.domain= [2013-05-18 10:14:21,953 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.instance=RSA [2013-05-18 10:14:21,953 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.type=MSSQL [2013-05-18 10:14:21,953 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.msserverinstance=VCSQL [2013-05-18 10:14:21,953 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.hostname=VCENTER [2013-05-18 10:14:21,955 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.port=0
Case studies – DB host not reachable • $ ls catalina* | tail -n 1 | xargs awk -F "INFO " '{print $2}' | tail -n 4 | grep Starting | wc –l 0 $ ls localhost.* | tail -n 1 | xargs cat | egrep '(Caused)‘ Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ims' defined in class path resource [beanRefContext.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Network error IOException: Connection refused: connect) Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Network error IOException: Connection refused: connect) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Network error IOException: Connection refused: connect) Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Network error IOException: Connection refused: connect) Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Network error IOException: Connection refused: connect) Caused by: java.sql.SQLException: Network error IOException: Connection refused: connect Caused by: java.net.ConnectException: Connection refused: connect Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException Caused by: java.lang.reflect.InvocationTargetException Caused by: java.lang.IllegalStateException: ComponentUtils not initialized Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vim.lookup.impl.DbStorage]: Constructor threw exception; nested exception is com.vmware.vim.lookup.exception.StorageException: Error executing statement Caused by: com.vmware.vim.lookup.exception.StorageException: Error executing statement Caused by: java.sql.SQLException: Network error IOException: Connection refused: connect Caused by: java.net.ConnectException: Connection refused: connect
Case studies – RSA database offline • $ ls catalina* | tail -n 1 | xargs awk -F "INFO " '{print $2}' | tail -n 4 | grep Starting | wc –l 0 • $ ls localhost.* | tail -n 1 | xargs cat | egrep '(Caused)‘ Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ims' defined in class path resource [beanRefContext.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Cannot open database "RSA" requested by the login. The login failed.) Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Cannot open database "RSA" requested by the login. The login failed.) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Cannot open database "RSA" requested by the login. The login failed.) Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Cannot open database "RSA" requested by the login. The login failed.) Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Cannot open database "RSA" requested by the login. The login failed.) Caused by: java.sql.SQLException: Cannot open database "RSA" requested by the login. The login failed. Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException Caused by: java.lang.reflect.InvocationTargetException Caused by: java.lang.IllegalStateException: ComponentUtils not initialized Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vim.lookup.impl.DbStorage]: Constructor threw exception; nested exception is com.vmware.vim.lookup.exception.StorageException: Error executing statement Caused by: com.vmware.vim.lookup.exception.StorageException: Error executing statement Caused by: java.sql.SQLException: Cannot open database "RSA" requested by the login. The login failed.
Case studies – Expired RSA_USER password • $ ls catalina* | tail -n 1 | xargs awk -F "INFO " '{print $2}' | tail -n 4 | grep Starting | wc –l 0 • $ ls localhost.* | tail -n 1 | xargs cat | egrep '(Caused)‘ Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ims' defined in class path resource [beanRefContext.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'RSA_USER'. Reason: The password of the account must be changed.) Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'RSA_USER'. Reason: The password of the account must be changed.) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'RSA_USER'. Reason: The password of the account must be changed.) Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'RSA_USER'. Reason: The password of the account must be changed.) Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'RSA_USER'. Reason: The password of the account must be changed.) Caused by: java.sql.SQLException: Login failed for user 'RSA_USER'. Reason: The password of the account must be changed. Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException Caused by: java.lang.reflect.InvocationTargetException Caused by: java.lang.IllegalStateException: ComponentUtils not initialized Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vim.lookup.impl.DbStorage]: Constructor threw exception; nested exception is com.vmware.vim.lookup.exception.StorageException: Error executing statement Caused by: com.vmware.vim.lookup.exception.StorageException: Error executing statement Caused by: java.sql.SQLException: Login failed for user 'RSA_USER'. Reason: The password of the account must be changed.
Case studies – Incorrect RSA_USER password • $ ls catalina* | tail -n 1 | xargs awk -F "INFO " '{print $2}' | tail -n 4 | grep Starting | wc –l 0 • $ ls localhost.* | tail -n 1 | xargs cat | egrep '(Caused)‘ Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ims' defined in class path resource [beanRefContext.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'RSA_USER'.) Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'RSA_USER'.) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'RSA_USER'.) Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'RSA_USER'.) Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'RSA_USER'.) Caused by: java.sql.SQLException: Login failed for user 'RSA_USER'. Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException Caused by: java.lang.reflect.InvocationTargetException Caused by: java.lang.IllegalStateException: ComponentUtils not initialized Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vim.lookup.impl.DbStorage]: Constructor threw exception; nested exception is com.vmware.vim.lookup.exception.StorageException: Error executing statement Caused by: com.vmware.vim.lookup.exception.StorageException: Error executing statement Caused by: java.sql.SQLException: Login failed for user 'RSA_USER'.
Case studies – Incorrect db information after db move • $ ls catalina* | tail -n 1 | xargs awk -F "INFO " '{print $2}' | tail -n 4 | grep Starting | wc –l 0 • $ ls localhost.* | tail -n 1 | xargs cat | egrep '(Caused)‘ Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vim.lookup.impl.DbStorage]: Constructor threw exception; nested exception is com.vmware.vim.lookup.exception.StorageException: Error executing statement Caused by: com.vmware.vim.lookup.exception.StorageException: Error executing statement Caused by: java.sql.SQLException: Unable to get information from SQL Server: VCENTER2. • $ less config.txt | grep -F db. | tail -n 11 | grep com.rsa [2013-05-18 11:30:10,664 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.domain= [2013-05-18 11:30:10,664 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.instance=RSA [2013-05-18 11:30:10,664 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.type=MSSQL [2013-05-18 11:30:10,664 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.msserverinstance=VCSQL [2013-05-18 11:30:10,664 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.hostname=VCENTER [2013-05-18 11:30:10,665 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.port=0 • $ less config.txt | grep -F db. | tail -n 11 | grep " db“ [2013-05-18 11:30:10,665 INFO StaticDataReporter com.vmware.vim.ssoconfig] db.url=jdbc:jtds:sqlserver://;serverName=;instance=VCSQL;databaseName=RSA [2013-05-18 11:30:10,665 INFO StaticDataReporter com.vmware.vim.ssoconfig] db.user=RSA_USER [2013-05-18 11:30:10,665 INFO StaticDataReporter com.vmware.vim.ssoconfig] db.type=Mssql [2013-05-18 11:30:10,665 INFO StaticDataReporter com.vmware.vim.ssoconfig] db.host=VCENTER2
Case studies – DC unavailable • $ ls catalina* | tail -n 1 | xargs awk -F "INFO " '{print $2}' | tail -n 4 | grep Starting | wc –l 4 • $ ls localhost.* | tail -n 1 | xargs cat | egrep '(Caused)‘ • $ egrep '(Caused)' imsSystem.log Caused by: javax.naming.CommunicationException: DC.newhire.local:3268 [Root exception is java.net.ConnectException: Connection timed out: connect] Caused by: java.net.ConnectException: Connection timed out: connect Caused by: javax.resource.spi.ResourceAdapterInternalException: Unable to create a managed connection 'ldap://DC.newhire.local:3268' with 'newhireadministrator' Reason: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection DC.newhire.local:3268 Caused by: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection DC.newhire.local:3268 Caused by: javax.naming.CommunicationException: DC.newhire.local:3268 [Root exception is java.net.ConnectException: Connection timed out: connect] Caused by: java.net.ConnectException: Connection timed out: connect Caused by: javax.naming.NamingException: getInitialContext failed. javax.resource.spi.ResourceAdapterInternalException: Unable to create a managed connection 'ldap://DC.newhire.local:3268' with 'newhireadministrator' Reason: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection DC.newhire.local:3268 [Root exception is javax.resource.spi.ResourceAdapterInternalException: Unable to create a managed connection 'ldap://DC.newhire.local:3268' with 'newhireadministrator' Reason: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection DC.newhire.local:3268] Caused by: javax.resource.spi.ResourceAdapterInternalException: Unable to create a managed connection 'ldap://DC.newhire.local:3268' with 'newhireadministrator' Reason: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection DC.newhire.local:3268 Caused by: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection DC.newhire.local:3268 Caused by: javax.naming.CommunicationException: DC.newhire.local:3268 [Root exception is java.net.ConnectException: Connection timed out: connect] Caused by: java.net.ConnectException: Connection timed out: connect
Useful command line options
Useful Command Line options 1/8 • Autodiscovery after installation is finished • ssocli configure-riat -a discover-is -u admin --verbose
Useful Command Line options 2/8 • List identity sources (useful as this information is not collected in an SSO log bundle) • ssocli manage-identity-sources -a list -u admin
Useful Command Line options 3/8 • Delete an identity source • ssocli manage-identity-sources -a delete -u admin -g <ims.id>
Useful Command Line options 4/8 • Create an identity source • ssocli manage-identity-sources -a create -u admin
Useful Command Line options 5/8 • Change the password for SSO admins and the SSO master password • ssocli reset-admin-password • ssocli manage-secrets -a change
Useful Command Line options 6/8 • Unlock the system after too many system changes • ssocli manage-secrets -a recover
Useful Command Line options 7/8 • Testing database connectivity without SQL Management Studio • sqlcmd -S hostnameinstance -U RSA_USER -d RSA
Useful Command Line options 8/8 • Silent uninstall of SSO • msiexec /qn /x {DEC4C346-414B-4814-9BF3-CAC14154B55A} MASTER_PASSWORD <master password>

More Related Content

PPTX
vCenter Server 5.5 Single Sign-On VMDir deep dive
fbuechsel
 
PPTX
SSL deep dive vCenter Server 5.5
fbuechsel
 
PDF
VMworld 2013: vSphere vCenter Single Sign-on Best Practices
VMworld
 
PDF
VMworld 2013: vCenter Deep Dive
VMworld
 
PDF
VMworld 2014: vCenter Server Architecture and Deployment Deep Dive
VMworld
 
PDF
vCenter and ESXi network port communications
Animesh Dixit
 
PDF
VMworld 2013: vSphere Upgrade Series Part 1: vCenter Server
VMworld
 
PDF
Introduction to Role Based Administration in WildFly 8
Dimitris Andreadis
 
vCenter Server 5.5 Single Sign-On VMDir deep dive
fbuechsel
 
SSL deep dive vCenter Server 5.5
fbuechsel
 
VMworld 2013: vSphere vCenter Single Sign-on Best Practices
VMworld
 
VMworld 2013: vCenter Deep Dive
VMworld
 
VMworld 2014: vCenter Server Architecture and Deployment Deep Dive
VMworld
 
vCenter and ESXi network port communications
Animesh Dixit
 
VMworld 2013: vSphere Upgrade Series Part 1: vCenter Server
VMworld
 
Introduction to Role Based Administration in WildFly 8
Dimitris Andreadis
 

What's hot (20)

PPTX
Introduction to vSphere logs
Kaushik Chakraborty
 
PPTX
WebLogic Administration course outline
Vybhava Technologies
 
PDF
Oracle WebLogic Diagnostics & Perfomance tuning
Michel Schildmeijer
 
PDF
WebLogic in Practice: SSL Configuration
Simon Haslam
 
PDF
WebLogic for DBAs
Simon Haslam
 
PDF
WildFly v9 - State of the Union Session at Voxxed, Istanbul, May/9th 2015.
Dimitris Andreadis
 
PDF
Emad Younis - Keynote
VMUG IT
 
PPTX
Nashville VMUG Keynote April 8 2015 - vSphere 6
Adam Eckerle
 
PDF
JBoss Enterprise Application Platform 6 Troubleshooting
Alexandre Cavalcanti
 
PDF
Devoxx 2013, WildFly BOF
Dimitris Andreadis
 
PPT
How to hack VMware vCenter server in 60 seconds
Positive Hack Days
 
PDF
WildFly BOF and V9 update @ Devoxx 2014
Dimitris Andreadis
 
PDF
WebLogic FAQs
Amit Sharma
 
PDF
JBoss EAP / WildFly, State of the Union
Dimitris Andreadis
 
PPT
J boss
jrfx448
 
PDF
WildFly AppServer - State of the Union
Dimitris Andreadis
 
PDF
Oracle WebLogic 11g Topology
Rakesh Gujjarlapudi
 
PDF
RHT Design for Security
virtualsouthwest
 
PDF
WildFly & WildFly Swarm
Dimitris Andreadis
 
PDF
An introduction to weblogic console
bispsolutions
 
Introduction to vSphere logs
Kaushik Chakraborty
 
WebLogic Administration course outline
Vybhava Technologies
 
Oracle WebLogic Diagnostics & Perfomance tuning
Michel Schildmeijer
 
WebLogic in Practice: SSL Configuration
Simon Haslam
 
WebLogic for DBAs
Simon Haslam
 
WildFly v9 - State of the Union Session at Voxxed, Istanbul, May/9th 2015.
Dimitris Andreadis
 
Emad Younis - Keynote
VMUG IT
 
Nashville VMUG Keynote April 8 2015 - vSphere 6
Adam Eckerle
 
JBoss Enterprise Application Platform 6 Troubleshooting
Alexandre Cavalcanti
 
Devoxx 2013, WildFly BOF
Dimitris Andreadis
 
How to hack VMware vCenter server in 60 seconds
Positive Hack Days
 
WildFly BOF and V9 update @ Devoxx 2014
Dimitris Andreadis
 
WebLogic FAQs
Amit Sharma
 
JBoss EAP / WildFly, State of the Union
Dimitris Andreadis
 
J boss
jrfx448
 
WildFly AppServer - State of the Union
Dimitris Andreadis
 
Oracle WebLogic 11g Topology
Rakesh Gujjarlapudi
 
RHT Design for Security
virtualsouthwest
 
WildFly & WildFly Swarm
Dimitris Andreadis
 
An introduction to weblogic console
bispsolutions
 
Ad

Similar to SSO 5.1 start-up issues (20)

PDF
WebSphere Technical University: Top WebSphere Problem Determination Features
Chris Bailey
 
PPTX
Lean microservices through ahead of time compilation (Tobias Piper, Loveholid...
London Microservices
 
PPTX
Building WebLogic Domains With WLST
C2B2 Consulting
 
PPT
Python Deployment with Fabric
andymccurdy
 
PPT
SaltConf14 - Ben Cane - Using SaltStack in High Availability Environments
SaltStack
 
ODP
DB proxy server test: run tests on tens of virtual machines with Jenkins, Vag...
Timofey Turenko
 
PDF
Head First Zend Framework - Part 1 Project & Application
Jace Ju
 
PPTX
MySQL Audit using Percona audit plugin and ELK
YoungHeon (Roy) Kim
 
PPTX
PHP on Windows Azure
Microsoft Openness Japan
 
PPTX
A 2-2 php on windows azure
GoAzure
 
PDF
Blue Green Sitecore Deployments on Azure
Rob Habraken
 
PPTX
Deployment with Fabric
andymccurdy
 
PDF
Meetup Performance
Justin Cataldo
 
PDF
Meetup Performance
Greg Whalin
 
PDF
Deployment ibm connections - No Http Server required
Sarwar Javaid
 
PPT
SharePoint Topology
Information Technology
 
PPTX
Cloud computing 3702
Jess Coburn
 
PPTX
Sherlock Homepage - A detective story about running large web services - WebN...
Maarten Balliauw
 
PPTX
PowerShell-and-DSC-Enables-DSCDevOps-1.pptx
prabhatthunuguntla
 
PDF
Boris Stoyanov - Troubleshooting the Virtual Router - Run and Get Diagnostics
ShapeBlue
 
WebSphere Technical University: Top WebSphere Problem Determination Features
Chris Bailey
 
Lean microservices through ahead of time compilation (Tobias Piper, Loveholid...
London Microservices
 
Building WebLogic Domains With WLST
C2B2 Consulting
 
Python Deployment with Fabric
andymccurdy
 
SaltConf14 - Ben Cane - Using SaltStack in High Availability Environments
SaltStack
 
DB proxy server test: run tests on tens of virtual machines with Jenkins, Vag...
Timofey Turenko
 
Head First Zend Framework - Part 1 Project & Application
Jace Ju
 
MySQL Audit using Percona audit plugin and ELK
YoungHeon (Roy) Kim
 
PHP on Windows Azure
Microsoft Openness Japan
 
A 2-2 php on windows azure
GoAzure
 
Blue Green Sitecore Deployments on Azure
Rob Habraken
 
Deployment with Fabric
andymccurdy
 
Meetup Performance
Justin Cataldo
 
Meetup Performance
Greg Whalin
 
Deployment ibm connections - No Http Server required
Sarwar Javaid
 
SharePoint Topology
Information Technology
 
Cloud computing 3702
Jess Coburn
 
Sherlock Homepage - A detective story about running large web services - WebN...
Maarten Balliauw
 
PowerShell-and-DSC-Enables-DSCDevOps-1.pptx
prabhatthunuguntla
 
Boris Stoyanov - Troubleshooting the Virtual Router - Run and Get Diagnostics
ShapeBlue
 
Ad

Recently uploaded (20)

PDF
Addressing The Cult of Project Management Tools-Why Disconnected Work is Hold...
OnePlan Solutions
 
PPTX
Transform Your Business with a Software ERP System
Canada Software Company
 
PPTX
CHAPTER 12 - CYBER SECURITY AND FUTURE SKILLS (1) (1).pptx
AnujKumar530915
 
PPTX
history of c programming in notes for students .pptx
Vijayakumari829030
 
PPTX
Agentic AI Use Case- Contract Lifecycle Management (CLM).pptx
mhxyzzp
 
PPTX
CHAPTER 2 - PM Management and IT Context
KevinPutra14
 
PDF
Adobe Illustrator 28.6 Crack My Vision of Vector Design
ghrom2211g
 
PDF
Design an Analysis of Algorithms II-SECS-1021-03
DilanEscobar1
 
PDF
T3DD25 TYPO3 Content Blocks - Deep Dive by André Kraus
Andre Kraus
 
PPTX
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
PPTX
Introduction to Artificial Intelligence
lohedi9363
 
PDF
Which alternative to Crystal Reports is best for small or large businesses.pdf
Varsha Nayak
 
PDF
Odoo Companies in India – Driving Business Transformation.pdf
azhuhardeen1968
 
PPTX
Oracle E-Business Suite: A Comprehensive Guide for Modern Enterprises
Conacent Consulting
 
PPTX
ManageIQ - Sprint 268 Review - Slide Deck
ManageIQ
 
PDF
Digital Strategies for Manufacturing Companies
Virendra Rai, PMP
 
PDF
Internet Downloader Manager (IDM) Crack 6.42 Build 41
ghrom2211g
 
PDF
AI in Product Development-omnex systems
omnex systems
 
PDF
Audit Checklist Design Aligning with ISO, IATF, and Industry Standards — Omne...
rohnjim07
 
PPT
Introduction Database Management System for Course Database
ReinertYosua
 
Addressing The Cult of Project Management Tools-Why Disconnected Work is Hold...
OnePlan Solutions
 
Transform Your Business with a Software ERP System
Canada Software Company
 
CHAPTER 12 - CYBER SECURITY AND FUTURE SKILLS (1) (1).pptx
AnujKumar530915
 
history of c programming in notes for students .pptx
Vijayakumari829030
 
Agentic AI Use Case- Contract Lifecycle Management (CLM).pptx
mhxyzzp
 
CHAPTER 2 - PM Management and IT Context
KevinPutra14
 
Adobe Illustrator 28.6 Crack My Vision of Vector Design
ghrom2211g
 
Design an Analysis of Algorithms II-SECS-1021-03
DilanEscobar1
 
T3DD25 TYPO3 Content Blocks - Deep Dive by André Kraus
Andre Kraus
 
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
Introduction to Artificial Intelligence
lohedi9363
 
Which alternative to Crystal Reports is best for small or large businesses.pdf
Varsha Nayak
 
Odoo Companies in India – Driving Business Transformation.pdf
azhuhardeen1968
 
Oracle E-Business Suite: A Comprehensive Guide for Modern Enterprises
Conacent Consulting
 
ManageIQ - Sprint 268 Review - Slide Deck
ManageIQ
 
Digital Strategies for Manufacturing Companies
Virendra Rai, PMP
 
Internet Downloader Manager (IDM) Crack 6.42 Build 41
ghrom2211g
 
AI in Product Development-omnex systems
omnex systems
 
Audit Checklist Design Aligning with ISO, IATF, and Industry Standards — Omne...
rohnjim07
 
Introduction Database Management System for Course Database
ReinertYosua
 

SSO 5.1 start-up issues

  • 1. © 2014 VMware Inc. All rights reserved. Troubleshooting SSO 5.1 Start-Up issues Frank Buechsel
  • 2. Agenda – SSO architecture – Symptoms of SSO not running or malfunctioning – Suggested troubleshooting procedure – Case studies – Useful command line options
  • 4. SSO 5.1 architecture • Status for the actual java wrapper • Does not reflect the status of the 4 sub systems • IMS (https://localhost:7444/ims/STSService?wsdl) • Lookupservice (https://localhost:7444/lookupservice/sdk) • ROOT (vFabric) (https://localhost:7444) • SSO-Adminserver (https://localhost:7444/sso-adminserver/sdk)
  • 5. Symptoms of SSO not running or malfunctioning
  • 6. Symptoms of SSO not running or malfunctioning 1/3 • vCenter Server unable to start vpxd.log: 2013-05-19T02:45:27.312-07:00 [04628 info '[SSO][CreateSsoFacade]'] [CreateUserDirectory] STS URI set to: https://vCenter.newhire.local:7444/ims/STSService?wsdl 2013-05-19T02:45:27.312-07:00 [04628 info '[SSO][CreateSsoFacade]'] [CreateUserDirectory] Admin URI set to: https://vCenter.newhire.local:7444/sso-adminserver/sdk 2013-05-19T02:45:27.312-07:00 [04628 info '[SSO][CreateSsoFacade]'] [CreateUserDirectory] Groupcheck URI set to: https://vCenter.newhire.local:7444/sso-adminserver/sdk 2013-05-19T02:45:27.555-07:00 [04628 error 'Default'] Found dangling SSL error: [0] error:00000001:lib(0):func(0):reason(1) 2013-05-19T02:45:27.555-07:00 [04628 error 'Default'] Found dangling SSL error: [1] error:00000001:lib(0):func(0):reason(1) 2013-05-19T02:45:27.555-07:00 [04628 error '[SSO][SsoFactory_CreateFacade]'] Unable to create SSO facade: vmodl.fault.SystemError. 2013-05-19T02:45:27.555-07:00 [04628 error 'vpxdvpxdMain'] [Vpxd::ServerApp::Init] Init failed: Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr) --> Backtrace: […] 2013-05-19T02:45:27.556-07:00 [04628 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...
  • 7. Symptoms of SSO not running or malfunctioning 2/3 • No login possible in the Web Client / vSphere Client
  • 8. Symptoms of SSO not running or malfunctioning 3/3 • Installation of dependent services failing (Inventory Service, vCenter Server, Web Client)
  • 10. Suggested troubleshooting procedure 1/3 • Check in services.msc if the service is running • Remember that this only referes to the wrapper and tomcat • Check for used ports • Look at the most recent catalina.log • Look at the most recent localhost.log • Look at the imsSystem.log • Look at the config.txt • Look at the database
  • 11. Suggested troubleshooting procedure 2/3 Java exceptions are hard to read without being filtered for useful information • Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Cannot open database "RSA" requested by the login. The login failed.) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:286) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:993) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:897) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291) at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:626) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:407) at com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext.access$001(SecurityAwareClassPathXmlApplicationContext.java:27) at com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext$1.run(SecurityAwareClassPathXmlApplicationContext.java:164) at com.rsa.ims.security.spi.SimpleSecurityContextImpl.doAs(SimpleSecurityContextImpl.java:82) at com.rsa.security.SecurityContext.doAs(SecurityContext.java:419) at com.rsa.security.SecurityContext.doAsSystem(SecurityContext.java:461) at com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext.refresh(SecurityAwareClassPathXmlApplicationContext.java:161) at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139) at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:105) at com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext.<init>(SecurityAwareClassPathXmlApplicationContext.java:90) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:126) ... 29 more Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Cannot open database "RSA" requested by the login. The login failed.) at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:141) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:105) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:278) ... 57 more
  • 12. Suggested troubleshooting procedure 3/3 • Log bundle analysis the manual way (will always read in the latest log) • $ ls -tr catalina* | tail -n 1 | xargs awk -F "INFO " '{print $2}' | tail -n 4 | grep Starting | wc -l Expected output would be “4” • $ ls -tr localhost.* | tail -n 1 | xargs cat | egrep '(Caused)' Expected output would be nothing • $ egrep '(Caused)' imsSystem.log Expected output would be nothing • $ cat config.txt | grep -F db. | tail -n 11 | grep com.rsa (IMS component configuration) • $ cat config.txt | grep -F db. | tail -n 11 | grep " db" (lookupservice db configuration) Expected output would be matching information
  • 14. Case studies – Normal start-up 1/3 No exceptions in catalina.log, localhost.log and imsSystem.log catalina.log
  • 15. Case studies – Normal start-up 2/3 localhost.log imsSystem.log
  • 16. Case studies – Normal start-up 3/3 IMS db configuration $ less config.txt | grep -F db. | tail -n 11 | grep " db" [2013-05-18 10:14:21,956 INFO StaticDataReporter com.vmware.vim.ssoconfig] db.url=jdbc:jtds:sqlserver://; serverName=;instance=VCSQL;databaseName=RSA [2013-05-18 10:14:21,956 INFO StaticDataReporter com.vmware.vim.ssoconfig] db.user=RSA_USER [2013-05-18 10:14:21,957 INFO StaticDataReporter com.vmware.vim.ssoconfig] db.type=Mssql [2013-05-18 10:14:21,985 INFO StaticDataReporter com.vmware.vim.ssoconfig] db.host=VCENTER Lookupservice db configuration $ less config.txt | grep -F db. | tail -n 11 | grep com.rsa [2013-05-18 10:14:21,953 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.domain= [2013-05-18 10:14:21,953 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.instance=RSA [2013-05-18 10:14:21,953 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.type=MSSQL [2013-05-18 10:14:21,953 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.msserverinstance=VCSQL [2013-05-18 10:14:21,953 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.hostname=VCENTER [2013-05-18 10:14:21,955 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.port=0
  • 17. Case studies – DB host not reachable • $ ls catalina* | tail -n 1 | xargs awk -F "INFO " '{print $2}' | tail -n 4 | grep Starting | wc –l 0 $ ls localhost.* | tail -n 1 | xargs cat | egrep '(Caused)‘ Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ims' defined in class path resource [beanRefContext.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Network error IOException: Connection refused: connect) Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Network error IOException: Connection refused: connect) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Network error IOException: Connection refused: connect) Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Network error IOException: Connection refused: connect) Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Network error IOException: Connection refused: connect) Caused by: java.sql.SQLException: Network error IOException: Connection refused: connect Caused by: java.net.ConnectException: Connection refused: connect Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException Caused by: java.lang.reflect.InvocationTargetException Caused by: java.lang.IllegalStateException: ComponentUtils not initialized Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vim.lookup.impl.DbStorage]: Constructor threw exception; nested exception is com.vmware.vim.lookup.exception.StorageException: Error executing statement Caused by: com.vmware.vim.lookup.exception.StorageException: Error executing statement Caused by: java.sql.SQLException: Network error IOException: Connection refused: connect Caused by: java.net.ConnectException: Connection refused: connect
  • 18. Case studies – RSA database offline • $ ls catalina* | tail -n 1 | xargs awk -F "INFO " '{print $2}' | tail -n 4 | grep Starting | wc –l 0 • $ ls localhost.* | tail -n 1 | xargs cat | egrep '(Caused)‘ Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ims' defined in class path resource [beanRefContext.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Cannot open database "RSA" requested by the login. The login failed.) Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Cannot open database "RSA" requested by the login. The login failed.) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Cannot open database "RSA" requested by the login. The login failed.) Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Cannot open database "RSA" requested by the login. The login failed.) Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Cannot open database "RSA" requested by the login. The login failed.) Caused by: java.sql.SQLException: Cannot open database "RSA" requested by the login. The login failed. Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException Caused by: java.lang.reflect.InvocationTargetException Caused by: java.lang.IllegalStateException: ComponentUtils not initialized Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vim.lookup.impl.DbStorage]: Constructor threw exception; nested exception is com.vmware.vim.lookup.exception.StorageException: Error executing statement Caused by: com.vmware.vim.lookup.exception.StorageException: Error executing statement Caused by: java.sql.SQLException: Cannot open database "RSA" requested by the login. The login failed.
  • 19. Case studies – Expired RSA_USER password • $ ls catalina* | tail -n 1 | xargs awk -F "INFO " '{print $2}' | tail -n 4 | grep Starting | wc –l 0 • $ ls localhost.* | tail -n 1 | xargs cat | egrep '(Caused)‘ Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ims' defined in class path resource [beanRefContext.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'RSA_USER'. Reason: The password of the account must be changed.) Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'RSA_USER'. Reason: The password of the account must be changed.) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'RSA_USER'. Reason: The password of the account must be changed.) Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'RSA_USER'. Reason: The password of the account must be changed.) Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'RSA_USER'. Reason: The password of the account must be changed.) Caused by: java.sql.SQLException: Login failed for user 'RSA_USER'. Reason: The password of the account must be changed. Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException Caused by: java.lang.reflect.InvocationTargetException Caused by: java.lang.IllegalStateException: ComponentUtils not initialized Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vim.lookup.impl.DbStorage]: Constructor threw exception; nested exception is com.vmware.vim.lookup.exception.StorageException: Error executing statement Caused by: com.vmware.vim.lookup.exception.StorageException: Error executing statement Caused by: java.sql.SQLException: Login failed for user 'RSA_USER'. Reason: The password of the account must be changed.
  • 20. Case studies – Incorrect RSA_USER password • $ ls catalina* | tail -n 1 | xargs awk -F "INFO " '{print $2}' | tail -n 4 | grep Starting | wc –l 0 • $ ls localhost.* | tail -n 1 | xargs cat | egrep '(Caused)‘ Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ims' defined in class path resource [beanRefContext.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'RSA_USER'.) Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.components.spring.SecurityAwareClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'RSA_USER'.) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DatabaseMetadataBean' defined in class path resource [ims-components-common.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'RSA_USER'.) Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.rsa.ims.common.DatabaseMetadataBean]: Constructor threw exception; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'RSA_USER'.) Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'RSA_USER'.) Caused by: java.sql.SQLException: Login failed for user 'RSA_USER'. Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException Caused by: java.lang.reflect.InvocationTargetException Caused by: java.lang.IllegalStateException: ComponentUtils not initialized Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vim.lookup.impl.DbStorage]: Constructor threw exception; nested exception is com.vmware.vim.lookup.exception.StorageException: Error executing statement Caused by: com.vmware.vim.lookup.exception.StorageException: Error executing statement Caused by: java.sql.SQLException: Login failed for user 'RSA_USER'.
  • 21. Case studies – Incorrect db information after db move • $ ls catalina* | tail -n 1 | xargs awk -F "INFO " '{print $2}' | tail -n 4 | grep Starting | wc –l 0 • $ ls localhost.* | tail -n 1 | xargs cat | egrep '(Caused)‘ Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vim.lookup.impl.DbStorage]: Constructor threw exception; nested exception is com.vmware.vim.lookup.exception.StorageException: Error executing statement Caused by: com.vmware.vim.lookup.exception.StorageException: Error executing statement Caused by: java.sql.SQLException: Unable to get information from SQL Server: VCENTER2. • $ less config.txt | grep -F db. | tail -n 11 | grep com.rsa [2013-05-18 11:30:10,664 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.domain= [2013-05-18 11:30:10,664 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.instance=RSA [2013-05-18 11:30:10,664 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.type=MSSQL [2013-05-18 11:30:10,664 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.msserverinstance=VCSQL [2013-05-18 11:30:10,664 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.hostname=VCENTER [2013-05-18 11:30:10,665 INFO StaticDataReporter com.vmware.vim.ssoconfig] com.rsa.db.port=0 • $ less config.txt | grep -F db. | tail -n 11 | grep " db“ [2013-05-18 11:30:10,665 INFO StaticDataReporter com.vmware.vim.ssoconfig] db.url=jdbc:jtds:sqlserver://;serverName=;instance=VCSQL;databaseName=RSA [2013-05-18 11:30:10,665 INFO StaticDataReporter com.vmware.vim.ssoconfig] db.user=RSA_USER [2013-05-18 11:30:10,665 INFO StaticDataReporter com.vmware.vim.ssoconfig] db.type=Mssql [2013-05-18 11:30:10,665 INFO StaticDataReporter com.vmware.vim.ssoconfig] db.host=VCENTER2
  • 22. Case studies – DC unavailable • $ ls catalina* | tail -n 1 | xargs awk -F "INFO " '{print $2}' | tail -n 4 | grep Starting | wc –l 4 • $ ls localhost.* | tail -n 1 | xargs cat | egrep '(Caused)‘ • $ egrep '(Caused)' imsSystem.log Caused by: javax.naming.CommunicationException: DC.newhire.local:3268 [Root exception is java.net.ConnectException: Connection timed out: connect] Caused by: java.net.ConnectException: Connection timed out: connect Caused by: javax.resource.spi.ResourceAdapterInternalException: Unable to create a managed connection 'ldap://DC.newhire.local:3268' with 'newhireadministrator' Reason: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection DC.newhire.local:3268 Caused by: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection DC.newhire.local:3268 Caused by: javax.naming.CommunicationException: DC.newhire.local:3268 [Root exception is java.net.ConnectException: Connection timed out: connect] Caused by: java.net.ConnectException: Connection timed out: connect Caused by: javax.naming.NamingException: getInitialContext failed. javax.resource.spi.ResourceAdapterInternalException: Unable to create a managed connection 'ldap://DC.newhire.local:3268' with 'newhireadministrator' Reason: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection DC.newhire.local:3268 [Root exception is javax.resource.spi.ResourceAdapterInternalException: Unable to create a managed connection 'ldap://DC.newhire.local:3268' with 'newhireadministrator' Reason: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection DC.newhire.local:3268] Caused by: javax.resource.spi.ResourceAdapterInternalException: Unable to create a managed connection 'ldap://DC.newhire.local:3268' with 'newhireadministrator' Reason: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection DC.newhire.local:3268 Caused by: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection DC.newhire.local:3268 Caused by: javax.naming.CommunicationException: DC.newhire.local:3268 [Root exception is java.net.ConnectException: Connection timed out: connect] Caused by: java.net.ConnectException: Connection timed out: connect
  • 24. Useful Command Line options 1/8 • Autodiscovery after installation is finished • ssocli configure-riat -a discover-is -u admin --verbose
  • 25. Useful Command Line options 2/8 • List identity sources (useful as this information is not collected in an SSO log bundle) • ssocli manage-identity-sources -a list -u admin
  • 26. Useful Command Line options 3/8 • Delete an identity source • ssocli manage-identity-sources -a delete -u admin -g <ims.id>
  • 27. Useful Command Line options 4/8 • Create an identity source • ssocli manage-identity-sources -a create -u admin
  • 28. Useful Command Line options 5/8 • Change the password for SSO admins and the SSO master password • ssocli reset-admin-password • ssocli manage-secrets -a change
  • 29. Useful Command Line options 6/8 • Unlock the system after too many system changes • ssocli manage-secrets -a recover
  • 30. Useful Command Line options 7/8 • Testing database connectivity without SQL Management Studio • sqlcmd -S hostnameinstance -U RSA_USER -d RSA
  • 31. Useful Command Line options 8/8 • Silent uninstall of SSO • msiexec /qn /x {DEC4C346-414B-4814-9BF3-CAC14154B55A} MASTER_PASSWORD <master password>

Editor's Notes

  • #5: IMS and Lookupservice are dependent on database connection to be able to start.
  • #28: It is not possible to edit an identity source using the command line