VMware App Volumes Troubleshooting
Download as PPTX, PDF14 likes26,545 views
The document provides information about troubleshooting VMware App Volumes, including where to find relevant log files and events. It discusses log files from the App Volumes Manager, App Volumes Agent, ESXi host, and SQL database. It also provides examples of issues like the SQL database becoming unreachable, Active Directory becoming unreachable, and the vCenter server becoming unreachable. The document describes how to use the Ruby console and PowerShell to help troubleshoot the App Volumes Manager.
![App Volumes Manager PowerShell
• Initiate session
– $url="http://appvol01.corp.itbubble.ru"
– $person = @{'user[account_name]'=‘admin'; 'user[password]'='P@ssw0rd'}
– Invoke-RestMethod -Uri $url/login -Method POST -SessionVariable session -Body
$person
• Get version
– Invoke-RestMethod -Uri $url/login -Method POST -SessionVariable session -Body
$person
• Get online users
– (Invoke-RestMethod -Uri $url/cv_api/online_entities -WebSession $session -Method
GET).online.records
• Get configuration
– (Invoke-RestMethod -Uri $url/cv_api/summary -WebSession $session -Method
GET).current_configuration
38](https://image.slidesharecdn.com/vmwareappvolumes-troubleshooting-150723154427-lva1-app6892/85/VMware-App-Volumes-Troubleshooting-37-320.jpg)
![AppVolumes PowerShell Module Preview
• Examples:
• Get-AppVolAppStack [-All] - Returns all available appstacks
• Get-AppVolAppStack -AppStackIds 88,19 – return appstacks with IDs 88 and 19
• 88,19|Get-AppVolAppStack – return appstacks with IDs 88 and 19 thru the pipe
• Get-AppVolAppStack -Name office – returns all appstacks where the name contains “office”
• Get-AppVolAppStack -Name office -Not - returns all appstacks where the name NOT contains “office”
• Get-AppVolAppStack -Name office -Exact – all apstacks where the name is exactly “office”
• Get-AppVolAppStack -Path "cloudvolumes" – appstacks that has “cloudvolumes” in the datastore path
• Get-AppVolAppStack -DataStore iSCSI -Exact – appstacks located on datastore iSCSI
• Get-AppVolAppStack -FileName office – appstacks where vmdk name contains word office
• Get-AppVolAppStack -CreatedAt "4/28/2015" –ge – appstacks created after or on 4/28/2015
• Get-AppVolAppStack -CreatedAt "4/28/2015" -gt appstacks created after but not on 4/28/2015
• Get-AppVolAppStack -MountedAt $((get-date).AddDays(-30)) -ge -Not - appstacks not mounted in past 30 days
• Get-AppVolAppStack -TemplateVersion "2.5.1" – appstacks with template version 2.5.1
• Get-AppVolAppStack -AssignmentsTotal 2 -ge - appstacks that have 2 or more assignments
•](https://image.slidesharecdn.com/vmwareappvolumes-troubleshooting-150723154427-lva1-app6892/85/VMware-App-Volumes-Troubleshooting-40-320.jpg)
VMware App Volumes Troubleshooting
- 1. © 2015 VMware Inc. All rights reserved. VMware App Volumes Troubleshooting Denis Gundarev Sr. MTS/Application Architect, App Volumes R&D
- 2. About me C:>whoami /all USER INFORMATION ---------------- User Name Twitter E-Mail ================ ============ ================== VMWAREdgundarev @fdwl DGundarev@vmware.com GROUP INFORMATION ----------------- Group Name Type SID ======================================== ================ ================= BUILTINGeeks Mandatory group S-1-5-32-540 Mandatory LabelCrazy Russian Label S-1-16-8192 VMWAREApp Volumes R&D Mandatory group S-1-5-32-544
- 3. App Volumes Dynamic Delivery Benefits Dynamic delivery of managed application containers in seconds. Real time native application and data delivery. Agility – Logically manage application sets based on business needs. – Deliver or upgrade application sets across all VMs in seconds. Simplicity – Integrate into existing infrastructure in minutes. – Provision applications as easily as installing them. Flexibility – Persistent user experience with non-persistent economics. – Works with VMware Horizon 6 with View and Citrix XenApp 6.5. Efficiency – Optimize use of storage, SAN IOPS, and network.
- 4. App Volumes • Feb 3, 2015 - VMware App Volumes v2.5 – Horizon View integration – Multiple templates for writable volumes • Mar 12, 2015 - VMware App Volumes v2.6 – AppStack Grouping – Storage Groups • Apr 28, 2015 - VMware App Volumes 2.7 – One-way AD trusts – Dynamic VHD permissions – XenDesktop 7.x support • Jun 15, 2015 - VMware App Volumes 2.9 – Support for application deployment to physical machines – Multi vCenter Configurations
- 5. App Volumes components App Volumes Manager App Volumes Agent AppStack volume(s) • Console for assignments and configuration • Broker for App Volume Agent for the assignment of applications and writable volumes • File system and registry abstraction layer running on the target system • Virtualizes file system writes as appropriate (when used with optional writable volume) • Read-only volume containing applications • Can map more than one AppStack per user, target • Deploy apps to VDI or RDSH Writable volume • Per user read-write volume used to persist changes written in the session • One writable volume per user
- 6. vCenter App Volumes Architecture vSphere ESXi SOAP App Volumes Manager NGINX HTTP/HTTPS (80/443) HTTPS (443) hostd SQL Database Virtual Machine Minifilter driver Ruby svmanager TCP 1433 HTTPS (443) Domain Controller LDAP (389) / LDAPS (636) Connection Server Broker ServiceHTTP/HTTPS (80/443) HTTP Manager (Web Interface) API Scheduler Workers Service Base OS AppStack AppStack
- 7. App Volumes Operation Mode • VMDK Direct Attached Operation Mode – preferred operations mode for App Volumes – AppStacks and Writable Volumes are stored within a hypervisor datastores – VMDK file format – attached to the virtual machine using standard hypervisor functionality. • VHD In-Guest Operation Mode – AppStacks and Writable Volumes are stored on the (CIFS) file share – VHD file format – attached to the target computer using OS functionality – Works with physical and other hypervisors
- 8. VHD In-Guest • VHD AppStacks mounted inside the target computer, similar to disk manager command • Target computer should have access to the file share • VHD file permissions can be dynamically controlled by the manager • Works with physical and third-party hypervisors
- 9. Hypervisor Connection Type • Connection to VMware vSphere vCenter – preferred connection type for most environments – VMDK Direct Attached Operation Mode • ESX (Single Host) Connection – VMDK Direct Attached Operation Mode – works only for a single hypervisor host – use for small deployments and PoC • VHD In-Guest – disables hypervisor connection – enables use of VHD In-Guest Operation Mode CONFIDENTIAL 9
- 10. App Volumes - Scalability • Observe vSphere 5.5 Maximums – 2048 virtual disks per host – 2048 powered-on virtual machines per VMFS volume – 60 SCSI devices per virtual machine • Recommended Practice: – 1 AppStack per 2,000 virtual machines – Up to 20 AppStack volumes per VM – 10,000 Agents per App Volume Manager • Core applications – Part of the base image or single AppStack – AppStack datastore for every 2,000 users
- 11. AppStack • App Volumes is not ThinApp Alternative – “ghost” for applications – Works with ThinApp or App-V • The contents of the system drive (C:) located under SVROOT in the VHD. – Only file system entries that had contents changed / updated or security information changed / updated. – The files are ordered in the same directory tree as in the source machine – All the file system entries have the original security information / attributes. • The registry changes are located in snapvol.dat and have a valid Windows registry hive formatting. – The changes / updates to the SOFTWARE hive are under MACHINESOFTWARE key in the hive. – The changes / updates to the SYSTEM hive are under MACHINESYSTEM key in the hive. – The changes / updates to the DEFAULT hive are not captured as part of the VHD.
- 14. App Volumes Provisioning Volumes Workflow Target System App Volumes Manager vCenter 1. Configuration of AppStacks 2. Mount Command sent to vCenter - Target system - VMDK information - DataStore Information 3. Target system receive AppStacks 1 2 3
- 15. Login App Volumes Agent Events Logout Startup Shutdown 43 21
- 16. App Volumes Agent Startup App Volumes Manager Startup 1. App Volumes Agent checks into App Volumes Manager 2. App Volumes Manager provides license and machine assignments 2 App Volumes Agent 1
- 17. App Volumes Agent Shutdown App Volumes Manager Shutdown 1. App Volumes Agents checks into App Volumes Manager 2. User based assignments are detached 2 App Volumes Agent 1
- 18. App Volumes Agent Login App Volumes Manager Login 1. App Volumes Agent checks into App Volumes Manager 2. App Volumes Agent checks for pending attachments/un-attachments 3. App Volumes Agent checks for machine based attachments – if present then no user based attachments are honored 4. App Volumes Manager checks database for logged in user assignments 5. Attach assigned volumes - Writable then AppStack(s) 2 App Volumes Agent 1 Database 3 4 5
- 19. App Volumes Agent Logout App Volumes Manager Logout 1. App Volumes Agents checks into App Volumes Manager 2. User based assignments are detached 2 App Volumes Agent 1
- 20. App Volumes Provisioning • Start with Clean VM image • Create AppStack in App Volumes Manager • Choose Provisioning VM • VMDK gets attached to Provisioning VM • Install application • Complete AppStack creation • VM restarts, VMDK is detached • Ready to assign AppStack
- 21. Scripts Script name Execution condition prestartup.bat Called when a volume is dynamically attached or on during system startup but before virtualization is activated startup.bat Called when a volume is dynamically attached or on during system startup startup_postsvc.bat Called as and called after services have been started on the volume (not called if there are no services on volume) logon.bat Called at logon and before Windows Explorer starts logon_postsvc.bat Called after services have been started (not called if there are no services on volume) shellstart.bat Called when a volume is dynamically attached or when Windows Explorer starts. shellstop.bat Called when the user is logging off before Windows Explorer is terminated. logoff.bat Called at logoff and Windows Explorer has terminated shutdown_presvc.bat Called when the computer is being shutdown before services have been stopped. shutdown.bat Called when the computer is being shutdown after services have been stopped. allvolattached.bat Called after all volumes have been processed (so if user has 3 AppStack, this will be called after all 3 have loaded) post_prov.bat Called at the end of provisioning to do any one-time steps that should be performed at the end of provisioning. Invoked at the point of clicking the provisioning complete pop-up while the volume is still virtualized. prov_p2.bat Invoked at phase 2 of the provisioning process. After the machine has rebooted but before App Volumes Manager has been notified that provisioning has completed. This is the last chance to perform any actions on the provisioned volume with virtualization disabled.
- 22. Troubleshooting
- 23. App Volumes – Agent Log files • App Volumes Agent – C:Windowssvservice.log – SvService responsible for communication with App Volumes Manager, preparing volume, running post-attach scripts, refreshing variables, registering fonts • App Volumes Agent – C:WindowsSystem32LogFilesWMI AppVolumesAgent.etl – Minifilter drivers for NTFS – Policy driven; the files, directories, registry keys and processes that are virtualized are governed by the policy file snapvol.cfg on each volume. • App Volumes Agent svcapture.log – C:Windowssvcapture.log – Perform the provisioning/editing functions, generate policy files, metadata and report it to Agent/Manager.
- 24. Reading SVDriver log • Converting an ETL file to xml format – Flush buffers to disk • logman update AppVolumesAgent -fd –ets – Convert log to xml • tracerpt.exe AppVolumesAgent.etl -of xml -gmt -tp “C:Program Files (x86)CloudVolumesAgenttmf” -o AppVolumesAgent.xml – Open AppVolumesAgent.xml using Excel • Event Viewer: – Copy TMF files from C:Program Files (x86)CloudVolumesAgenttmf to C:windowsSystem32WinevtTraceFormat – Open "C:windowsSystem32LogFilesWMIAppVolumesAgent.etl“ using windows Event Viewer
- 26. App Volumes Manager – Production Log • The Production.log file can help identify issues • You can view the Production.log at http://<App Volumesmgr>:8080/log • The default log level is set to ‘info’. • Logfile is also located at: C:Program Files (x86)CloudVolumesManagerLog production.log
- 27. ESXi Logs • ESXi Host Management Service Log (hostd) /var/log/hostd.log – Watch for disk locking, VM state transition, hot-add operations • Virtual Machine Log /vmfs/volumes/<datastore>/<vm_name>/vmware.log
- 28. • Debug provides SQL logging App Volumes Manager – Increasing Logging Level • We can increase the logging level to debug, by editing: C:Program Files(x86)CloudVolumesManagerconfiglog4r.yml
- 29. Example 1: Provision a new App Stack CONFIDENTIAL 30 Template VMDK copied to new App Stack VMDK UUID of clean machine to mount App Stack for provisioning App Volumes module initializes capture, RvSphere module mounts volume Operation: New App Stack ‘Office2010_RTM’ created Where to Look: App Volumes Manager Log C:Program Files (x86)CloudVolumesManagerlogproduction.log
- 30. Example 1: Provision a new App Stack (continued) CONFIDENTIAL 31 VM State transitions during operation: VM_STATE_ON VM_STATE_RECONFIGURED VM_STATE_RECONFIGURING VM_STATE_ON. Hot Add completed and App Stack mounted on VM Operation: New App Stack ‘Office2010_RTM’ created Where to Look: ESXi hostd.Log: /var.log.hostd.log Relevant Log entries: Hot-Add, VM transition, disk locking.
- 31. CONFIDENTIAL 32 Example 2: SQL Database Unreachable Behaviour: Manager: UI is unavailable – CVManager.exe service stays up, attempting reconnection. ODBC error in Manager.log. Agent: Where user remains logged in - App Stacks / Writables work as normal For new logins - App Stacks / Writables work as normal Issue: SQL Server Database is unreachable by App Volumes Manager(s) Extract from App Volumes Manager.log Where to Look: App Volumes Manager Log C:Program Files (x86)CloudVolumesManagerlogproduction.log Relevant Module entries: ‘ODBC’
- 32. CONFIDENTIAL 33 Example 3: AD Unreachable Issue: Active Directory Domain Services unreachable (including DNS) Extract from App Volumes Manager.log Behaviour: Manager: UI is functional, but AD queries fail – error in Manager.log Agent: Where user remains logged in - App Stacks / Writables work as normal For new logins - App Stacks / Writables unavailable Where to Look: App Volumes Manager Log C:Program Files (x86)CloudVolumesManagerlogproduction.log Relevant Module entries: ‘RADIR’
- 33. CONFIDENTIAL 34 Example 4: vCenter Unreachable Issue: vCenter unreachable (Note: Mount on ESXi host option was set during install) Extract from App Volumes Manager.log Behaviour: Manager: UI is partially functional, but error on querying datastores. Unable to create new App Stacks/Writables Agent: App Stacks / Writables fully functional Where to Look: App Volumes Manager Log C:Program Files (x86)CloudVolumesManagerlogproduction.log Relevant Module entries: ‘RvSphere’
- 34. App Volumes Manager – Ruby Console • The Ruby on Rails Console can be used to troubleshoot, execute queries and run scripts – Common use to examine internal objects not visible in the UI • Open command prompt as Administrator on App Volumes Manager machine • Navigate to: C:Program Files (x86)CloudVolumesManager Enter the command: rubybinruby.exe scriptrails console 35 Any ruby command should be used with CAUTION ! ! !
- 35. App Volumes Manager – Ruby Console • List registered App Volume Machines: – ap Machine.all • Show configuration – ap Svconfiguration.current • Delete pending activities – Delayed::Job.destroy_all • Search for machines – ap Machine.where('name LIKE ?', "PVS%") • Delete all disabled computers – ap Computer.where("disabled_at IS NOT NULL").delete_all 36
- 36. App Volumes Manager – Ruby Console • List admin groups – ap GroupPermission.all • Delete all administrators – GroupPermission.delete_all • Add administrator group – Permission.named(:admin).add_group_name("CN=Domain Admins,OU=Groups, DC=test,DC=com") 37
- 37. App Volumes Manager PowerShell • Initiate session – $url="http://appvol01.corp.itbubble.ru" – $person = @{'user[account_name]'=‘admin'; 'user[password]'='P@ssw0rd'} – Invoke-RestMethod -Uri $url/login -Method POST -SessionVariable session -Body $person • Get version – Invoke-RestMethod -Uri $url/login -Method POST -SessionVariable session -Body $person • Get online users – (Invoke-RestMethod -Uri $url/cv_api/online_entities -WebSession $session -Method GET).online.records • Get configuration – (Invoke-RestMethod -Uri $url/cv_api/summary -WebSession $session -Method GET).current_configuration 38
- 38. AppVolumes PowerShell Module Preview • Open-AppVolSession • Test-AppVolSession • Close-AppVolSession • Get-AppVolAppStack • Get-AppVolAppStackFile • Get-AppVolAssignment • Get-AppVolComputer • Get-AppVolGroup • Get-AppVolOrgUnit • Get-AppVolUser • Get-AppVolVersion
- 39. AppVolumes PowerShell Module Preview • Open-AppVolSession -Uri http://manager.domain.com -Username admin -Password password1 • Or you can omit the parameter names: • Open-AppVolSession http://manager.domain.com admin password1 •
- 40. AppVolumes PowerShell Module Preview • Examples: • Get-AppVolAppStack [-All] - Returns all available appstacks • Get-AppVolAppStack -AppStackIds 88,19 – return appstacks with IDs 88 and 19 • 88,19|Get-AppVolAppStack – return appstacks with IDs 88 and 19 thru the pipe • Get-AppVolAppStack -Name office – returns all appstacks where the name contains “office” • Get-AppVolAppStack -Name office -Not - returns all appstacks where the name NOT contains “office” • Get-AppVolAppStack -Name office -Exact – all apstacks where the name is exactly “office” • Get-AppVolAppStack -Path "cloudvolumes" – appstacks that has “cloudvolumes” in the datastore path • Get-AppVolAppStack -DataStore iSCSI -Exact – appstacks located on datastore iSCSI • Get-AppVolAppStack -FileName office – appstacks where vmdk name contains word office • Get-AppVolAppStack -CreatedAt "4/28/2015" –ge – appstacks created after or on 4/28/2015 • Get-AppVolAppStack -CreatedAt "4/28/2015" -gt appstacks created after but not on 4/28/2015 • Get-AppVolAppStack -MountedAt $((get-date).AddDays(-30)) -ge -Not - appstacks not mounted in past 30 days • Get-AppVolAppStack -TemplateVersion "2.5.1" – appstacks with template version 2.5.1 • Get-AppVolAppStack -AssignmentsTotal 2 -ge - appstacks that have 2 or more assignments •
- 41. AppVolumes PowerShell Module Preview • Get-AppVolAppStackFile -DataStore datastore1|Get-AppVolAppStack|Format- Table returns a table with all appstacks that have files on datastore1 • Get-AppVolAppStackFile -Reachable -Not -returns all unreachable files • Get-AppVolAppStack -Name office |Get-AppVolAssignment - get all assignments for appstacks that has “office” in the name • Get-AppVolAssignment -EntityDn "cn=users,dc=domain,dc=com" – assignments for users in specific OU • Get-AppVolAssignment -EntitySamAccountName denis -Exact – assignments for user “denis” •
- 42. App Volumes Agent - Registry Entries HKLMSoftwareCloudVolumesAgent HKLMSystemCurrentControlSetservicessvserviceparameters • vCenter / App Volumes Manager Hostname and Port • Additional App Volumes Managers – registry entries need to be added manually after standard install – Can be set during Msiexec install MANAGER_ADDR=test.company.c om MANAGER_PORT=80
- 43. App Volumes & MSOffice
- 44. Microsoft Office Support Basics • We don’t officially support scenarios that are not supported by Microsoft – Office 2007 – Mix of 64-bit and 32-bit apps • We only support: – Office Professional Plus – Office Standard – Office 365 ProPlus (installed from Office Professional Plus media) • Only media downloaded from Microsoft Volume Licensing Service Center (VLSC) can be used – No MSDN – No Retail/OEM – Verify if your installation media is correct: • You must have a folder named “Admin” on your ISO • Name of the folder with .WW extension (ProPlus.WW, Visio.WW, etc.) shouldn’t have a lowercase “r” before the dot - ProPlusr.WW, Visior.WW
- 45. Microsoft Office Activation • KMS should be used for most deployments • MAK activation can be used for persistent deployments starting with 2.7 • Retail, OEM, Office 365 activation not supported • Active Directory-Based Activation (Office 2013 on Windows 8) may work, but not tested • KMS DNS discovery preferred • Rearm Office after installation • If activation fails, verify that KMS activation works for a machine with no App Volumes agent
- 46. Microsoft Office Recommendations • Recommend Office in the base image • One AppStack for app Office Applications • Never launch Office Apps during provisioning • Rearm Office after installation • Add-ins/plug-ins must be on the same AppStack with Office • Recommend 32-bit editions • 32-bit and 64-bit editions should never meet, even from different AppStacks
- 47. Multiple Office Versions Within AppStack Office 2010 Office 2013 Project 2010 Project 2013 Visio 2010 Visio 2013 Office 2010 N/A Install Office 2010 first Supported Install Office 2010 first Supported Install Office 2010 first Office 2013 Install Office 2010 first N/A Install Project 2010 first Supported Install Visio 2010 first Supported Project 2010 Supported Install Project 2010 first N/A Install Project 2010 first Supported Install Project 2010 first Project 2013 Install Office 2010 first Supported Install Project 2010 first N/A Install Visio 2010 first Supported Visio 2010 Supported Install Visio 2010 first Supported Install Visio 2010 first N/A Install Visio 2010 first Visio 2013 Install Office 2010 first Supported Install Project 2010 first Supported Install Visio 2010 first N/A
- 48. Native Office Coexistence AppStack Office 2010 Office 2013 Project 2010 Project 2013 Visio 2010 Visio 2013 Nativelyinstalled Offic e 2010 Not Supported Supported except Outlook Supported Supported Supported Supported Offic e 2013 Not Supported Not Supported Not Supported Supported Not Supported Supported
Editor's Notes
- #6: Last AppStack wins in an application conflict. AppStack precedence in version 2.3 and above
- #7: SOAP listens on port 443 (Simple Object Access Protocol). App Volumes is essentially an orchestrator that calls SOAP to perform mounting functions. App Volumes Manager uses HTTPS (443) for communication between vCenter and ESXi hosts HTTP (80) Direct connections to hosts is optional LDAP (389) or LDAPS (636) App Volumes Manager uses TCP 1433 to SQL Can change the default Agent port of 80 to something else.
- #11: Last AppStack wins in an application conflict. No AppStack dependency management done today Install kernel mode drivers in base image Applications running when user is logged out should be installed in base image http://www.vmware.com/pdf/vsphere5/r55/vsphere-55-configuration-maximums.pdf
- #28: You don’t need to include the port number unless you specified a custom port at install Baretail can be used to troubleshoot logs and look for patterns. Manual process to log in to App Volumes Agent/Manager and investigate logs vRealize Log Insight 2.5.0 can automatically harvest the logs and store them centrally
- #44: If we have multiple App Volume Managers in our environment we can see what we are connected to here. msiexec /I "c:\Temp\CUSTOM - App Volumes Agent.msi" MANAGER_ADDR=test.company.com MANAGER_PORT=80 [16/12/2014 19:29:21] Rasmus Jensen: as an example [16/12/2014 19:29:33] Rasmus Jensen: MANAGER_ADDR and MANAGER_PORT is what you are looking fo