Mitigating Common CloudStack Instance Deployment Failures

Editor's Notes

• #4: We are going to discuss 15 common instance deployment failure scenarios, their usual causes, and suggested mitigation steps.
• #5: If you have used cloudstack already you must have seen this error. Insufficient capacity is a generic error thrown for many failures and obviously when there is no available capacity as well. Most of the time the capacity is fully utilized and the ways to resolve are straightforward. We could add more compute or storage resources depending on the situation. If there is an opportunity to delete some unused resources such as instances or volumes that could also help. Another way to handle this situation is with efficient use of resource limits at account, domain levels.
• #6: When you are out of public IPs and the new network requires a public IP the instance deployment could fail. The way around this situation is mostly by adding more public IPs. To avoid getting into this situation we could use resource limits efficiently. If you know the use case and the expected usage of public IPs upfront you could add the capacity accordingly. If you pay attention to the resource utilization you should be able to provision new IPs before it results in any failure.
• #7: When you deploy an instance where the network needs to be implemented a new VLAN needs to be allocated. There are situations where the provisioned VLAN IDs are fully consumed, this could also result in an instance deployment failure. The typical way around this is to extend the VLAN range. If you have used resource limits effectively, to some extent you may not see this issue that often. If the VLAN capacity is planned well you may not face this issue at all.
• #8: The deployment planner is unable to find host / storage matching the tag having enough capacity. We can solve this by reviewing the current host / storage tagging configuration and fixing them. We could avoid using tags if that’s not required. Or we could add tags to more resources.
• #9: Over provisioning has no magic to give us more than the capacity the infrastructure has. We need to use the over provisioning values realistically. If you do thorough testing you should be able to find the suitable values. Also keep some head room to avoid resource contentions.
• #10: We should avoid choosing mutually exclusive deployment choices such as selecting dedicated affinity group and selecting an offering have a host or storage that is not available in the dedicated resource. We can avoid this by selecting only the appropriate choices for instance deployment.
• #11: You might see this instance deployment error when hosts are unavailable due to any reason. It could hardware, network or software issues. This is also seen where there is no free capacity as well. You can avoid this by ensuring the host and its hardware health. Make sure there is free capacity. Implement any hypervisor monitoring.
• #12: I have seen this capability mismatch error during instance deployment after a server hardware upgrade or replacement. The older CPUs could be say 3500 MHZ and the compute offering were created based on them so we may have used CPU MHZ value of 3500. If the new CPU is of 2000MHz the instance deployment would fail. Its always good practice to revise the compute offering according to the hardware changes.
• #13: Although I suggested using resource limits as a solution earlier a couple of times, resource limits themselves could lead to instance deployment failure. If the instance resource limit is reached for either account , domain or project level the result is a failure. We do have much options in this case, we can either increase the limit or free up the resources.
• #14: Java Null pointer exceptions are another set of errors that could lead to any operation failure and instance deployment as well. Mostly these appear if you have done wrong DB changes. We hit this error caused by bugs as well. If we can avoid manual DB changes we will not be affected by this error most likely. If it’s a bug either we can find the fixes version and upgrade or report the bug or even fix it ourselves if that’s an option.
• #15: Any issues with database can can break the cloudstack installation. It is essential to keep it away from any errors. Issues such as DB server performance issues, JDBC errors, some wrong mysql statements etc are the most common ones leading to failures such as instance deployment failure. We need to monitor the DB server to avoid any service outages. We need to ensure good network connectivity to DB server. Also we can avoid some issues if we use supported and tested mysql versions.
• #16: Sometimes the instance deployment job succeeds but the guest OS fails to boot. This could happen due to incorrect guest OS mappings, or unsupported guest OS type. Unsupported controller type or corrupted template or ISO. If you access the console you should be able to tell what caused the issue. Depending on the cause you should be able to fix it by either fixing the guest OS mapping, using the correct controller types or by fixing the template.
• #17: Timeouts another common failure scenario. We could avoid this by increasing the timeout. We can also identify the sub tasks causing the delay and fix it. Most of times the underlysing platform is slow, its worth reviewing it fixing it solves the problem.
• #18: Unless the network is configured the instance won't be deployed. Any system issue with VR could result in instance deployment failure. It could be an unresponsive VR, its filesystem could be filled up. VR may be unreachable. There could storage issues where the VR is hosted. There could be some bug causing the network configuration on the VR to fail. In these situations we should inspect the VR to identify what is causing the issue. Sometimes restarting the network with clean-up would solve the issue. Sometimes it is best to report the bug or fix the bug if that’s the case.