Advanced Cluster Settings
Download as PPTX, PDF0 likes453 views
The document discusses advanced cluster settings for Rundeck, covering topics such as job ownership for scheduled and one-off jobs, heartbeat settings to monitor server activity, and cluster remote execution policies that determine how jobs are assigned across cluster members. It provides configuration details for load balancing and auto-takeover policies to ensure job continuity in case of server failures. The document also includes examples of different execution policies and settings to optimize job assignment based on server availability and load metrics.
Advanced Cluster Settings
- 1. Rundeck How-To: Advanced Cluster Settings Nathan Fluegel May 30, 2019
- 2. Agenda ● Cluster Architecture ● Job Ownership ○ Scheduled jobs ○ One-off jobs ● Heartbeat Setting ● Cluster Remote Execution Policy ● Load Policy ● Auto-Takeover Policy ● Do you need these settings?
- 5. Job Ownership: Scheduled Jobs Scheduled jobs are always owned by a single Rundeck server cluster member. The default job owner is the cluster where the schedule was first created. If I had been logged in to Server1 when I first set a schedule for JobA, JobA’s scheduled runs will happen on Server1. This behavior can be over-ridden through Cluster Manager.
- 6. Job Ownership: One-off Jobs Jobs that are run at a point in time are owned by a single Rundeck server instance within the cluster. By default, the job owner is the server instance a user is connected to when the job run is initiated. If I had been logged in to Server1 when I initiated a job run for JobA, that run happened on Server1. This default behavior can be over-ridden by Cluster Remote Execution Policies.
- 7. Heartbeat Settings “Heartbeat” is a setting used with all Rundeck servers to measure whether a another server is still alive and able to execute jobs. The results of a heartbeat check can be used to determine logically whether a particular server can be depended on by other settings or processes, such as remote execution or autotakeover policies.
- 8. # heartbeat interval in seconds rundeck.clusterMode.heartbeat.interval=30 # initial delay after startup to send heartbeat rundeck.clusterMode.heartbeat.delay=10 # remote execute/abort message processing interval in seconds rundeck.clusterMode.remoteExec.process.interval=10 # age in seconds since last heartbeat to consider another member inactive rundeck.clusterMode.heartbeat.considerInactive=150 # age in seconds since last heartbeat to consider another member dead rundeck.clusterMode.heartbeat.considerDead=300 Heartbeat Settings Configuration
- 9. Cluster Remote Execution Policy Based on policy configurations, Rundeck Enterprise cluster members may forward job executions to other cluster members. The default policy is to have each job executed locally. If a job is executed in a project which is not assigned to a specific profile, the default policy is used. You can define multiple profiles and assign different projects to different profiles. Execution policies are defined on each server in rundeck-config.properties.
- 10. rundeck.clusterMode.remoteExecution.policy = <Policy> Valid settings for <Policy>: None Default. Executes locally only Random Executes randomly among allowed members RoundRobin Executes round-robin style among allowed members Preset Executes on one other preset member Load Executes on a member based on load (Cluster 2.3.0+) Cluster Remote Execution Configuration Policy Policy setting indicates what method of assignment will be used for jobs in a particular policy.
- 11. rundeck.clusterMode.remoteExecution.config.allowedTags = <List of Member Tags> “Allowed” tags indicate any machines that could have a job assigned to them as part of this policy. rundeck.clusterMode.remoteExecution.config.preferredTags = <List of Member Tags> “Preferred” tags indicate machines that should be given job executions by preference as part of this policy. Preferred machines should be a subset of your allowed machine. If your policy has both allowed and preferred machines, allowed tags would usually only receive a job if there are no Preferred+Allowed tagged machines available. Cluster Remote Execution Configuration Member Tags Tags are used to restrict which Cluster Members could be assigned to run a remote execution. Tags are not the only way to define which cluster members could receive jobs but they are the most flexible and are commonly used. Tags are assigned to individual servers by being added in each framework.properties file as desired.
- 12. rundeck.clusterMode.remoteExecution.config.allowed = <List of Allowed Members> Self - Execute locally Other - Any other member except this one UUID - A particular UUID /regex/ - A regular expression matching a UUID rundeck.clusterMode.remoteExecution.config.activeOnly = true/false Cluster Remote Execution Configuration List of Allowed Members Rather than using tags, you can define allowed members statically. If used in conjunction with tags, the tagged servers would be used by preference. Active Only If set to true, jobs will only be delegated to an active machine (one with a heartbeat).
- 13. rundeck.clusterMode.remoteExecution.profiles = profile1, profile2 This setting is un-necessary if you’re only using one defined profile. rundeck.clusterMode.remoteExecution.profile.profile1.projec ts=projectA, projectB The optional project name setting is provided in the context of a specific profile. Typically, this would only occur if you have more than one profile in play. Cluster Remote Execution Configuration Profile Name Profile names are used to distinguish multiple profiles from one another for more complex configurations. Project Name Each policy can be associated with one or more projects, if desired. This can be used to determine if jobs in a specific project should always be executed on specific machines.
- 14. rundeck.clusterMode.remoteExecution.config.criteria = threadRatio,load This setting identifies which criteria will be used to compute load for this policy. rundeck.clusterMode.remoteExecution.config.weights = 1.0,1.5 Defines relative computational value of the criteria. In this example, load is weighted at 50% more than threadRatio. rundeck.clusterMode.remoteExecution.config.groupWeight=1,0, 0,0 Cluster members are sorted by weighted load and placed into groups. Each group has a weight and the policy randomly chooses a group based on the proportional weight of the group. A group member is then chosen randomly and used. This setting defines four groups, each with 25% of available members. Weights define 100% chance of using group 1. Cluster Remote Execution Configuration Load Balanced Policy When a remote execution configuration policy is set to load, jobs will be assigned to servers based on statistics calculated through the heartbeat process. Load is calculated based on thread ratio and percentage of CPU for each member.
- 15. Example #1 rundeck.clusterMode.remoteExecution.policy = Random rundeck.clusterMode.remoteExecution.config.allowed = other rundeck.clusterMode.remoteExecution.config.activeOn ly = true Example #2 rundeck.clusterMode.remoteExecution.policy = RoundRobin rundeck.clusterMode.remoteExecution.config.allowed = self,/1C519C5A-4E78-4BE9-85EC-.+/ rundeck.clusterMode.remoteExecution.config.activeOn ly = true Cluster Remote Execution Configuration Example #1 A simple policy that sends all jobs to a random peer machine as long as that machine has a heartbeat. Example #2 Round robin assignment among self and any instances with a UUID that matches the regex.
- 16. rundeck.clusterMode.remoteExecution.profiles = profile1, profile2 Example #3 rundeck.clusterMode.remoteExecution.profile.profile1.policy= RoundRobin rundeck.clusterMode.remoteExecution.profile.profile1.projects=projectA,projectB rundeck.clusterMode.remoteExecution.profile.profile1.config.allowed = other rundeck.clusterMode.remoteExecution.profile.profile1.config.allowedTags = * rundeck.clusterMode.remoteExecution.profile.profile1.config.preferredTags = worker,secondary rundeck.clusterMode.remoteExecution.profile.profile1.config.activeOnly = true Example #4 rundeck.clusterMode.remoteExecution.profile.profile2.policy = Preset rundeck.clusterMode.remoteExecution.profile.profile2.config.allowed = eff4405a- 58aa-4f14-b7bd-68e1e44b53d4 rundeck.clusterMode.remoteExecution.profile.profile2.config.activeOnly = true Cluster Remote Execution Configuration - Examples
- 17. Auto-Takeover Policy If a cluster member goes down, an auto- takeover policy helps moved scheduled jobs to another cluster member. Autotakeover uses heartbeat settings to determine whether a machine is alive or not for purposes of triggering autotakeover. For non-scheduled jobs, autotakeover will not kick in. Instead, your remote execution policies can account for this situation.
- 18. # enables autotakeover for members detected as "dead" rundeck.clusterMode.autotakeover.enabled=true # policy indicates which nodes to take over. "Any": all dead nodes. "Static": only allowed uuids rundeck.clusterMode.autotakeover.policy=any # delay in seconds to wait after sending autotakeover proposal rundeck.clusterMode.autotakeover.delay = 60 # sleep in minimum seconds between autotakeover attempts for a particular destination rundeck.clusterMode.autotakeover.sleep = 300 # Name servers to act as takeover destinations, by UUID rundeck.clusterMode.autotakeover.config.allowed=<uuid1>,<uuid2>,... Auto-Takeover Configuration
- 19. Do you need these settings?