Scheduling

The MidoNet Cluster uses the Containers Service to schedule existing service containers at different compute hosts. Container scheduling considers the following input data:

Scheduling is performed at the container level, and it considers the following possible states for a container:

StateDescription

DOWN

The container is not scheduled at a host.

SCHEDULED

The container is scheduled at a host, but that host has not yet reported the container status.

UP

The container is scheduled at a host, and that host has reported the container status as running.

Only DOWN containers are being scheduled, and scheduling is performed when the container is created and subsequently whenever the available hosts and their state changes. By host state, we understand whether the agent at that host is running the container service and its corresponding container weight.

A host is selected to run a container if it is eligible, that is it meets the first three criteria from above about running the container service, having a positive container weight, does not currently run more containers than its container limit, and if the host has not been previously marked as bad for the same container. A bad host is a host that has previously failed to launch a container. This may happen when the host is overloaded, or it does not have installed the necessary software that is needed to launch the container. Marking the hosts as bad is a temporary action, and it prevents the rescheduling logic for retrying indefinitely with the same agent. The interval during which a host is marked as bad after failing to launch a container is configurable in the cluster containers configuration. See the section called “Cluster Configuration”.

[Important]Important

The values of the scheduler_timeout and scheduler_bad_host_lifetime configuration keys are paramount to ensure that containers are scheduled properly, and their values should consider both the acceptable downtime during a container or agent failure and the time required to start the container at the host. Some containers, such as the IPSEC containers, requires a non-negligible time interval to start, and therefore setting the container timeout to a very low value coupled with a long bad host lifetime will lead to containers not being scheduled. This situation may occur during the bulk creation of containers with a small number of compute hosts. In such cases, we recommend increasing the scheduler_timeout or setting the scheduler_bad_host_lifetime to zero.

A host is selected from the eligible hosts using the current allocation policy, specified by the service container group. See the section called “Allocation Policies”.

[Note]Note

Running containers are not affected by changes to a host’s container weight, except when it is set to zero.

After a container has been scheduled, if an agent fails to report the container as running within the allowed scheduler timeout, the scheduling logic is repeated. The service also monitors continuously the host and container state, such that the container is re-scheduled when any of the following situations occurs.

  • The host container service is not running, because the agent has been shut down or has failed.
  • The host container weight is set to zero.
  • The container is in DOWN state and the list of eligible hosts has changed.
  • The container is in SCHEDULED state and the scheduling timeout has expired.
  • The container is in UP state and the container state is stopping, stopped or error.

 Host Container Weight

You can use the container weight of a host to determine whether is is eligible to run service containers, and the weight it should be given when selecting it during container scheduling.

To view the container weight assigned to all hosts, use the command:

midonet> host list
host host1 name host1 alive true addresses 10.0.0.1 flooding-proxy-weight 1 container-weight 0 container-limit no-limit
host host2 name host2 alive true addresses 10.0.0.2 flooding-proxy-weight 1 container-weight 5 container-limit no-limit
host host3 name host3 alive true addresses 10.0.0.3 flooding-proxy-weight 1 container-weight 10 container-limit no-limit

To set the container weight to a different value, use the command:

midonet> host host1 set container-weight <new-value>
[Note]Note

Modifying the container weight on hosts takes an immediate effect on the container scheduling. If the weight is set to zero, all containers that are running or have been scheduled to run at that host will be migrated to other eligible hosts.

 Host Container Limit

Compute hosts can be configured to run up to a configurable number of service containers. By default, a host does not have an upper limit for how many container can be launched.

To view the container limit assigned to each host, use the command:

midonet> host list
host host1 name host1 alive true addresses 10.0.0.1 flooding-proxy-weight 1 container-weight 0 container-limit no-limit
host host2 name host2 alive true addresses 10.0.0.2 flooding-proxy-weight 1 container-weight 5 container-limit no-limit
host host3 name host3 alive true addresses 10.0.0.3 flooding-proxy-weight 1 container-weight 10 container-limit 100

In the host listing, the container-limit property indicates the container limit for each host. For the hosts that do not have a container limit, the value of the property is no-limit, whereas for the hosts with a limit the property is an integer representing the maximum number of service containers that can be launched by that host.

To set the container limit to a different value, use the command:

midonet> host host1 set container-limit <new-value>
[Note]Note

In this version of MidoNet the container limit is enforced only for new containers launched by a host, and existing containers are not affected. This is unlike that container weight, which takes immediate effect when set to zero, forcing the migration of all containers running at that host.

[Important]Important

If you want to gracefully migrate all container from a host, for example in order to take down a compute host for maintenance, use the container-weight property of a host rather than the container-limit. The latter is intended to established a container capacity for the hosts and it does not trigger a container migration.

To unset the container limit for a host use the command:

midonet> host host1 clear container-limit

 Scheduling Policies

MidoNet supports three scheduling policies for service containers. Each policy is configured by assigning the container to an appropriate service container group.

  • Anywhere Policy: This policy corresponds to service container groups that set neither hostGroupId nor portGroupId. With this policy containers are scheduled at any of the MidoNet hosts that run the container service and have a positive container weight.
  • Host Group Policy: Containers with this policy are scheduled only at the hosts that are member of the host group set in the service container group. Hosts must still meet the eligibility requirements to be selected.
  • Port Group Policy: Containers with this policy are scheduled only at the hosts bound to the exterior ports that are members of the port group set in the service container group. Hosts must still meet the eligibility requirements. When using the port group policy, the set of hosts will change when ports migrate between hosts. Therefore, this policy ensures that containers will be scheduled at the same compute hosts where the ports in the port group are located. For more information on port groups, see the section called “Stateful port groups”.
[Important]Important

To ensure host-to-host traffic, all eligible hosts for a particular scheduling policy must be configured in the same tunnel zone. The Containers Service does not use the tunnel zone membership as an eligibility requirement during the host selection.

 Allocation Policies

MidoNet supports two host allocation policies for service containers. These policies specify how a host is selected to launch a container from the pool of hosts determined by the scheduling policy. For a host to be selected with a given allocation policy, it otherwise must meet the eligibility requirements for launching a new container: it has a positive container weight, it does not exceed its container limit, it is not marked as a bad host, it is active and it currently runs the containers service.

  • Least Scheduler: The least policy selects the host from the pool that has the smallest number of containers.
  • Weighted Scheduler: The weighted policy selects a host from the pool randomly, with each host having a probability of being selected proportional to its container weight relative to all other hosts. For example, three hosts with the container weights 1, 3 and 4 will have 12.5%, 37.5% and 50% probability in the statistical sense, respectively, of being selected to run a container.

You can view the current allocation policy by typing:

midonet> list container-group
cgroup cgroup0 policy least
cgroup cgroup1 policy least
cgroup cgroup2 policy weighted

To change the current allocation policy, use:

midonet> container-group cgroup0 set policy <policy-name>

Setting the allocation policy will apply to scheduling new containers belonging to this service container group, and whenever existing containers are being rescheduled. However, running containers are not affected.

 Manual Scheduling

By default, the MidoNet Cluster manages the service container scheduling automatically for all new service containers using both the scheduling policy and the allocation policy specified by the service container group. Launching the service containers once created is therefore transparent to the user, who does not have to take any other action.

However, sometimes you may want to control the location of a container manually, for example to troubleshoot a problem with a particular container. When this is the case, MidoNet offers you the possibility to control the container scheduling at the service container granularity.

To this end, the service container API allows you to suggest a host to launch a given service container. If the suggestion is accepted by the service container scheduler, the container will be immediately migrated to the selected host.

To request that a service container be scheduled at a particular host, type:

midonet> container <container-id> set host <host-id>
[Important]Important

The manual scheduling of a container is only a request by the operator to MidoNet to run a service container at a particular host. The request will be ignored if the host does not meet the eligibility criteria to launch the container, or if the host fails to start the container. When this is the case, the MidoNet container scheduling algorithm will migrate the container at an alternative host or leave the container at its current location.

It is also possible to request to the MidoNet container scheduler to re-trigger the automatic scheduling for a particular service container. This allows you to troubleshoot a malfunctioning container but relying on the automatic scheduling to select the container location.

To re-trigger the automatic scheduling for a service container, type:

midonet> container <container-id> clear host
Questions? Discuss on Mailing Lists or Chat.
Found an error? Report a bug.


loading table of contents...