MidoNet Agent (Midolman) configuration options

This section covers all configuration options for the MidoNet Agent.

We don’t recommend making changes to the default values, except possibly the zookeeper.session_gracetime and agent.datapath.send_buffer_pool_buf_size_kb setting values.

[Warning]Warning

Do not modify the root key, cluster name, or keyspace unless you know what you are doing.

 MidoNet behavior after ZooKeeper cluster failure

Nodes running the MidoNet Agent, Midolman, depend on a live ZooKeeper session to load pieces of a virtual network topology on-demand and watch for updates to those virtual devices.

When ZooKeeper becomes inaccessible, a MidoNet Agent instance will continue operating for as long as there’s a chance to recover connectivity while keeping the same ZooKeeper session. The amount of operating time is thus dictated by the session timeout, which you can control by editing the zookeeper session_gracetime setting in mn-conf(1).

Once the session expires, the MidoNet Agent will give up and shut itself down, prompting upstart to re-launch it. If the ZooKeeper connection and session are recovered within the session_gracetime, MidoNet Agent operation will resume uneventfully. The MidoNet Agent will learn of all the updates that happened to the virtual topology while it was disconnected and will update its internal state and flow tables accordingly.

While the MidoNet Agent is running disconnected from ZooKeeper, waiting for the session to come back, traffic will still be processed, but with reduced functionality, as follows:

  • The MidoNet Agent will not see updates to the virtual topology, thus packets may be processed with a version of the network topology that’s up to session_gracetime too old.
  • The MidoNet Agent will be unable to load new pieces of the network topology. Packets that traverse devices that had never been loaded on a particular MidoNet Agent will error out.
  • The MidoNet Agent will not be able to perform or see updates to Address Resolution Protocol (ARP) tables and Media Access Control (MAC) learning tables.

As time passes, a disconnected MidoNet Agent will become less and less useful. The trade-offs presented above are key to choosing a sensible session_gracetime value; the default is 30 seconds.

ZooKeeper connectivity is not an issue for the MidoNet API server. The API requests are stateless and will simply fail when there is no ZooKeeper connectivity.

 ZooKeeper configuration

You may use the ZooKeeper configuration section in mn-conf(1) to adjust:

  • the ZooKeeper session timeout value (in milliseconds). This value determines when the system considers the connection between ZooKeeper and the MidoNet Agent to be interrupted.
  • the session grace timeout value (in milliseconds). This value determines the period of time during which the Agent can reconnect to ZooKeeper without causing node outage.
  • the root path for MidoNet data.
zookeeper {
    zookeeper_hosts = <comma separated IPs>
    session_timeout = 30000
    root_key = /midonet/v1
    session_gracetime = 30000
}

 Considerations for network partitions and ZooKeeper failure and maintenance

For the MidoNet Agents that run at compute hosts or L2GW hosts, setting the session gracetime is primarily dictated by tradeoffs between maintaining liveness of the datapath and maintaining consistency relative to the authoritative network topology and security policy.

For L3GW hosts with BGP enabled, the considerations are more difficult. While the MidoNet Agent at the L3GW continues to operate disconnected, it continues to announce routes to its BGP peers. If the disconnection from ZooKeeper is due to a network partition that also separates the L3GW from a majority of compute hosts (rather than ZooKeeper failure/maintenance), then it is practically black-holing the traffic destined to computes that are outside its partition. Hence session gracetime presents a tradeoff between increasing L3GW tolerance to ZooKeeper failure (by increasing gracetime) and reducing duration of traffic blackholing (by reducing gracetime).

MidoNet currently does not have a good solution to this tradeoff. In the future we intend to build a blackhole detection mechanism at the L3GW so that it can shut down its BGP session during gracetime if it determines that it cannot deliver most of the traffic it receives.

 Scheduling ZooKeeper maintenance

When it’s necessary to take all (or a majority) of ZooKeeper Quorum nodes down for maintenance, the best practice is to increase the session_gracetime to a value higher than the anticipated maintenance. This should be done for all MidoNet Agents, both on computes and gateway hosts. After the maintenance event the session_gracetime should be returned to the usual values, targeted at the desired tradeoffs of consistency, ZooKeeper failure-tolerance and blackhole duration (see discussion above).

 Cassandra configuration

You may use the Cassandra configuration section to adjust:

  • the database replication factor
  • the MidoNet cluster name
cassandra {
    servers = <comma separated IPs>
    replication_factor = 1
    cluster = midonet
}

 Datapath configuration

The agent uses a pool of reusable buffers to send requests to the datapath. You may use the options in the agent.datapath of mn-conf(1) to tune the pool’s size and its buffers. One pool is created for each output channel, the settings defined here will apply to each of those pools.

If you notice decreased performance because packet sizes exceed the maximum buffer size, you can increase the value for the buf_size_kb setting. This setting controls the buffer size (in KB). Be aware that the buffer size puts a limit on the packet size that the MidoNet Agent can send. In a network that jumbo frames traverse, adjust the size so one buffer will accommodate a whole frame, plus enough room for the flow’s actions.

 BGP failover configuration

The default BGP fail-over time is 2-3 minutes. However, you can reduce this time by changing some parameters on both ends of the session: in mn-conf(1) (the MidoNet side) and the remote end BGP peer configuration. The example below shows how to reduce the BGP fail-over time to one minute on the MidoNet side:

agent {
    midolman {
        bgp_connect_retry=1
        bgp_holdtime=3
        bgp_keepalive=1
    }
}

The settings in mn-conf must match those on the remote end BGP peer configuration. For more information about how to set them, see the section called “BGP failover configuration on a BGP peer”.

Questions? Discuss on Mailing Lists or Chat.
Found an error? Report a bug.


loading table of contents...