Cloud maintenance
Inevitably, the cloud will undergo maintenance. The platform itself is highly redundant, and will be a robust starting point for allowing us to go throughout the years with as little impact to you as cloud users as possible.
Hypervisor maintenance
Hypervisors will routinely undergo software and firmware patching. Hypervisors may also experience hardware failure.
If a hypervisor experiences a hardware problem where we can gracefully move the instance to a new hypervisor, we will attempt to do so automatically.
In the event where we want to do any kind of maintenance on the hypervisors, we will usually perform an automatic task of evacuating all load before we undergo maintenance. By default, we will live-migrate instances. When live-migrating instances, live connections may need to reconnect if passing through a floating IP when the cut-off from one hypervisor to another happens.
If your instance use EVPN networking, and/or advertise routes through BGP on said network connection, there may be a short amount of unstable network as a consequence of a live migration.
Live-migration opt-out
If you wish that a subset of your instances are NOT exposed to live-migration, you can opt out on an instance basis by setting the property rlnc_maint_evacuation_strategy=cold.
If you want to undo this:
In the event where we need to move the instance, we will communicate to the instance by setting a property: rlnc_maintenance_pending=true. When this property is set, we are actively polling the instance state to check if it is currently running. If it is observed as SHUTOFF, it will immediatly be cold-migrated to a new hypervisor and started again.
If you opt out of live migration, we expect that you implement methods for monitoring this property, and shutting down the instance in a reasonable time frame.
A simple example of how to programatically test within the instance if rlnc_maintenance_pending=true is set, that will immediately power off the instance if maintenance is pending:
curl -s http://169.254.169.254/openstack/latest/meta_data.json | jq .meta | grep -q rlnc_maintenance_pending
if [ $? -eq 0 ]; then
# Maintenance pending, powering off for cold migration.
systemctl poweroff
fi
Note
The http://169.254.169.254-endpoint will work out of the box if the instance is set to DHCP. If you were to set a static IP address, you need to ensure that the instance has a route to the distributed port on its network.
To identify which port this is, list ports with device-owner=network:distributed:
$ openstack port list --network 7be33e74-61e2-46d8-af66-17e3239242aa --long --device-owner network:distributed -c 'Fixed IP Addresses'
+-----------------------------------------------------------------------------------------------------+
| Fixed IP Addresses |
+-----------------------------------------------------------------------------------------------------+
| ip_address='10.0.0.2', subnet_id='7e7ca1e1-a52d-40c2-911c-d9668bdd0694' |
+-----------------------------------------------------------------------------------------------------+
In the example above, you need to statically configure a route to 169.254.169.254, with 10.0.0.2 as its nexthop.