Multi-node development environment
Airship-in-a-bottle multi-node is a simulated Treasure map deployment using only virtual machines that run within a single hypervisor. This gives great control over things like network topology, ‘hardware’ configuration and the like. It also introduces some limitations. Overall the framework should allow most developers to test with a high degree of certainty that a proposed change will be successful.
Mechanics
The automation framework is contained in the
airship-in-a-bottle
repository. This framework is based on original work by Mark Burnett in
the Airship/Promenade project.
There are several parallel pieces of automation in this repository, we will
focus on the tooling under tools/multi_nodes_gate.
Currently the framework is hard-coded to source the deployment
definition from the deployment_files directory within the repository.
Specifically the framework attempts to deploy the site
deployment_files/site/gate-multinode. Below we will describe how we
can merge this framework with our
Treasure map definitions.
Here is a brief overview of the components within tools/multinode_gate:
./setup_gate.sh- Must be run once on the host machine to install necessary packages/tools and set them up appropriately./gate.sh- entrypoint to running the framework, run with an argument of the scenario you’d like to simulate. Default scenario ismultinode_deploy../airship_gate/manifests- JSON files that drive the various “scenarios” that can be executed by the gate../airship_gate/stages- The scripts that are called based on the stages section of a scenario manifest. These should be loose wrappers around calling library scripts from lib../airship_gate/lib- All of the framework script libraries./airship_gate/bin- Utility binaries that can be used during testing/troubleshooting
Host Setup (Required once)
Recommended minimum requirements:
64GB RAM
24 vCPUs
300 GB disk
Configure /etc/profile
Add the following lines to /etc/profile or create a seperate env
file that you can source separately. These environment variables are used
at various stages throughout the pipeline,
NOTE: /etc/environment does not work well with (”) or with nested env variables
export TEMP_DIR="$HOME/airship-in-a-bottle/tools/multi_nodes_gate/temp"
export IMAGE_PROMENADE_CLI="quay.io/airshipit/promenade:b417f422e9cd2a921646ee0af4a05fc4e211beab"
export IMAGE_PEGLEG_CLI="quay.io/airshipit/pegleg:50ce7a02e08a0a5277c2fbda96ece6eb5782407a"
export IMAGE_SHIPYARD_CLI="quay.io/airshipit/shipyard:f4f57a1bbf3cc9ca6c868a11cc8923326c81b6dc"
export IMAGE_COREDNS="docker.io/coredns/coredns:1.1.3"
export IMAGE_QUAGGA="cumulusnetworks/quagga:CL3.3.2"
export IMAGE_DRYDOCK_CLI="quay.io/airshipit/drydock@:54ea0e1374ddd18a5195edc418b5c0b042666f45"
export BASE_IMAGE_URL="https://cloud-images.ubuntu.com/releases/16.04/release/ubuntu-16.04-server-cloudimg-amd64-disk1.img"
export UPSTREAM_DNS="8.8.8.8 8.8.4.4"
export AIRSHIP_KEYSTONE_URL="http://keystone.gate.local:80/v3"
export SHIPYARD_OS_PASSWORD=password123
export VIRSH_CPU_OPTS="Westmere"
export USE_EXISTING_SECRETS="true"
export GATE_SSH_KEY="$HOME/.ssh/id_rsa_airship"
# export NTP_POOLS=""
Don’t forget to source in the new set of env variables
source /etc/profile
source < your own env file >
Create libvirtd group and add user virtmgr
sudo groupadd libvirtd
sudo useradd virtmgr
sudo usermod -a -G libvirtd virtmgr
If you are using Ubuntu 18.04, add this step:
sudo groupadd libvirt
sudo usermod -a -G libvirt virtmgr
Clone the Repositories
git clone https://opendev.org/airship/in-a-bottle.git ~/airship-in-a-bottle
git clone https://opendev.org/airship/treasuremap.git ~/airship-in-a-bottle/treasuremap
Set up SSH Keys
cat ~/airship-in-a-bottle/treasuremap/site/seaworthy-virt/secrets/passphrases/airship_drydock_kvm_ssh_key.yaml
=> Copy the block from '-----BEGIN RSA PRIVATE KEY-----' till '-----END RSA PRIVATE KEY-----' and create ${HOME}/.ssh/id_rsa_airship
cat ~/airship-in-a-bottle/treasuremap/site/seaworthy-virt/secrets/passphrases/airship_ubuntu_ssh_public_key.yaml
=> Copy only the ssh-rsa string and create ${HOME}/.ssh/id_rsa_airship.pub
Replace multinode_deploy.json with virtual seaworthy stages
~/airship-in-a-bottle/tools/multi_nodes_gate/airship_gate/manifests/multinode_deploy.json
{
"configuration": {
"site": "seaworthy-virt",
"primary_repo": "treasuremap",
"aux_repos": []
},
"ingress": {
"domain": "gate.local",
"ca": "-----BEGIN CERTIFICATE-----\nMIIDIDCCAgigAwIBAgIUfikFVpFSQKVjACP9i8P4tUMnQbcwDQYJKoZIhvcNAQEL\nBQAwKDERMA8GA1UEChMIU25ha2VvaWwxEzARBgNVBAMTCmluZ3Jlc3MtY2EwHhcN\nMTgxMjAzMjEzOTAwWhcNMjMxMjAyMjEzOTAwWjAoMREwDwYDVQQKEwhTbmFrZW9p\nbDETMBEGA1UEAxMKaW5ncmVzcy1jYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC\nAQoCggEBAOR6+3dCF5mtKvu2TlaYNHc6/v8VPvw3I0+EI+jRskXVQHZxF0kcLAVH\n/LM2maTMzNc1sZnxCnj8YYHxfhdIco+zwzCbG1YGolSPrPaslYmMmDjR0eVl1+tb\nmLnEHDZ88ds5rXNlUXDhAURzYPJivG2aYBVImvaS4GHztndaFFNE0Q7HQpldCs1Q\n5+xbFlKWHBt/xPM4QjoD/ReLEE5m5HhkT4WN0hWC0NC1OwW6bBhVkrk4D2kDTq8d\n/b5MH4FG2HHJYHXKR4caasrCHUrmuq7m6WoicwF7z53FvlM782EsNx6vSoBKYs39\n/AC4meM/9D8rjUlWaG3AjP0KFrFCLYECAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEG\nMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFJFfhFd1reBWgmrWe6PBV2z5W/Ee\nMA0GCSqGSIb3DQEBCwUAA4IBAQAZygjSCRSJrvgPllyWDpyKN1fg2r7P2ioI0WR9\nWkSrPKzdhi2hR8VdJxkMvRpEmWRhkQT7jNGEIWgy2jtyWiYKnKYobbY/kMU86QgL\nZazh2DiIeJim+Vt3RREyfOcNDwGMX7NpfwMTz7Dzl+jvtlBwKLFN0L15d0X+4J9V\ndRp5ZkooVjiOJb6vNcozDWxBrRPAowrvzLlJkFMaKgJQmGigEpgEygnCRH++NCle\n/ivGbdFuCsYzUTlR77xf9kGXMh3socMXcdu5SOtaDS7sl52DAJnAPxo9S6l0270G\na0989is2yCgDNmld5lpphVPaQSusGa8/XTaXR7YH+oc7qn1l\n-----END CERTIFICATE-----",
"172.24.1.5": [
"maas"
],
"172.24.1.6": ["drydock", "shipyard", "keystone"]
},
"stages": [
{
"name": "Gate Setup",
"script": "gate-setup.sh"
},
{
"name": "Pegleg Collection",
"script": "pegleg-collect.sh"
},
{
"name": "Pegleg Render",
"script": "pegleg-render.sh"
},
{
"name": "Generate Certificates",
"script": "generate-certificates.sh"
},
{
"name": "Build Scripts",
"script": "build-scripts.sh"
},
{
"name": "Create VMs",
"script": "create-vms.sh"
},
{
"name": "Register Ingress",
"script": "ingress-dns.sh",
"arguments": ["build"]
},
{
"name": "Create BGP router",
"script": "bgp-router.sh",
"arguments": ["build"]
},
{
"name": "Pre Genesis Setup",
"script": "genesis-setup.sh"
},
{
"name": "Genesis",
"script": "genesis.sh",
"on_error": "collect_genesis_info.sh"
},
{
"name": "Validate Genesis",
"script": "validate-genesis.sh",
"on_error": "collect_genesis_info.sh"
},
{
"name": "Load Site Design",
"script": "shipyard-load-design.sh"
},
{
"name": "Deploy Site",
"script": "shipyard-deploy-site.sh"
},
{
"name": "Validate Kube",
"script": "validate-kube.sh",
"on_error": "collect_genesis_info.sh"
}
],
"vm": {
"build": {
"memory": 4072,
"vcpus": 2,
"mac": "52:54:00:00:be:31",
"ip": "172.24.1.9",
"io_profile": "fast",
"bootstrap": true,
"userdata": "packages: [docker.io]"
},
"n0": {
"memory": 24768,
"vcpus": 16,
"mac": "52:54:00:00:a4:31",
"ip": "172.24.1.10",
"io_profile": "fast",
"bootstrap": true
},
"n1": {
"memory": 8072,
"vcpus": 4,
"mac": "52:54:00:00:a3:31",
"ip": "172.24.1.11",
"io_profile": "fast",
"bootstrap": false
},
"n2": {
"memory": 8072,
"vcpus": 4,
"mac": "52:54:00:1a:95:0d",
"ip": "172.24.1.12",
"io_profile": "fast",
"bootstrap": false
},
"n3": {
"memory": 8072,
"vcpus": 4,
"mac": "52:54:00:31:c2:36",
"ip": "172.24.1.13",
"io_profile": "fast",
"bootstrap": false
}
},
"bgp": {
"quagga_as": 64688,
"calico_as": 64671
}
}
Create a temp directory for generated files
mkdir ~/airship-in-a-bottle/tools/multi_nodes_gate/temp/
Run the setup script
cd ~/airship-in-a-bottle/tools/multi_nodes_gate/
./setup_gate.sh
A reboot is recommended after this
Deploy the Site
Repeat this process as many times as required.
If you want to test specific patch sets, pull them to the treasure map repository before running the gate.
Remove the generated files and run
cd ~/airship-in-a-bottle/tools/multi_nodes_gate/
rm -rf temp/*
./gate.sh
Update the Site
Replace the contents of update_site.json
~/airship-in-a-bottle/tools/multi_nodes_gate/airship_gate/manifests/update_site.json
{
"configuration": {
"site": "seaworthy-virt",
"primary_repo": "treasuremap",
"aux_repos": []
},
"ingress": {
"domain": "gate.local",
"ca": "-----BEGIN CERTIFICATE-----\nMIIDIDCCAgigAwIBAgIUfikFVpFSQKVjACP9i8P4tUMnQbcwDQYJKoZIhvcNAQEL\nBQAwKDERMA8GA1UEChMIU25ha2VvaWwxEzARBgNVBAMTCmluZ3Jlc3MtY2EwHhcN\nMTgxMjAzMjEzOTAwWhcNMjMxMjAyMjEzOTAwWjAoMREwDwYDVQQKEwhTbmFrZW9p\nbDETMBEGA1UEAxMKaW5ncmVzcy1jYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC\nAQoCggEBAOR6+3dCF5mtKvu2TlaYNHc6/v8VPvw3I0+EI+jRskXVQHZxF0kcLAVH\n/LM2maTMzNc1sZnxCnj8YYHxfhdIco+zwzCbG1YGolSPrPaslYmMmDjR0eVl1+tb\nmLnEHDZ88ds5rXNlUXDhAURzYPJivG2aYBVImvaS4GHztndaFFNE0Q7HQpldCs1Q\n5+xbFlKWHBt/xPM4QjoD/ReLEE5m5HhkT4WN0hWC0NC1OwW6bBhVkrk4D2kDTq8d\n/b5MH4FG2HHJYHXKR4caasrCHUrmuq7m6WoicwF7z53FvlM782EsNx6vSoBKYs39\n/AC4meM/9D8rjUlWaG3AjP0KFrFCLYECAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEG\nMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFJFfhFd1reBWgmrWe6PBV2z5W/Ee\nMA0GCSqGSIb3DQEBCwUAA4IBAQAZygjSCRSJrvgPllyWDpyKN1fg2r7P2ioI0WR9\nWkSrPKzdhi2hR8VdJxkMvRpEmWRhkQT7jNGEIWgy2jtyWiYKnKYobbY/kMU86QgL\nZazh2DiIeJim+Vt3RREyfOcNDwGMX7NpfwMTz7Dzl+jvtlBwKLFN0L15d0X+4J9V\ndRp5ZkooVjiOJb6vNcozDWxBrRPAowrvzLlJkFMaKgJQmGigEpgEygnCRH++NCle\n/ivGbdFuCsYzUTlR77xf9kGXMh3socMXcdu5SOtaDS7sl52DAJnAPxo9S6l0270G\na0989is2yCgDNmld5lpphVPaQSusGa8/XTaXR7YH+oc7qn1l\n-----END CERTIFICATE-----",
"172.24.1.5": [
"maas"
],
"172.24.1.6": ["drydock", "shipyard", "keystone"]
},
"stages": [
{
"name": "Pegleg Collection",
"script": "pegleg-collect.sh",
"arguments": [
"update"
]
},
{
"name": "Load Site Design",
"script": "shipyard-load-design.sh",
"arguments": [
"-g",
"-o"
]
},
{
"name": "Deploy Site",
"script": "shipyard-update-site.sh"
}
],
"vm": {
"build": {
"memory": 3072,
"vcpus": 2,
"mac": "52:54:00:00:be:31",
"ip": "172.24.1.9",
"bootstrap": true,
"userdata": "packages: [docker.io]"
},
"n0": {
"memory": 32768,
"vcpus": 8,
"mac": "52:54:00:00:a4:31",
"ip": "172.24.1.10",
"bootstrap": true
},
"n1": {
"memory": 3072,
"vcpus": 2,
"mac": "52:54:00:00:a3:31",
"ip": "172.24.1.11",
"bootstrap": false
},
"n2": {
"memory": 3072,
"vcpus": 2,
"mac": "52:54:00:1a:95:0d",
"ip": "172.24.1.12",
"bootstrap": false
},
"n3": {
"memory": 3072,
"vcpus": 2,
"mac": "52:54:00:31:c2:36",
"ip": "172.24.1.13",
"bootstrap": false
}
}
}
Run update site
Once you have made the changes on your manifests, Run the update site action
cd ~/airship-in-a-bottle/tools/multi_nodes_gate
./gate.sh update_site