{"id":2053,"date":"2021-07-20T10:39:20","date_gmt":"2021-07-20T07:39:20","guid":{"rendered":"https:\/\/upcloud.com\/global\/us\/resources\/tutorials\/reduce-downtime-terraform-redeployments\/"},"modified":"2021-07-20T10:39:20","modified_gmt":"2021-07-20T07:39:20","slug":"reduce-downtime-terraform-redeployments","status":"publish","type":"tutorial","link":"https:\/\/upcloud.com\/global\/resources\/tutorials\/reduce-downtime-terraform-redeployments\/","title":{"rendered":"How to reduce downtime in Terraform redeployments"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Managing cloud infrastructure using <a href=\"https:\/\/www.terraform.io\/\" target=\"_blank\" rel=\"noopener\">Terraform<\/a> has become a de facto standard, and it certainly makes managing our complex cloud platform easier. However, it doesn\u2019t come without challenges. Anyone that has deployed changes to live infrastructure has likely experienced extended application downtime while resources are being replaced. Thankfully we can employ a few features in Terraform that can reduce downtime and increase the availability of managed resources.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In this post, we will use UpCloud\u2019s cloud platform along with their <a href=\"https:\/\/registry.terraform.io\/providers\/UpCloudLtd\/upcloud\/latest\" target=\"_blank\" rel=\"noopener\">Terraform provider<\/a> and the <a href=\"https:\/\/github.com\/UpCloudLtd\/upcloud-cli\" target=\"_blank\" rel=\"noopener\">upctl<\/a> command-line tools to demonstrate what you can do to minimise interruptions to your services when making changes to your cloud infrastructure.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In this blog post, we will cover:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Using UpCloud\u2019s Floating IPs to ensure a server has a consistent IP address.<\/li>\n\n\n\n<li>Using Terraform\u2019s <tt>lifecycle<\/tt> configuration to change the order in which our resources are created and destroyed, meaning that our new resource is created before the old one is destroyed, reducing our downtime.<\/li>\n\n\n\n<li>Using a <tt>local-exec<\/tt> provisioner to check that our resource is ready for traffic before we go ahead and destroy the old one.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">With these configurations in place, we will be able to make changes to the core parts of our infrastructure with certainty that the new pieces will become available with minimal downtime to Terraform managed resources.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Prerequisites<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The steps demonstrated in this tutorial are made intentionally generic to keep the examples simple. This is done to allow anyone with even just a passing knowledge of Terraform to follow along. The only requirements are to have a working installation of <a rel=\"noopener\" href=\"https:\/\/upcloud.com\/global\/community\/tutorials\/get-started-terraform\/\" target=\"_blank\">Terraform<\/a> and <a rel=\"noopener\" href=\"https:\/\/upcloud.com\/global\/community\/tutorials\/get-started-upcloud-command-line-interface\/\" target=\"_blank\">upctl<\/a> on your local computer.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Check out their respective started tutorials to learn more if you haven\u2019t used these before.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Initial state<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Downtime can happen to any resources managed by Terraform and occurs when a change requires the resource to be destroyed and a new resource created in its place. A common example of this is changing the machine image or boot disk of a Cloud Server.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To start with, we need to have some cloud resources that\u2019ll simulate our web application. Our initial Terraform configuration is as follows:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">resource \"upcloud_server\" \"app\" {\n  hostname = \"myapplication.com\"\n  zone     = \"uk-lon1\"\n  plan     = \"1xCPU-1GB\"\n\n  template {\n    storage = \"Ubuntu Server 20.04 LTS (Focal Fossa)\"\n    size    = 25\n  }\n\n  network_interface {\n    type = \"public\"\n  }\n  \n  user_data = &lt;&lt;-EOF apt-get update apt-get -y install apache2 jq echo '(1.0) Hello!' &gt; \/var\/www\/html\/index.html\n    systemctl restart apache2\n  EOF\n}<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">We are just deploying a fresh Ubuntu 20.04 Cloud Server with a public IP. Additionally, we are installing Apache2 and creating a simple index page to have something to query against.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You should include login credentials in your deployment if you wish to access the server. However, for this tutorial, we are going to perform all operations on the local command line.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">  login {\n    user            = \"root\"\n    keys            = [ \"ssh-rsa key\" ]\n    create_password = false\n  }<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If we deploy our code and configuration at this point, we would expect it to all deploy nicely. We can determine the IP address of our server by using the <tt>upctl<\/tt> CLI tool. Note that calling the server resources by their hostname only works when the hostname is unique on your UpCloud account:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">upctl server show myapplication.com<\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\">  Common\n    UUID:          0099b58a-c5f5-4a39-b63e-f6120d701f74\n    Hostname:      myapplication.com\n    Title:         myapplication.com (managed by terraform)\n    Plan:          1xCPU-1GB\n    Zone:          uk-lon1\n    State:         started\n    Simple Backup: no\n    Licence:       0\n    Metadata:      True\n    Timezone:      UTC\n    Host ID:       5767971459\n    Tags:\n\n  Storage: (Flags: B = bootdisk, P = part of plan)\n\n     UUID                                   Title                              Type   Address   Size (GiB)   Flags\n    \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \u2500\u2500\u2500\u2500\u2500\u2500 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\n     01bda3be-ad76-44d3-afc8-fe3d2489ae57   terraform-myapplication.com-disk   disk   ide:0:0           25   P\n    \n  NICs: (Flags: S = source IP filtering, B = bootable)\n\n     #   Type     IP Address                 MAC Address         Network                                Flags\n    \u2500\u2500\u2500 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\n     1   public   IPv4: 94.237.121.69        ee:1b:db:ca:61:ee   03000000-0000-4000-8100-000000000000   S\n<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Next, by using curl to query that endpoint we should get the following response:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ curl http:\/\/94.237.121.69\n(1.0) Hello!<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Now, let\u2019s update our Terraform configuration, for example, by changing the contents of the index.html.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">resource \"upcloud_server\" \"app\" {\n  hostname = \"myapplication.com\"\n  ...\n\n  user_data = &lt;&lt;-EOF\n    apt-get update\n    apt-get -y install apache2 jq\n    echo '<span style=\"color: #ff0000;\">(2.0) Hello again!<\/span>' &gt; \/var\/www\/html\/index.html\n    systemctl restart apache2\n  EOF\n}\n<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If we then apply the changes, Terraform will recognise the difference and require the Cloud Server to be recreated. This will cause Terraform to destroy our existing server and create a new one.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">Terraform will perform the following actions:\n\n  # <strong>upcloud_server.app<\/strong> must be <strong><span style=\"color: #ff0000;\">replaced<\/span><\/strong>\n<span style=\"color: #26a269;\">+<\/span>\/<span style=\"color: #ff0000;\">-<\/span> resource \"upcloud_server\" \"app\" {\n      <span style=\"color: #a2734c;\">~<\/span> cpu       = 1 -&gt; (known after apply)\n      <span style=\"color: #ff0000;\">-<\/span> firewall  = false -&gt; null\n      <span style=\"color: #a2734c;\">~<\/span> id        = \"00980377-5a6e-4b1f-90ba-fe73bf68ca6e\" -&gt; (known after apply)\n      <span style=\"color: #a2734c;\">~<\/span> mem       = 1024 -&gt; (known after apply)\n      <span style=\"color: #ff0000;\">-<\/span> metadata  = false -&gt; null\n      <span style=\"color: #ff0000;\">-<\/span> tags      = [] -&gt; null\n      <span style=\"color: #a2734c;\">~<\/span> title     = \"myapplication.com (managed by terraform)\" -&gt; (known after apply)\n      <span style=\"color: #a2734c;\">~<\/span> user_data = &lt;&lt;-EOT <span style=\"color: #ff0000;\"># forces replacement<\/span>\n            apt-get update\n            apt-get -y install apache2 jq\n          <span style=\"color: #c01c28;\">-<\/span> echo '(1.0) Hello!' &gt; \/var\/www\/html\/index.html\n          <span style=\"color: #26a269;\">+<\/span> echo '(2.0) Hello again!' &gt; \/var\/www\/html\/index.html\n            systemctl restart apache2\n        EOT\n<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Go ahead and hit apply to see for yourself.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Once Terraform gets to work, we\u2019ll discover that our service has become inaccessible as the first instance is destroyed.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">upcloud_server.app: Destroying... [id=00980377-5a6e-4b1f-90ba-fe73bf68ca6e]\nupcloud_server.app: Still destroying... [id=00980377-5a6e-4b1f-90ba-fe73bf68ca6e, 10s elapsed]\nupcloud_server.app: Destruction complete after 11s\nupcloud_server.app: Creating...\nupcloud_server.app: Still creating... [10s elapsed]\nupcloud_server.app: Still creating... [20s elapsed]\nupcloud_server.app: Creation complete after 27s [id=00d512f8-9bf4-4f11-8d4e-3c56011c83a3]\n<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Since our web server is a very simple example, it doesn\u2019t take long to initialise. However, even then, our service might be down for more than a minute.&nbsp;Let\u2019s change that!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Adding Floating IP<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">In addition to the Terraform operations taking time, it\u2019s also possible our services never come up again on the same endpoint. A common reason why redeployment may result in extended downtime is that the public IP address of the new server is different from the old IP.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The IP addresses assigned to your Cloud Servers are given at random. While you may get the same IP address between server to destroy\/create cycles, it isn\u2019t guaranteed. To overcome this in our effort to reduce downtime in our Terraform deployments we will use <a rel=\"noopener\" href=\"https:\/\/upcloud.com\/global\/community\/tutorials\/floating-ip-addresses\/\" target=\"_blank\">UpCloud\u2019s Floating IP<\/a> facility.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We will add a new resource using <tt>upcloud_floating_ip_address<\/tt> to our Terraform configuration. Simply add the following to the end of your Terraform configuration file:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">resource \"upcloud_floating_ip_address\" \"app_ip\" {\n  mac_address = upcloud_server.app.network_interface[0].mac_address\n}<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This resource will assign a public IPv4 address to our server\u2019s network interface as identified by the <tt>mac_address<\/tt> value. As of today, this isn\u2019t quite enough as the metadata service, UpCloud provided web service that allows a server to query data about itself, doesn\u2019t refresh automatically. As a workaround, we can turn the metadata service off and on again by using a Terraform <tt>null_resource<\/tt>\u00a0and calling <tt>upctl<\/tt>:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">resource \"null_resource\" \"metadata_update\" {\n\n  triggers = {\n    mac_address = upcloud_floating_ip_address.app_ip.mac_address\n  }\n\n  provisioner \"local-exec\" {\n    interpreter = [\"\/bin\/bash\", \"-c\"]\n    command = &lt;&lt;-EOF\n      upctl server modify ${upcloud_server.app.id} --disable-metadata\n      upctl server modify ${upcloud_server.app.id} --enable-metadata\n    EOF\n  }\n}<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This <tt>null_resource<\/tt> is triggered by any change to the associated <tt>upcloud_floating_ip_address.mac_address<\/tt> value. For example, when assigning the IP to a new Cloud Server. Once triggered, it uses a <tt>local-exec<\/tt> provisioner to run a small bash script on the machine performing the deployment.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This script uses <tt>upctl<\/tt> to turn off the metadata service and then turn it back on again.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To use the floating IP address, we need to make the Linux operating system aware of it. You can find an example of automating this here:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/github.com\/opencredo\/upcloudtfdowntime\/blob\/main\/packer\/artefact\/floating_ip.sh\" target=\"_blank\" rel=\"noopener\">floating_ip.sh<\/a> \u2013 A shell script that pings the metadata service looking for a floating IP. If it already has a floating IP, this script does nothing.<\/li>\n\n\n\n<li><a href=\"https:\/\/github.com\/opencredo\/upcloudtfdowntime\/blob\/main\/packer\/artefact\/floating_ip.service\" target=\"_blank\" rel=\"noopener\">floating_ip.service<\/a> \u2013 A systemd unit that runs the above script as a one-shot, i.e. once per boot.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">We can update our server to use the above files by adding the following to our user_data script.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">resource \"upcloud_server\" \"app\" {\n  hostname = \"myapplication.com\"\n  ...\n  user_data = &lt;&lt;-EOF apt-get update apt-get -y install apache2 jq echo '(2.0) Hello again!' &gt; \/var\/www\/html\/index.html\n    systemctl restart apache2\n    \n<span style=\"color: #33da7a;\">    wget https:\/\/raw.githubusercontent.com\/opencredo\/upcloudtfdowntime\/main\/packer\/artefact\/floating_ip.sh -O \/usr\/local\/bin\/floating_ip.sh\n    chmod +x \/usr\/local\/bin\/floating_ip.sh\n    wget https:\/\/raw.githubusercontent.com\/opencredo\/upcloudtfdowntime\/main\/packer\/artefact\/floating_ip.service -O \/lib\/systemd\/system\/floating_ip.service\n    systemctl daemon-reload\n    echo 'source \/etc\/network\/interfaces.d\/*' &gt;&gt; \/etc\/network\/interfaces\n    systemctl start floating_ip<\/span>\n  EOF<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Afterwards, re-deploy the resources again using Terraform. We will then see a new floating IP created.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If we were to cause the server to be redeployed, for example by tainting it, we would see our endpoint go down and approximately a minute later come back up again at the same IP address.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Configuring resource lifecycle<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Having recreated our Terraform resources already a few times, we have a good idea of the time it takes for Terraform to apply changes. In the Terraform output, we can see the following:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">upcloud_server.app: Destroying... [id=00379689-3649-403f-b639-50c8d2a7309b]\nupcloud_server.app: Still destroying... [id=00379689-3649-403f-b639-50c8d2a7309b, 10s elapsed]\nupcloud_server.app: Destruction complete after 16s\nupcloud_server.app: Creating...\nupcloud_server.app: Still creating... [10s elapsed]\nupcloud_server.app: Still creating... [20s elapsed]\nupcloud_server.app: Still creating... [30s elapsed]\nupcloud_server.app: Creation complete after 34s [id=00953906-2578-4f3a-b790-cae1e62001bb]<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The terminal output shows the old server\u2019s destruction before the new server is created, which is the bulk of our downtime. We can mitigate this by using Terraform\u2019s <a href=\"https:\/\/www.terraform.io\/docs\/language\/meta-arguments\/lifecycle.html\" target=\"_blank\" rel=\"noopener\">lifecycle block<\/a> that tells Terraform to create the new server before it destroys the old one.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This configuration is available on all Terraform resources, but you should consider if it is needed as it adds complexity and maybe be difficult to troubleshoot. Some resources may need to destroy their old version first to free up a dependency.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In our case, the lifecycle condition will be useful. Let\u2019s add this block to the <tt>upcloud_server<\/tt> resource in our Terraform configuration:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">resource \"upcloud_server\" \"app\" {\n  hostname = \"myapplication.com\"\n  ...\n\n<span style=\"color: #33da7a;\">  lifecycle {\n    create_before_destroy = true\n  }<\/span>\n  ...\n}\n<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">That should be all we need to do to reduce downtime in our Terraform redeployments<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If we were to cause our server to be replaced, maybe by updating the <tt>user_data<\/tt> section again, we would see our endpoint go down. It will take longer for the endpoint to go down this time, and it only takes around 5 seconds for it to come back up again.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We can retake a look at the logs:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">upcloud_server.app: Creating...\nupcloud_server.app: Still creating... [10s elapsed]\nupcloud_server.app: Still creating... [20s elapsed]\nupcloud_server.app: Creation complete after 27s [id=00ae5717-2ead-4c68-9339-1c737f5b0678]<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">And then later in the logs:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">upcloud_server.app (deposed object 14b4d5c5): Destroying... [id=007c66bc-4c9a-4e18-a2d5-d82d1c3fe483]\nupcloud_server.app: Still destroying... [id=007c66bc-4c9a-4e18-a2d5-d82d1c3fe483, 10s elapsed]\nupcloud_server.app: Destruction complete after 11s<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">We can see the new Cloud Server being created first before the destruction of the old one. At this point, we\u2019ve reduced the redeployment downtime to but a few seconds. However, eliminating the last few seconds of downtime would require tweaking our infrastructure and introducing something like a reverse proxy or a load balancer. Because those last seconds are due to the time it takes for the floating IP to be reassigned.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Protecting against failed deployments<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We\u2019ve now managed to reduce downtime caused by Terraform redeployments considerably! However, there\u2019s still one additional problem we need to consider. What if the new version of our service fails upon deployment or it returns an error for a period after the initial startup? For example, it may take some time for a database connection to be established. Is it possible to protect against that scenario?<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Yes! In essence, we would want Terraform to check the health of our new <tt>upcloud_server<\/tt> resource before it goes ahead and destroys the old instance and flips the floating IP over to our new server.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We can use the <tt>local-exec<\/tt> provisioner to do this again.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We add the following to the terraform file:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">resource \"upcloud_server\" \"app\" {\n  hostname = \"myapplication.com\"\n  ...\n\n  provisioner \"local-exec\" {\n    interpreter = [\"\/bin\/bash\", \"-c\"]\n\n    command = &lt;&lt;-EOF fail_count=0 while true; do response=$(curl --write-out %%{http_code} --silent --output \/dev\/null http:\/\/${self.network_interface[0].ip_address}) echo \"Response: $response\" if [[ \"$response\" == \"200\" ]]; then echo \"Application is available\" exit 0 fi fail_count=$((fail_count + 1)) if (( fail_count &gt; 30 )); then\n          echo \"Application is still unavailable\"\n          exit 2\n        fi\n\n        echo \"Sleeping\"\n        sleep 10\n      done\n    EOF\n  }\n}\n<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This <tt>local-exec<\/tt> provisioner calls a short bash script that uses <tt>curl<\/tt> query on the new server. We can figure out the state of the services by checking the HTTP response when accessing the new <tt>upcloud_server<\/tt> resource via its IP address <tt>self.network_interface[0].ip_address<\/tt>. If the response is 200, then the new resource is considered available, and the Terraform apply can continue. It will attempt to probe the endpoint for 5 minutes (30 fails * 10s sleep per loop) and, if it is still not up during this period, it will fail.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This failure will leave the old infrastructure untouched, but the failed <tt>upcloud_server<\/tt> resource will not be cleared up. A future Terraform apply with a fix will tidy it up.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To test this, we can make another change to our deployment. This time, let\u2019s add a sleep command that purposefully increases the time for the services to become available.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">  user_data = &lt;&lt;-EOF\n    apt-get update\n    <span style=\"color: #33da7a;\">sleep 60<\/span>\n    apt-get -y install apache2 jq\n    echo '(2.0) Hello again!' &gt; \/var\/www\/html\/index.html\n    systemctl restart apache2\n<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If we deploy this, we should see the deployment now takes much longer. However, there should still only be a few seconds of downtime.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We can examine the Terraform logs to see this in action:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">upcloud_server.app: Provisioning with 'local-exec'...\nupcloud_server.app (local-exec): Executing: [\"\/bin\/bash\" \"-c\" \"fail_count=0 nwhile true; do n  response=$(curl --write-out %{http_code} --silent --output \/dev\/null http:\/\/94.237.61.152) n  echo \"Response: $response\" n      n  if [[ \"$response\" == \"200\" ]]; then n    echo \"Application is available\"n    exit 0 n  fi n        n  fail_count=$((fail_count + 1))n        n  if (( fail_count &gt; 30 )); thenn    echo \"Application is still unavailable\"n    exit 2n  finn  echo \"Sleeping\"n  sleep 10ndonen\"]\nupcloud_server.app: Still creating... [40s elapsed]\nupcloud_server.app (local-exec): Response: 000\nupcloud_server.app (local-exec): Sleeping\n<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">We can see the creation of the new server resource and, we can also see our local-exec provisioner starting and getting error responses from the server. Later in the log (after our 2 minutes), we can see:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">upcloud_server.app: Still creating... [2m0s elapsed]\nupcloud_server.app (local-exec): Response: 200\nupcloud_server.app (local-exec): Application is available\nupcloud_server.app: Creation complete after 2m9s [id=00bb0b8e-7a62-44bc-a34a-95ef0e50529d]\n<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">We can see the local-exec provisioner receiving a 200 response which completes the new server resource creation. The rest of the Terraform apply can then continue, and we will see our latest version come up.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Congratulations! You should now have a few tricks up your sleeve to reduce downtime in your Terraform deployments.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For something more to learn about, check out our practical demonstration incorporating Terraform cloud, a GitHub Actions continuous deployment pipeline and Packer over at YouTube:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The code demonstrating the concepts in the video and this blog post is available on <a rel=\"noopener\" href=\"https:\/\/github.com\/opencredo\/upcloudtfdowntime\" target=\"_blank\">GitHub<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If a further blog post or video with an in-depth look at any of the concepts covered here is of interest, please leave a comment on the video or get in touch.<\/p>\n","protected":false},"author":55,"featured_media":19536,"comment_status":"open","ping_status":"closed","template":"","community-category":[235],"class_list":["post-2053","tutorial","type-tutorial","status-publish","has-post-thumbnail","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/2053","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial"}],"about":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/types\/tutorial"}],"author":[{"embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/users\/55"}],"replies":[{"embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/comments?post=2053"}],"version-history":[{"count":0,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/2053\/revisions"}],"wp:attachment":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/media?parent=2053"}],"wp:term":[{"taxonomy":"community-category","embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/community-category?post=2053"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}