Cloud benchmark at Slush 2015

Slush 2015 is a 15.000 attendee startup event in Helsinki where more than 1700 startups and 800 investors arrived for the two-day event on 11th and 12th of November. Like last year, UpCloud was one of the key partners of Slush and we were showcasing our cloud hosting service at our stand. For this year’s event, we wanted to show off some benchmarks and we created a benchmarking game comparing UpCloud, AWS EC2, and DigitalOcean.

Traditional benchmarks are boring, so we created a Commodore64-style game theme, which includes a live setup of a cloud server, running the benchmarks and then finally destroying the cloud server. The benchmark game runs from the start (server creation) to finish (server destroyed) in a timeframe of couple minutes. We managed the cloud servers in real-time using Ansible and its modules for the cloud hosting providers. During the two days at Slush, the benchmarks were run 167 times and we collected all the results.

Let’s show the results first and then go through what they mean:

We’ve marked the winner of each category with the colour green. The averages give a view of the performance and the standard deviation is a good way to measure the consistency; ie. how close to the average you can expect each test run to be. UpCloud really shines in both of these only losing in random write IOPS consistency whereas our average results are phenomenal compared to AWS EC2 and DigitalOcean.

The only benchmark where UpCloud is not the winner is network; this is due to the fact that we limit each cloud server to a port speed of 500Mbps (it can be upgraded to 1Gbps upon request). This is a service quality measure as we don’t want any server affecting the network performance of others by accident.

Server setup

Server setup is the time in seconds from starting the Ansible playbook to when Ansible tells us that the instance is responding over SSH. You can expect similar results when using the control panels of these providers to create servers, as Ansible is using the standard APIs of each provider. UpCloud is fastest in deploying a cloud server thanks to our quick MaxIOPS storage system that allows quick cloning of the OS template. Interestingly, DigitalOcean is the slowest with an average deploy time of 120 seconds.

An example Ansible playbook for UpCloud. For AWS EC2 and DigitalOcean, use their Ansible modules instead for setting up servers.

---
- hosts: localhost
  connection: local
  serial: 1
  gather_facts: no
  vars:
    server_name: "{{ SERVER_NAME|default('www.example.com') }}"
    tag_name: "{{ TAG_NAME|default('www') }}"
  tasks:
    - name: Create UpCloud server
      upcloud:
        state: present
        hostname: "{{ server_name }}"
        title: "{{ server_name }}"
        zone: de-fra1
        core_number: 2
        memory_amount: 4096
        storage_devices:
          - { size: 60, os: Ubuntu 14.04, address: "virtio:0" }
      register: upcloud_server
    - name: Wait for SSH to come up
      wait_for: host={{ upcloud_server.public_ip }} port=22 delay=5 timeout=320 state=started

CPU ~ cyclictest

We used cyclictest to measure the average latency of the CPU, or the time it takes to complete a CPU cycle. This is essentially a measure of how overbooked the physical CPUs are; how many virtual CPUs are “lining up” to get their turn at crunching numbers at the actual physical CPU. Thus, a smaller number is better as it is a sign of how responsive the CPU is, and what kind of performance you are getting for your money.

UpCloud demonstrates much smaller latencies with a much better consistency. Create a cloud server at our Frankfurt site right now, and you will get similar results with a very high probability. AWS can also show a very good consistency in their results, unlike DigitalOcean who had a very high standard deviation in its results.
Exact command to reproduce test is shown below. We used the avg_lat value.

apt-get install rt-tests
sudo cyclictest -D 10s -q

RAM ~ mbw

Mbw is a measure of how fast data can be moved in and out of RAM. The faster the better. We didn’t expect huge differences between cloud providers but for some reason, DigitalOcean seems to be lacking. Given the equal size of RAM, this is an important measure as moving data in and out of RAM is one of the common bottlenecks in computing in general. Given an equal amount of RAM, it is certainly better to get good results from Mbw – you wouldn’t like paying for less, right?

Exact command to reproduce test is shown below. We used the avg_copy value.

apt-get install mbw
mbw -n 250 -t 0 200

Network ~ wget

Measuring network performance is a bit tricky as it isn’t easy to get completely objective results. UpCloud loses here as we currently limit cloud servers to a 500Mbps connection (can be upgraded to 1Gbps upon request). The reason behind this decision is one of service quality, we do not want any single customer or their cloud server affecting the network performance of others.

Exact command to reproduce test is shown below. We used the average speed value for this test.

wget -4 -O /dev/null https://mirror.de.leaseweb.net/kernel/v4.x/linux-4.2.4.tar.gz

Storage I/O ~ fio

Fio is the traditional benchmark for disk I/O. We are measuring random read and write IOPS to compare the disk performance of 60GB SSDs. UpCloud performs superbly, especially in random writes. Thanks to our proprietary MaxIOPS system, our competitors can not match or even approach our disk I/O performance that reaches 100 000 IOPS. The speed of storage is very noticeable; your databases and file servers can see a huge performance boost on top of MaxIOPS. The performance boost isn’t related to only database servers, as regular web servers perform much better with no bottlenecks in storage. This is something we’re especially proud of.

Exact command to reproduce test is shown below. We used the IOPS value.

apt-get install fio
# device depends on the provider; make sure you run against the correct disk
sudo fio --minimal -name=temp-fio --bs=4k --ioengine=libaio --iodepth=64 --size=1G --direct=1 --rw=randwrite --filename=/dev/"+str(device)+" --numjobs=12 --time_based --runtime=5
--group_reporting
sudo fio --minimal -name=temp-fio --bs=4k --ioengine=libaio --iodepth=64 --size=1G --direct=1 --rw=randread --filename=/dev/"+str(device)+" --numjobs=12 --time_based --runtime=5
--group_reporting

Summary

UpCloud can offer clear price and performance benefits against AWS EC2 and DigitalOcean as shown in our benchmark test. We can also offer much predictable performance as can be seen from the standard deviations. Performance and predictability together mean that you don’t have to be afraid of sudden slowdowns and reserve extra resources to cope with that – again adding to the cost of your infrastructure. Performance in general means that you don’t need as many resources in the first place, to begin with. For example, try running a database on top of our MaxIOPS and you will realize what we are talking about.

Try UpCloud for free!

The results of these benchmarks show that you get more performance for your money on UpCloud, so why not take us for a test drive?

We have free trials available here and you can always contact [email protected] if you’d like to do more extensive testing or have some questions answered.
Cover photo by Sami Heiskanen