{"id":2278,"date":"2018-01-22T10:41:11","date_gmt":"2018-01-22T08:41:11","guid":{"rendered":"https:\/\/upcloud.com\/global\/us\/resources\/tutorials\/benchmark-magento-with-jmeter\/"},"modified":"2018-01-22T10:41:11","modified_gmt":"2018-01-22T08:41:11","slug":"benchmark-magento-with-jmeter","status":"publish","type":"tutorial","link":"https:\/\/upcloud.com\/global\/resources\/tutorials\/benchmark-magento-with-jmeter\/","title":{"rendered":"How to benchmark Magento Open Source with Apache JMeter"},"content":{"rendered":"\n

The open-source community edition of the Magento e-commerce<\/a> platform is a great option to set up an online storefront. Especially growing small new businesses will appreciate the low upfront charges and cost-effectiveness of hosting your store on a cloud provider.<\/p>\n\n\n\n

But not all clouds are made equal, and much of your store sales will depend on your capacity to serve customers. To be able to quantify performance between providers you will need a standardized test. This is where Apache JMeter<\/a> and Magento Performance Toolkit <\/a>come in.<\/p>\n\n\n\n

\n
Test hosting on UpCloud!<\/a><\/div>\n<\/div>\n\n\n\n

Apache JMeter<\/h2>\n\n\n\n

JMeter is an open-source load generator software written in Java. Originally designed for testing web applications, it is well-suited for trialling the functional behaviour and measuring the performance of a Magento site. As cloud service providers may offer very different performance levels depending on the available resources, JMeter will help you evaluate the offerings on a level playing field.<\/p>\n\n\n\n

It is important to be able to test and measure the actual capabilities of the servers hosting your application. The best way to gauge the performance is to run a benchmark, a series of simulated operations with comparable results across the providers.<\/p>\n\n\n\n

<\/a>Magento provides a performance toolkit<\/a> with a set of metrics that can be used to judge the performance and overall load capacity of your server.<\/p>\n\n\n\n

Preparing the Magento server<\/h2>\n\n\n\n

The best way to purely measure the differences between providers is to set up identical test sites with no actual product data. If you have not installed the open-source Magento site yet, check out our getting started guide<\/a> to get up and running. For the best results, install the site without any sample data. The test products will be generated in the next step.<\/p>\n\n\n\n

<\/a>Once you have a test site configured, you can continue with the benchmarking. However, before you can start load testing the site, you will need to go through a few steps to prepare the server for the task. Start by generating the test data for the purpose of this benchmark.<\/p>\n\n\n\n

Generating test data on your Magento server<\/h2>\n\n\n\n

The following is best used only on a Magento install without sample data. If you have already configured a storefront including the samples, you can uninstall the unneeded entries by following the instructions<\/a> provided at the Magento GitHub repository.<\/p>\n\n\n\n

With a clean, empty site, proceed by setting the following global parameter in your database.<\/p>\n\n\n\n

mysql -u root -p -e \"SET GLOBAL log_bin_trust_function_creators = 1;\"\nmysql -u root -p -e \"USE magento2; UPDATE quote SET is_active = 0 WHERE is_active = 1;\"<\/code><\/pre>\n\n\n\n
Then go to the bin directory in the Magento web root.<\/p>\n\n\n\n
cd \/var\/www\/html\/magento2\/bin\/<\/code><\/pre>\n\n\n\n
You should see a PHP script simply named magento<\/em>. It can be used to set up the site for testing by generating a suitably sized set of test data.<\/p>\n\n\n\n
If the script does not have execution rights by default, enable them with the next command.<\/p>\n\n\n\n
sudo chmod +x magento<\/code><\/pre>\n\n\n\n
Depending on the amount of data to be generated and indexed, the process might take quite some time. We recommend using the small data set for convenience, as large data sets might take hours to index.<\/p>\n\n\n\n
First, generate the product data with the following command while including the -s parameter to skip the initial indexing.<\/p>\n\n\n\n
.\/magento setup:performance:generate-fixtures -s \/var\/www\/html\/magento2\/setup\/performance-toolkit\/profiles\/ce\/small.xml > ~\/magento-setup.out<\/code><\/pre>\n\n\n\n
Once the data has been generated, index it manually with the next command.<\/p>\n\n\n\n
.\/magento indexer:reindex<\/code><\/pre>\n\n\n\n
When the test data is ready, you might need to fix some file permissions. Run the following command to make sure everything is set to the right owner for the site to work.<\/p>\n\n\n\n
sudo chown -R www-data:www-data \/var\/www\/html\/magento2\/<\/pre>\n\n\n\n
\"Magento<\/figure>\n\n\n\n
<\/a>You should now see a number of generic categories with simple placeholder products when browsing your test storefront like in the picture above.<\/p>\n\n\n\n
Configure your Magento site for benchmarking<\/h2>\n\n\n\n
Besides generating the test data, you should also adjust a few settings in the Magento administration control panel.<\/p>\n\n\n\n
Open your site\u2019s admin page and log in, then browse to the following menu.<\/p>\n\n\n\n
1. Stores > Configuration > General > Web > Search Engine Optimization<\/h4>\n\n\n\n
Enable the web server rewrites in the first category if not already set. Unselect the \u201cUse system value\u201d<\/em> to change any of the default settings that are greyed out.<\/p>\n\n\n\n
Use Web Server Rewrites \u201cYes\u201d<\/em><\/p>\n\n\n\n
Then save the change by clicking the Save Config<\/em> button at the top of the page.<\/p>\n\n\n\n
\"Use<\/figure>\n\n\n\n
2. Stores > Configuration > Catalog > Catalog > Storefront<\/h4>\n\n\n\n
You will find the two settings that are set to \u201cNo\u201d<\/em> by default, and change them to \u201cYes\u201d<\/em> as shown below.<\/p>\n\n\n\n
Use Flat Catalog Category \u201cYes\u201d<\/em>
Use Flat Catalog Product \u201cYes\u201d<\/em><\/p>\n\n\n\n
Confirm the changes by clicking the Save Config<\/em> button.<\/p>\n\n\n\n
\"Flat<\/figure>\n\n\n\n
3. Stores > Configuration > Advanced > Admin > Security<\/h4>\n\n\n\n
Enable Admin Account Sharing to run a multithreaded admin pool by setting the following.<\/p>\n\n\n\n
Admin Account Sharing \u201cYes\u201d<\/em><\/p>\n\n\n\n
With the setting changed, apply it by clicking the Save Config<\/em> button at the top of the page.<\/p>\n\n\n\n
\"Share<\/figure>\n\n\n\n
4. Stores > Configuration > Advanced > Developer<\/h4>\n\n\n\n
Click to expand the JavaScript Settings<\/strong> section.<\/p>\n\n\n\n
Set Merge JavaScript Files to \u201cYes.\u201d<\/em><\/p>\n\n\n\n
Click to expand the CSS Settings<\/strong> section.<\/p>\n\n\n\n
Set Merge CSS Files to \u201cYes.\u201d<\/em><\/p>\n\n\n\n
Again, apply the changes by clicking the Save Config<\/em> button at the top of the page.<\/p>\n\n\n\n
\"Merge<\/figure>\n\n\n\n
If you get an error when saving any changes, make sure the file permissions and ownership are set correctly and try saving again. Similarly, reset the permissions if you come up with an empty menu in the admin control panel.<\/p>\n\n\n\n
sudo find \/var\/www\/html\/magento2\/ -type d -exec chmod 775 {} ;\nsudo find \/var\/www\/html\/magento2\/ -type f -exec chmod 664 {} ;\nsudo chown -R www-data:www-data \/var\/www\/html\/magento2\/<\/pre>\n\n\n\n
Afterwards, check your site cache management page and update any invalidated data.<\/p>\n\n\n\n
5. System > Cache Management<\/h4>\n\n\n\n
Select all cache types and Refresh<\/em> the caches by clicking the Submit<\/em> button.<\/p>\n\n\n\n
\"Cache<\/figure>\n\n\n\n
6. System > Index Management<\/h4>\n\n\n\n
Lastly, check the site indexes and make sure all show a ready status.<\/p>\n\n\n\n
Set all to the \u201cUpdate by Schedule\u201d<\/em> mode by selecting all and clicking the Submit<\/em> button.<\/p>\n\n\n\n
\"Index<\/figure>\n\n\n\n
If you get an error while updating the index settings, confirm you enabled the database global to trust function creators.<\/p>\n\n\n\n
mysql -u root -p -e \"SET GLOBAL log_bin_trust_function_creators = 1;\"<\/pre>\n\n\n\n
Also, make sure you have the following cronjob entries enabled. Open the crontab on your Magento site users for example with the following command.<\/p>\n\n\n\n
sudo crontab -u magento -e<\/pre>\n\n\n\n
Choose the editor you wish to use, then check that the following lines appear in the file. If no, add them, save the file, and exit.<\/p>\n\n\n\n
* * * * * \/usr\/bin\/php \/var\/www\/html\/magento2\/bin\/magento cron:run | grep -v 'Ran jobs by schedule' >> \/var\/www\/html\/magento2\/var\/log\/magento.cron.log\n* * * * * \/usr\/bin\/php \/var\/www\/html\/magento2\/update\/cron.php >> \/var\/www\/html\/magento2\/var\/log\/update.cron.log\n* * * * * \/usr\/bin\/php \/var\/www\/html\/magento2\/bin\/magento setup:cron:run >> \/var\/www\/html\/magento2\/var\/log\/setup.cron.log<\/pre>\n\n\n\n
<\/a>When the indexers are run the next time, the status on the management page should show Processing<\/em> to indicate that the indexes are being updated. After a moment, all indexes should show green again.<\/p>\n\n\n\n
Installing JMeter on your load-generating computer<\/h2>\n\n\n\n
With the Magento site configured for benchmarking, continue by installing JMeter on another computer that will be used to generate the test load.<\/p>\n\n\n\n
JMeter is available on any operating system that supports Java Runtime Environment. The following instructions are for getting JMeter installed and ready on most Linux variants. You can find other download links<\/a> better suitable for your OS on their website.<\/p>\n\n\n\n
First, install the required packages e.g. with the command below on Ubuntu or Debian derivates. Use of version 8 of JRE is required until JMeter finishes adding support for the new version 9.<\/p>\n\n\n\n
sudo apt install curl openjdk-8-jre unzip<\/pre>\n\n\n\n
Then, download and unpack JMeter on your load generator system. The Magento benchmark plan provided by the performance toolkit was developed for JMeter version 3.1.<\/p>\n\n\n\n
curl -k -O http:\/\/apache.mirror.anlx.net\/jmeter\/binaries\/apache-jmeter-3.1.tgz ~\/\ntar zxf ~\/apache-jmeter-3.1.tgz<\/pre>\n\n\n\n
You will also need to install the JSON plugin for JMeter. Download it with the command below and unpack it to the JMeter directory.<\/p>\n\n\n\n
curl -k -O https:\/\/jmeter-plugins.org\/files\/packages\/jpgc-json-2.6.zip\nunzip jpgc-json-2.6.zip -d ~\/apache-jmeter-3.1\/<\/pre>\n\n\n\n
You can confirm the app is working with the following command.<\/p>\n\n\n\n
.\/apache-jmeter-3.1\/bin\/jmeter -v<\/pre>\n\n\n\n
<\/a>Once you have JMeter and the JSON plugin downloaded and ready, continue with the instructions on how to use it and how to run the benchmarks.<\/p>\n\n\n\n
Using JMeter to benchmark Magento<\/h2>\n\n\n\n
JMeter can be run either on the command line or with the graphical user interface. Below you will find the steps for both of these options to help with the benchmarking.<\/p>\n\n\n\n
Note: JMeter GUI should only be used for scenario debugging and viewing reports.<\/p>\n\n\n\n
The command line option should be used for real-world load testing, as it requires significantly fewer resources. For best results, make sure your load-generating computer is powerful enough to run the benchmark without limiting the throughput.<\/p>\n\n\n\n
To begin, download the testing plan from the Magento performance toolkit repository.<\/p>\n\n\n\n
cd ~\/apache-jmeter-3.1\/bin\/\ncurl -k -O https:\/\/raw.githubusercontent.com\/magento\/magento2\/2.3-develop\/setup\/performance-toolkit\/benchmark.jmx<\/code><\/pre>\n\n\n\n
The benchmark plan also uses preconfigured search terms supplied in the performance toolkit. Create a files<\/em> directory and download the search_terms.<\/em>csv into that folder.<\/p>\n\n\n\n
mkdir ~\/apache-jmeter-3.1\/bin\/files\ncd ~\/apache-jmeter-3.1\/bin\/files\ncurl -k -O https:\/\/raw.githubusercontent.com\/magento\/magento2\/2.3-develop\/setup\/performance-toolkit\/files\/search_terms.csv<\/code><\/pre>\n\n\n\n
<\/a>The benchmark plan includes a number of possible interactions a customer might have on your Magento site. It can simulate purchases, account management, and admin actions to get a good estimate of the load capacity of your server.<\/p>\n\n\n\n
JMeter on the command line<\/h2>\n\n\n\n
Running JMeter on the command line is really quite easy. You can find the app in the downloaded director for example with the following command.<\/p>\n\n\n\n
cd ~\/apache-jmeter-3.1\/bin\/<\/code><\/pre>\n\n\n\n
Start the application in the non-GUI mode by invoking the parameter -n<\/em>.<\/p>\n\n\n\n
.\/jmeter -n<\/code><\/pre>\n\n\n\n
Without the rest of the required parameters, JMeter will simply print out a list of available command-line options.<\/p>\n\n\n\n
To run the load test, you will need to include the benchmark plan, log and result files, and your server details as described in the example below.<\/p>\n\n\n\n
Replace the <public IP\/domain><\/em> with your server\u2019s public IP address or domain name, depending on how you configured Magento at installation. Also, include the path to the <admin_page><\/em> in the control panel as well as the <admin_username><\/em> and <admin_password><\/em>.<\/p>\n\n\n\n
.\/jmeter -n -t .\/benchmark.jmx -j .\/benchmark.log -l .\/benchmark-results.jtl \n-Jhost=<public IP\/domain> \n-Jbase_path=\/ \n-Jadmin_path=<admin_page> \n-Jadmin_user=<admin_username> \n-Jadmin_password=<admin_password> \n-JfrontendPoolUsers=90 \n-JadminPoolUsers=10<\/code><\/pre>\n\n\n\n
When run, you will see an output similar to the example below. It already shows some initial performance metrics with the response times and the average throughput by thread group. For a more detailed report, continue on to the results sections below to learn how to read the benchmark results.<\/p>\n\n\n\n
Creating summariser <summary>\nCreated the tree successfully using .\/benchmark.jmx\nStarting the test @ Wed Jan 10 12:11:00 GMT 2018 (1515586260784)\nWaiting for possible Shutdown\/StopTestNow\/Heapdump message on port 4445\nsummary + 1 in 00:00:00 = 17.2\/s Avg: 16 Min: 16 Max: 16 Err: 0 (0.00%) Active: 1 Started: 1 Finished: 0\nsummary + 1446 in 00:00:29 = 49.9\/s Avg: 214 Min: 0 Max: 3608 Err: 0 (0.00%) Active: 16 Started: 101 Finished: 85\nsummary = 1447 in 00:00:29 = 49.8\/s Avg: 214 Min: 0 Max: 3608 Err: 0 (0.00%)\nsummary + 370 in 00:00:18 = 20.5\/s Avg: 319 Min: 0 Max: 2666 Err: 0 (0.00%) Active: 0 Started: 1 Finished: 1\nsummary = 1817 in 00:00:47 = 38.6\/s Avg: 235 Min: 0 Max: 3608 Err: 0 (0.00%)\nTidying up ... @ Wed Jan 10 12:11:48 GMT 2018 (1515586308196)\n... end of run<\/code><\/pre>\n\n\n\n
If the test ends abruptly, check that the database global to trust function creators is enabled.<\/a><\/p>\n\n\n\n
mysql -u root -p -e \"SET GLOBAL log_bin_trust_function_creators = 1;\"<\/code><\/pre>\n\n\n\n
Also, make sure your load generator can reach your Magento server without issue.<\/p>\n\n\n\n
Reading results with JMeter GUI<\/h2>\n\n\n\n
Open the JMeter application in the GUI mode with the following command in the same ~\/apache-<\/em>jmeter-3.1\/bin\/<\/em> directory, but without any parameters.<\/p>\n\n\n\n
.\/jmeter<\/code><\/pre>\n\n\n\n
The test plan benchmark.jmx<\/em> can be opened in the application for viewing, debugging, and further development. Click the folder icon or select Open under the File menu to load the benchmark plan.<\/p>\n\n\n\n
\"JMeter<\/figure>\n\n\n\n
JMeter supports a number of ways to view the test results such as the Aggregate Report<\/em>, Results Tree<\/em>, and Graph Results<\/em>.<\/p>\n\n\n\n
Add a new view by selecting the Benchmark<\/em> top section in the left panel, then browse through to the following menu.<\/p>\n\n\n\n
Edit > Add > Listener > Graph Results<\/p>\n\n\n\n
\"JMeter<\/figure>\n\n\n\n
You can open the saved test result file by clicking the Browse<\/em> button for example in the Graph Results<\/em> view. Loading up the results will draw a graph that displays the server’s overall throughput performance with the corresponding statistical values beneath.<\/p>\n\n\n\n
\"JMeter<\/figure>\n\n\n\n
You may also like to review the benchmark results in JMeter using the other two listeners already mentioned.<\/p>\n\n\n\n
View Results Tree<\/strong> option shows all requests and responses made during the scenario in a report tree. In this report, you can find information about response times, headers, and response codes.<\/a><\/p>\n\n\n\n
Aggregate Report<\/strong> contains collected information about all of the requests, including counts, min, max, average, error rate, and approximate throughput.<\/p>\n\n\n\n
Summary<\/h2>\n\n\n\n
JMeter and the Magento performance toolkit enable you to test your e-commerce server capabilities, from the impact of customization to differences between cloud service providers. With the ability to generate sample data for and run test scenarios, you have everything needed to make the right choices based on repeatable test results.<\/p>\n\n\n\n
If you would like to further optimize your Magento server for the best performance per resource, you can easily change your server configuration in the server settings. Check out our other Cloud Server plans<\/a> to learn more.<\/p>\n","protected":false},"author":3,"featured_media":26902,"comment_status":"open","ping_status":"closed","template":"","community-category":[223],"class_list":["post-2278","tutorial","type-tutorial","status-publish","has-post-thumbnail","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/2278","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial"}],"about":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/types\/tutorial"}],"author":[{"embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/comments?post=2278"}],"version-history":[{"count":0,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/2278\/revisions"}],"wp:attachment":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/media?parent=2278"}],"wp:term":[{"taxonomy":"community-category","embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/community-category?post=2278"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}