{"id":2452,"date":"2015-07-29T16:39:06","date_gmt":"2015-07-29T13:39:06","guid":{"rendered":"https:\/\/upcloud.com\/global\/us\/resources\/tutorials\/troubleshoot-linux-memory-issues\/"},"modified":"2026-04-23T14:32:31","modified_gmt":"2026-04-23T13:32:31","slug":"troubleshoot-linux-memory-issues","status":"publish","type":"tutorial","link":"https:\/\/upcloud.com\/global\/resources\/tutorials\/troubleshoot-linux-memory-issues\/","title":{"rendered":"How to troubleshoot Linux server memory issues"},"content":{"rendered":"\n

Some unexpected behaviour on the server side may sometimes be caused by system resource limitations. Linux by its design aims to use all of the available physical memory as efficiently as possible, in practice, the Linux kernel follows a basic rule that a page of free RAM is wasted RAM. The system holds a lot more in RAM than just application data, most importantly mirrored data from storage drives for faster access. This debugging guide aims to explain how to identify how much of the resources are actually being used and how to recognise real resource outage issues.<\/p>\n\n\n\n\n\n

Process stopped unexpectedly<\/h2>\n\n\n\n

Suddenly killed tasks are often the result of the system running out of memory, which is when the so-called Out-of-memory (OOM) killer steps in. If a task gets killed to save memory, it gets logged into various log files stored at \/var\/log\/<\/p>\n\n\n\n

You can search the logs for messages of out-of-memory alerts.<\/p>\n\n\n\n

sudo grep -i -r 'out of memory' \/var\/log\/<\/pre>\n\n\n\n
Grep goes through all logs under the directory and, therefore, will show at least the just ran command itself from the \/var\/log\/auth.log. Actual log marks of OOM killed processes would look something like the following.<\/p>\n\n\n\n
kernel: Out of memory: Kill process 9163 (mysqld) score 511 or sacrifice child<\/pre>\n\n\n\n
The log note here shows the process killed was mysqld with pid 9163 and an OOM score of 511 at the time it was killed. Your log messages may vary depending on Linux distribution and system configuration.<\/p>\n\n\n\n
If, for example, a process crucial to your web application was killed as a result of an out-of-memory situation, you have a couple of options: reduce the amount of memory asked by the process, disallow processes to overcommit memory, or simply add more memory to your server configuration.<\/p>\n\n\n\n
\n
Try UpCloud for free!<\/a><\/div>\n<\/div>\n\n\n\n
Current resource usage<\/h2>\n\n\n\n
Linux comes with a few handy tools for tracking processes that can help identify possible resource outages. The command below, for example, can track memory usage.<\/p>\n\n\n\n
free -h<\/pre>\n\n\n\n
The command prints out current memory statistics, for example in a 1 GB system the output is something along the lines of the example underneath.<\/p>\n\n\n\n
                   total   used    free    shared  buffers cached\nMem:               993M    738M    255M    5.7M    64M     439M\n-\/+ buffers\/cache: 234M    759M\nSwap:              0B      0B      0B<\/pre>\n\n\n\n
It is important to distinguish between application-used memory, buffers, and caches here. On the Mem line of the output, it would appear that nearly 75% of our RAM is in use, but over half of the used memory is occupied by cached data.<\/p>\n\n\n\n
The difference is that while applications reserve memory for their own use, the cache is simply commonly used hard drive data that the kernel stores temporarily in RAM space for faster access, which on the application level is considered free memory.<\/p>\n\n\n\n
Keeping that in mind, it\u2019s easier to understand why used and free memory are listed twice. On the second line, the actual memory usage is conveniently calculated when taking into account the amount of memory occupied by buffers and cache.<\/p>\n\n\n\n
In this example, the system uses only 234MB of the total available 993MB, and no process is in danger of being killed to save resources.<\/p>\n\n\n\n
Another useful tool for memory monitoring is \u2018top\u2019, which displays useful, continuously updated information about processes\u2019 memory and CPU usage, runtime and other statistics. This is particularly useful for identifying exhaustive tasks for resources.<\/p>\n\n\n\n
top<\/pre>\n\n\n\n
You can scroll the list using the Page Up and Page Down buttons on your keyboard. The program runs in the foreground until canceled by pressing \u2018q\u2019 to quit. The resource usage is shown in percentages, giving an easy overview of your system\u2019s workload.<\/p>\n\n\n\n
top - 17:33:10 up 6 days,  1:22,  2 users,  load average: 0.00, 0.01, 0.05\nTasks:  72 total,   2 running,  70 sleeping,   0 stopped,   0 zombie\n%Cpu(s):  0.3 us,  0.0 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st\nKiB Mem:   1017800 total,   722776 used,   295024 free,    66264 buffers\nKiB Swap:        0 total,        0 used,        0 free.   484748 cached Mem\n\n  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND\n    1 root      20   0   33448   2784   1448 S  0.0  0.3   0:02.91 init\n    2 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kthreadd\n    3 root      20   0       0      0      0 S  0.0  0.0   0:00.02 ksoftirqd\/0\n    5 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kworker\/0:0H\n    6 root      20   0       0      0      0 S  0.0  0.0   0:01.92 kworker\/u2:0\n    7 root      20   0       0      0      0 S  0.0  0.0   0:05.48 rcu_sched\n<\/pre>\n\n\n\n
In the example output shown above, the system is idle, and the memory usage is nominal.<\/p>\n\n\n\n
Check if your process is at risk<\/h2>\n\n\n\n
If your server\u2019s memory is used up to the extent that it can threaten system stability, the Out-of-memory killer will choose which process to eliminate based on many variables, such as the amount of work done that would be lost and the total memory freed. Linux keeps a score for each running process, representing the likelihood of the process being killed in an OOM situation.<\/p>\n\n\n\n
This score is stored on file in \/proc\/<pid>\/oom_score, where pid is the identification number for the process you are looking into. The pid can be easily found using the following command.<\/p>\n\n\n\n
ps aux | grep <process name><\/pre>\n\n\n\n
The command output when searching for ‘mysql’, for example, would be similar to the example below.<\/p>\n\n\n\n
mysql     5872  0.0  5.0 623912 51236 ?        Ssl  Jul16   2:42 \/usr\/sbin\/mysqld<\/pre>\n\n\n\n
Here, the process ID is the first number on the row, 5872 in this case. It can then be used to obtain further information on this particular task.<\/p>\n\n\n\n
cat \/proc\/5872\/oom_score<\/pre>\n\n\n\n
The readout of this gives us a single numerical value for the chance of the OOM killer axing the process. The higher the number, the more likely the task is to be chosen if an out-of-memory situation should arise.<\/p>\n\n\n\n
If your important process has a very high OOM score, it may be wasting memory and should be looked into. However, a high OOM score, if the memory usage remains nominal, is no reason for concern. OOM killer can be disabled, but this is not recommended as it might cause unhandled exceptions in out-of-memory situations, possibly leading to a kernel panic or even a system halt.<\/p>\n\n\n\n
Disable over commit<\/h2>\n\n\n\n
In major Linux distributions, the kernel allows processes to request more memory than is currently free in the system by default to improve memory utilization. This is based on the heuristic that processes never truly use all the memory they request. However, if your system is at risk of running out of memory and you wish to prevent losing tasks to OOM killer, it is possible to disallow memory overcommit.<\/p>\n\n\n\n
To change how the system handles overcommit calls, Linux has an application called \u2018sysctl\u2019 that is used to modify kernel parameters at runtime. You can list all sysctl-controlled parameters using the following.<\/p>\n\n\n\n
sudo sysctl -a<\/pre>\n\n\n\n
The particular parameters that control memory are very imaginatively named vm.overcommit_memory and vm.overcommit_ratio. To change the overcommit mode, use the below command.<\/p>\n\n\n\n
sudo sysctl -w vm.overcommit_memory=2<\/pre>\n\n\n\n
This parameter has 3 different values:<\/p>\n\n\n\n