Articles / Linux

  • linux
  • emfile
  • ulimit
  • systemd
  • inotify
  • debugging

Why a Linux Process Gets EMFILE Even When ulimit Looks Generous

You run ulimit -n, see a comfortable 1048576, and then the service falls over with "Too many open files". Both facts are true. The number you checked belongs to your shell, and the failing process never inherited it.

The error is EMFILE, and a few different things can produce it. Here is how to find which one you have, in order of how often it turns out to be the culprit.

Read the limit off the process, not the shell

Always ask the running process what its limit is. The kernel keeps it in /proc:

pid=$(pidof myservice)
grep 'open files' /proc/$pid/limits
ls /proc/$pid/fd | wc -l

The first command shows the soft and hard limits that actually apply. The second counts descriptors currently open. If the count is close to the soft limit, you have your answer, and the rest is working out why the limit is lower than you expected.

Also note that ulimit -n in bash shows the soft limit. ulimit -Hn shows the hard one. A process may raise its own soft limit up to the hard limit, but never beyond it without privilege.

Why the service has a different limit from your shell

Limits are per process and inherited at fork. Your shell got its numbers from login configuration. A service started by systemd never went through that path.

  • /etc/security/limits.conf is applied by PAM at login. systemd services do not log in, so it does not touch them.
  • systemd sets its own default, which has historically been a soft limit of 1024 with a much higher hard limit.
  • Containers inherit whatever the runtime was started with, which may differ from the host again.

Check what systemd intends, then set it explicitly:

systemctl show myservice -p LimitNOFILE -p LimitNOFILESoft
sudo systemctl edit myservice
[Service]
LimitNOFILE=65536

Restart the service, then re-read /proc/$pid/limits. Do not trust the unit file alone; the proc file is the proof.

Quick detour: why a soft limit of 1024 is not a disaster by itself

The soft/hard split exists so that programs can ask for more when they need it. Go has done this for you since 1.19: the runtime raises the soft RLIMIT_NOFILE to the hard limit at startup. That is why a Go service often copes where a C program started the same way does not.

Other languages do not. A Python or Java service with soft 1024 and hard 524288 will hit the wall at 1024 while sitting under a huge ceiling it never reached for.

The EMFILE that is not about files at all

Here is the one that wastes an afternoon. inotify_init returns EMFILE when you hit the per-user inotify instance cap, not when you run out of descriptors.

The symptom: file watchers, dev servers, log shippers or editors report "Too many open files" while /proc/PID/fd shows only a handful. The relevant knobs:

sysctl fs.inotify.max_user_instances
sysctl fs.inotify.max_user_watches

The instance limit is per user, not per process. Ten tools each holding a few inotify instances can exhaust it together. Raising it is a sysctl change:

echo 'fs.inotify.max_user_instances = 1024' | sudo tee /etc/sysctl.d/60-inotify.conf
sudo sysctl --system

Running out of watches is a different error, ENOSPC, which confusingly reads as "no space left on device". Same subsystem, different message.

EMFILE or ENFILE?

They differ by one letter and by scope:

  • EMFILE: this process (or, for inotify, this user) hit its cap.
  • ENFILE: the whole system ran out of open file table entries, governed by fs.file-max.

If you see ENFILE, raising a per-process limit will not help. If you see EMFILE, raising fs.file-max will not help either.

Raising the limit versus finding the leak

A bigger limit only helps if the service legitimately needs more descriptors, such as a proxy with many concurrent connections. If the count climbs steadily and never falls, you have a leak, and a higher limit just delays the crash.

Watch the count over time and see what the descriptors point at:

watch -n1 "ls /proc/$pid/fd | wc -l"
ls -l /proc/$pid/fd | awk '{print $NF}' | sort | uniq -c | sort -rn | head

A pile of socket:[...] entries suggests connections not being closed. A pile of the same file path suggests a missing close in a loop. Anonymous inodes such as anon_inode:inotify point back at the previous section.

A checklist in order

  1. Read /proc/PID/limits for the failing process, not your shell.
  2. Compare against ls /proc/PID/fd | wc -l.
  3. If the count is low, suspect inotify and check fs.inotify.max_user_instances.
  4. If the soft limit is the problem, set LimitNOFILE= in the unit and verify after a restart.
  5. If the count only ever climbs, fix the leak instead of raising the number.

One caveat: I have described typical defaults loosely because distributions and systemd versions differ. Trust what /proc and sysctl tell you on your own machine over any number quoted here.