semget: No space left on device

On servers that run for a long period of time you may encounter an error message like the following.

   semget: No space left on device

Common places are the error.log from Apache or the postgresql.log, combined with the associated program not starting anymore. The cause seems to be that programs that didn't terminate properly still "use" semaphores in the kernel, although the corresponding process does not exist anymore. Semaphores are used for inter-process synchronisation.

If you have PostgreSQL installed, there is an easy way to get this problem solved. The ipcclean program removes all shared memory segments and semaphore sets owned by the current user. Hence the solution is:

• If you are running PostgreSQL, stop the currently running postmaster and associated processes. Then change the userid to postgres, the user PostgreSQL runs as, and execute the ipcclean program.
• If you are running Apache as well, stop the Apache server, change the userid to www-data and execute the ipcclean program.

If you would like to clean the semaphore pool on a regular bases, the following script snippet could be used in a cron job:

   apachectl stop
   su - -c /usr/lib/postgresql/bin/ipcclean www-data
   apachectl start

Please take into account that this will create a massive amount of output, so you should probably redirect the output to /dev/null or into a log file at your convenience.