Phusion Passenger and CageFS
============================

Current package version (unlike 5.3.5) should not require additional actions
by the server administrator besides reloading of httpd and nginx. Just install
"cagefs" and "passenger-cagefs" packages (could be done in any order).

To make Passenger Apache or Nginx module working when CageFS is enabled
the following directories must be available for system users that run
Node.js or Ruby applications:

1. Passenger files /usr/share/passenger
2. Node.js or Ruby interpreter, libraries, etc. (likely already configured)
3. Rbenv wrapper files for ruby applications /var/lib/rbenv
4. Temporary directory for application runtime files - instance registry
   directory, read and write permissions
5. Temporary directory that is used for handshake during application startup,
   read and write permissions (since passenger-5.3).


Diagnostics
-----------

Improper or inconsistent configuration of Passenger and CageFS leads
to 500 code in response to HTTP requests if URL is handled by Node.js
or Ruby application:

    HTTP/1.1 500 Internal Server Error
    Server: nginx
    Date: Tue, 23 Apr 2019 04:30:29 GMT
    Status: 500 Internal Server Error

Examples of some error messages written to Apache or Nginx error log
with brief description are shown below.

/usr/share/passenger directory is missed in CageFS template or
alternatively the mount point is not configured in /etc/cagefs/cagefs.mp:

    App 2846 output: module.js:545
    App 2846 output:     throw err;
    App 2846 output:     ^
    App 2846 output:
    App 2846 output: Error: Cannot find module '/usr/share/passenger/helper-scripts/node-loader.js'
    App 2846 output:     at Function.Module._resolveFilename (module.js:543:15)
    App 2846 output:     at Function.Module._load (module.js:470:25)
    App 2846 output:     at Function.Module.runMain (module.js:690:10)
    App 2846 output:     at startup (bootstrap_node.js:194:16)
    App 2846 output:     at bootstrap_node.js:666:3
    [ E 2019-04-19 17:53:23.7135 2717/Tb age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /var/www/vhosts/nodejs.example.com/app: The application process exited prematurely.


Ruby wrappers /var/lib/rbenv directory is not mounted to CageFS:

    App 192557 output: rbenv: version `2.3.7' is not installed (set by /var/www/vhosts/ruby.example.com/app/.ruby-version)
    [ E 2019-04-19 11:22:49.3656 100802/Tx age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /var/www/vhosts/ruby.example.com/app: The application process exited prematurely.
      Error ID: 794fdddb
      Error details saved to: /tmp/passenger-error-lizQOw.html

    [ E 2019-04-19 11:22:49.3748 100802/T7 age/Cor/Con/CheckoutSession.cpp:276 ]: [Client 1-4] Cannot checkout session because a spawning error occurred. The identifier of the error is 794fdddb. Please see earlier logs for details about the error.


Instance registry directory is missed in /etc/cagefs/cagefs.mp:

    [ W 2019-04-23 11:30:25.0236 1443/T4 age/Cor/App/Poo/AnalyticsCollection.cpp:102 ]: Process (pid=1649, group=/var/www/vhosts/ruby.example.com/app (production)) no longer exists! Detaching it from the pool.
    [ N 2019-04-23 11:30:25.0258 1443/T4 age/Cor/CoreMain.cpp:1118 ]: Checking whether to disconnect long-running connections for process 1649, application /var/www/vhosts/ruby.example.com/app (production)
    App 89798 output: Error: The application encountered the following error: No such file or directory - connect(2) for /run/passenger/registry/passenger.eSFVF8y/apps.s/ruby.iFMsRg3uEMcRqYS51cxuWa54lZSKyUTyjMCL0eMcK3A (Errno::ENOENT)
    App 89798 output: 
    App 89798 output:     /usr/share/passenger/phusion_passenger/request_handler.rb:302:in `initialize'
    App 89798 output:     /usr/share/passenger/phusion_passenger/request_handler.rb:302:in `new'
    App 89798 output:     /usr/share/passenger/phusion_passenger/request_handler.rb:302:in `block in create_unix_socket_on_filesystem'
    App 89798 output:     /usr/share/passenger/phusion_passenger/utils.rb:65:in `block in retry_at_most'
    App 89798 output:     /usr/share/passenger/phusion_passenger/utils.rb:63:in `times'
    App 89798 output:     /usr/share/passenger/phusion_passenger/utils.rb:63:in `retry_at_most'
    App 89798 output:     /usr/share/passenger/phusion_passenger/request_handler.rb:299:in `create_unix_socket_on_filesystem'
    App 89798 output:     /usr/share/passenger/phusion_passenger/request_handler.rb:100:in `initialize'
    App 89798 output:     /usr/share/passenger/helper-scripts/rack-loader.rb:123:in `new'
    App 89798 output:     /usr/share/passenger/helper-scripts/rack-loader.rb:123:in `block in <module:App>'
    App 89798 output:     /usr/share/passenger/phusion_passenger/loader_shared_helpers.rb:380:in `run_block_and_record_step_progress'
    App 89798 output:     /usr/share/passenger/helper-scripts/rack-loader.rb:120:in `<module:App>'
    App 89798 output:     /usr/share/passenger/helper-scripts/rack-loader.rb:28:in `<module:PhusionPassenger>'
    App 89798 output:     /usr/share/passenger/helper-scripts/rack-loader.rb:27:in `<main>'
    [ E 2019-04-23 11:30:29.3949 1443/Tg age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /var/www/vhosts/ruby.example.com/app: The application encountered the following error: No such file or directory - connect(2) for /run/passenger/registry/passenger.eSFVF8y/apps.s/ruby.iFMsRg3uEMcRqYS51cxuWa54lZSKyUTyjMCL0eMcK3A (Errno::ENOENT)
      Error ID: 6e5a7cd6
      Error details saved to: /tmp/passenger-error-L0G6QA.html

    [ E 2019-04-23 11:30:29.4089 1443/T5 age/Cor/Con/CheckoutSession.cpp:276 ]: [Client 1-2] Cannot checkout session because a spawning error occurred. The identifier of the error is 6e5a7cd6. Please see earlier logs for details about the error.


Spawn handshake directory or alternatively TMPDIR environment is not configured.
Files are created in global /tmp or HTTP server PrivateTmp directory
that is absent in CageFS user environment:

    App 2304 output: Warning: cannot open /tmp/passenger.spawn.XXXXHcguds/envdump/user_info for writing
    App 2304 output: Warning: cannot open /tmp/passenger.spawn.XXXXHcguds/envdump/envvars for writing
    App 2304 output: Warning: cannot open /tmp/passenger.spawn.XXXXHcguds/envdump/user_info for writing
    App 2304 output: Warning: cannot open /tmp/passenger.spawn.XXXXHcguds/envdump/envvars for writing
    App 2304 output: Warning: Cannot create file /tmp/passenger.spawn.XXXXHcguds/response/steps/subprocess_spawn_env_setupper_before_shell/state: No such file or directory (errno=2)
    App 2304 output: Warning: Cannot create file /tmp/passenger.spawn.XXXXHcguds/response/steps/subprocess_spawn_env_setupper_before_shell/begin_time_monotonic: No such file or directory (errno=2)
    App 2304 output: Warning: Cannot create file /tmp/passenger.spawn.XXXXHcguds/response/steps/subprocess_spawn_env_setupper_before_shell/end_time_monotonic: No such file or directory (errno=2)
    App 2304 output: Warning: Cannot create file /tmp/passenger.spawn.XXXXHcguds/response/steps/subprocess_spawn_env_setupper_after_shell/state: No such file or directory (errno=2)
    App 2304 output: Warning: Cannot create file /tmp/passenger.spawn.XXXXHcguds/response/steps/subprocess_spawn_env_setupper_after_shell/begin_time_monotonic: No such file or directory (errno=2)
    App 2304 output: Warning: Cannot create file /tmp/passenger.spawn.XXXXHcguds/response/steps/subprocess_spawn_env_setupper_after_shell/state: No such file or directory (errno=2)
    App 2304 output: Warning: Cannot create file /tmp/passenger.spawn.XXXXHcguds/response/steps/subprocess_spawn_env_setupper_after_shell/begin_time_monotonic: No such file or directory (errno=2)
    App 2304 output: Error: Cannot open '/tmp/passenger.spawn.XXXXHcguds/args.json' for reading: No such file or directory (errno=2)
    App 2304 output:      (empty)
    App 2304 output: Warning: Cannot create file /tmp/passenger.spawn.XXXXHcguds/response/steps/subprocess_spawn_env_setupper_after_shell/state: No such file or directory (errno=2)
    App 2304 output: Warning: Cannot create file /tmp/passenger.spawn.XXXXHcguds/response/steps/subprocess_spawn_env_setupper_after_shell/begin_time_monotonic: No such file or directory (errno=2)
    App 2304 output: Warning: Cannot create file /tmp/passenger.spawn.XXXXHcguds/response/steps/subprocess_spawn_env_setupper_after_shell/end_time_monotonic: No such file or directory (errno=2)
    App 2304 output: Warning: Cannot create file /tmp/passenger.spawn.XXXXHcguds/response/error/category: No such file or directory (errno=2)
    App 2304 output: Warning: Cannot create file /tmp/passenger.spawn.XXXXHcguds/response/error/summary: No such file or directory (errno=2)
    App 2304 output: Warning: Cannot create file /tmp/passenger.spawn.XXXXHcguds/response/error/advanced_problem_details: No such file or directory (errno=2)
    [ E 2019-04-19 17:51:54.8717 1450/Tb age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /var/www/vhosts/nodejs.example.com/app: The application process exited prematurely.
      Error ID: 9c88a6f8
      Error details saved to: /tmp/passenger-error-GfMR0N.html

    [ E 2019-04-19 17:51:54.8802 1450/T5 age/Cor/Con/CheckoutSession.cpp:276 ]: [Client 1-1] Cannot checkout session because a spawning error occurred. The identifier of the error is 9c88a6f8. Please see earlier logs for details about the error.


Temporary directories have not been created during server boot,
see tmpfiles.d(5):

    [ W 2019-04-22 17:45:05.8823 27344/T1 age/Wat/WatchdogMain.cpp:1005 ]: WARNING: unable to perform privilege escalation vulnerability detection:

     - Security check skipped on /run/passenger/registry: stat() failed: No such file or directory (errno=2)
     - Security check skipped on /run/passenger: stat() failed: No such file or directory (errno=2)
    [ W 2019-04-22 17:45:05.8834 27344/T1 age/Wat/WatchdogMain.cpp:1005 ]: WARNING: unable to perform privilege escalation vulnerability detection:

     - Security check skipped on /run/passenger/spawn: stat() failed: No such file or directory (errno=2)
     - Security check skipped on /run/passenger: stat() failed: No such file or directory (errno=2)
    [Mon Apr 22 17:45:05.895907 2019] [passenger:error] [pid 1447:tid 139774458599552] *** Passenger could not be initialized because of this error: Unable to start the Phusion Passenger watchdog because it encountered the following error during startup: Cannot create a subdirectory inside instance registry directory /run/passenger/registry: No such file or directory (errno=2)


Variants to get proper configuration
------------------------------------

Obviously CageFS may be disabled for system users who owns such applications
but likely this is the least desirable solution.

/usr/share/passenger could be added to mount points in /etc/cagefs/cagefs.mp
or to CageFS filesystem template by an additional file in /etc/cagefs/conf.d

/var/lib/rbenv could be added to /etc/cagefs/cagefs.mp as a read-only mount
point.


Directories for temporary files
-------------------------------

/tmp is used by default and it may be affected by PrivateTmp
systemd feature, see systemd.exec(5).

Mounting /tmp directly to CageFS with PrivateTmp disabled for httpd and nginx
services is against the idea of CageFS.

A bit better way is to move apache and nginx temporary directories away
from system-wide tmp by setting TMPDIR environment variable e.g. through
"Environment" systemd unit configuration parameter, see systemd.exec(5).
SetEnv / PassEnv directives of apache mod_env do not help for this purpose.

Registry directory is a configurable parameter:
PassengerInstanceRegistryDir for Apache
https://www.phusionpassenger.com/library/config/apache/reference/#passengerinstanceregistrydir
and passenger_instance_registry_dir for Nginx
https://www.phusionpassenger.com/library/config/nginx/reference/#passenger_instance_registry_dir
The only inconvenience is that the same value should be passed
to the passenger-status command.

Spawn handshake directory is configurable only through TMPDIR environment
variable in passenger versions 5.3-6.0 (it is not used in earlier versions).
Plesk provides a patched passenger build that has additional
server-wide parameter PassengerSpawnDir for Apache module and
passenger_spawn_dir for Nginx module. The patch has been accepted
to the passenger-6.0 upstream branch however it was submitted after
6.0.2 release.

The suggested way is to set passenger instance registry directory
and spawn handshake directory to some paths outside of /tmp,
e.g. in /run or /var. Any directory for passenger temporary files
must be explicitly configured in /etc/cagefs/cagefs.mp otherwise
user applications will be unable to communicate to main passenger
processes. If such directories reside on tmpfs, ensure that they
are created at the boot time.


Configuration prepared by this package
--------------------------------------

- The following directories are prepared for registry and for application
  spawn handshake

       /run/passenger/registry/
       /run/passenger/spawn/

  on CloudLinux-7 and CloudLinux-8 they are configured in the
  /lib/tmpfiles.d/passenger.conf file

       d /run/passenger/registry  755 root root
       d /run/passenger/spawn     755 root root

  More tight 711 permissions will provide better user isolation
  however it will break passenger-status command for ordinary users.

- Apache and Nginx configuration snippets to use directories for
  temporary files should be in action with no additional efforts:

  /etc/httpd/conf.d/zz-passenger-cagefs.conf

       <IfModule mod_passenger.c>
           PassengerInstanceRegistryDir /run/passenger/registry
           PassengerSpawnDir /run/passenger/spawn
       </IfModule>

  This snippet must be included after the related LoadModule directive.

  /etc/nginx/conf.d/phusion-passenger-cagefs.conf

       passenger_instance_registry_dir /run/passenger/registry;
       passenger_spawn_dir /run/passenger/spawn;

- The following directories are added as CageFS mount points to
  the /etc/cagefs/cagefs.mp file using RPM package trigger

       !/usr/share/passenger/
       !/var/lib/rbenv
       # For CloudLinux-7 and CloudLinux-8:
       /run/passenger

  The following script provided by this package could be used to restore mount
  points if they are lost for some reason.

       passenger-add-cagefs-mp

  Previously /usr/share/passenger was added to skeleton template by means of
  /etc/cagefs/conf.d/passenger.cfg file but it can cause upgrade problems since
  skeleton updates once a day. Different passenger version could exist in and
  outside of CageFS user directories, so mount point is considered more safe.

- The passenger package provided by Plesk is patched in order to auto-detect
  appropriate instance registry directory value in the Passenger CLI utilities,
  such as passenger-status and passenger-config. The detection is based on the
  /run/passenger/registry directory presence. So in order for these utilities
  to work properly you need to complete configuration as mentioned below after
  installing the passenger-cagefs package. Similarly, after removing the package
  ensure that the directory was removed by the package manager (reload the web
  servers and remove it, if not).

To complete configuration, reload apache and nginx services.
On CloudLinux-7 or CloudLinux-8 run:

       systemctl reload nginx httpd

See also https://www.plesk.com/kb/support/how-to-run-node-js-ruby-applications-on-plesk-server-with-cloudlinux-os-and-cagefs-enabled/ .

Check that applications can respond to HTTP request, inspect
/var/log/httpd/error_log and /var/log/nginx/error.log for error messages.

Now hosting users should be able to run Node.js and Ruby applications
on their web sites.
