Non-containerized applications can be observed using dfctl run and dfctl service register, depending on the type of application being observed and the deployment scenario. This document will outline several common ways of using dfctl run and dfctl service register to observe applications in these modes.
Typically, you would use the steps detailed in this document when you want to observe an application that is launched via a script or executable. If the application is configured to run as a system service (eg, launched at startup via systemd), the dfctl service register command can be used to modify the service configuration file; this will add the appropriate LD_PRELOAD and DF_MANIFEST environment settings required to allow the service to be observed at service/machine/VM start up.
Certain types of applications and application deployment scenarios don’t work with Deepfactor. Although the limitations listed below aren’t specific to running in a non-containerized fashion, it’s prudent to understand the interactions between Deepfactor, other system components, and other instrumentation software.
Deepfactor uses an LD_PRELOAD library for its instrumentation and observability functionality. LD_PRELOAD libraries are loaded by the runtime loader (ld.so) during application launch, before the application starts executing and are a popular way for instrumentation software to be deployed and removed easily since no code or binary change is required.To remove an LD_PRELOAD based instrumentation software package, one can simply unset the LD_PRELOAD environment variable. The dfctl run / dfctl service register commands automatically set / unset LD_PRELOAD as required for Deepfactor when an application is being observed.
Using LD_PRELOAD means that Deepfactor must abide by operating system limitations placed on LD_PRELOAD functionality, as well as certain Deepfactor specific limitations. Specifically, LD_PRELOAD cannot be used in scenarios where setuid/setguid applications are being launched, and cannot be used for static linked applications. Deepfactor will detect and report these situations via a warning message in the management portal since itcannot observe these applications.
Even though your application may not directly contain setuid programs, a startup script used by your application may invoke setuid programs installed on the system. For example, the sudo command is sometimes used in application startup scripts to elevate process privileges to root. However, sudo is a setuid program itself, and attempting to run sudo in a Deepfactor enabled environment will result in no telemetry being sent to the management portal for anything run using sudo.
Another limitation that is sometimes encountered happens when Deepfactor is used in conjunction with another monitoring or instrumentation tool that also uses LD_PRELOAD. While it’s possible, in theory, to have multiple LD_PRELOAD library settings defined, Deepfactor will not observe application behavior in these scenarios. If LD_PRELOAD is set by another application when Deepfactor is used, Deepfactor will choose to not set its own LD_PRELOAD variable. We do this because we cannot generally determine with 100% certainty what the other LD_PRELOAD library may be doing, and we prefer to ensure the application continues to function properly at the expense of not observing in these scenarios.
A final limitation that is worth being aware of is specifically what dfctl run can execute. When launching applications using dfctl run, the –cmd argument must be an actual executable or executable script. Shell builtins cannot be used as run arguments (eg, this means you cannot issue a command such as “dfctl run –cmd export FOO=BAR /usr/bin/myprogram”.
Simple scenarios using dfctl run
You can use dfctl run to run standard executables without much fuss:
dfctl run -a appname -c componentname –cmd /usr/bin/myprogram
You can use dfctl run to run standard “startup-style” scripts similarly:
dfctl run -a appname -c componentname –cmd /usr/bin/startup.sh
Working with programs launched with sudo
Scenarios involving sudo launching a single program can typically be done as follows:
sudo -E dfctl run -a app -c component –cmd /usr/bin/myprogram
In the above example, dfctl will be launched as root via sudo and will in turn launch /usr/bin/myprogram with Deepfactor. Note the inclusion of the -E sudo command line parameter to ensure that the caller's environment variables are available to dfctl.
Working with scripts that contain embedded sudo
Sometimes, you may have an application startup script that itself uses sudo internally to launch various application components as root. In this scenario, the simplest option is to run the script itself via sudo dfctl:
sudo -E dfctl run -a app -c component –cmd /usr/bin/startup.sh
In this example, even if startup.sh contains lines which themselves use sudo to launch other programs, since no privilege escalation is being performed by the “inner” sudo, Deepfactor will work properly. Note that the -E option must be used in the command above to preserve the LD_PRELOAD environment variable and the user running the sudo command must have permission to inherit environment variables (set in /etc/sudoers). Also note that any embedded sudo lines in the startup script must be modified to include the -E option.
Working with shell builtins
If your application’s launch requires setting an environment variable and is currently invoked using something like this:
… you can launch this application using Deepfactor by creating a launch script as follows:
… and then launching that script using dfctl run:
dfctl run -a app -c component –cmd my_new_launcher.sh
Working with systemd services
Systemd services can be observed using dfctl service register servicename. To unregister the service from Deepfactor observability, dfctl service deregister servicename can be used. This results in the service definition file (typically stored in /usr/lib/systemd/system) being changed; dfctl will add (or remove) the appropriate LD_PRELOAD setting in that file.
Note that with dfctl service register / deregister, you must restart the service before the LD_PRELOAD service changes will be honored.
For example, if Apache Tomcat was installed on the system as service “tomcat”, the following command could be used to observe Tomcat using Deepfactor:
dfctl service register -r tomcat
The -r option is used to automatically restart the Tomcat service if it is currently running. If -r is not used, you need to issue the following two commands in order to restart the service using Deepfactor:
systemctl daemon reload
systemctl restart tomcat