In certain circumstances, it may not be possible for Deepfactor to instrument (observe) a process. This results in a lack of event telemetry being delivered to the Deepfactor management portal, and consequently, lack of alerts or insights being shown.
In many cases, the Deepfactor runtime library can determine ahead of time if it will not be able to observe a new process being launched and communicate this fact to the management portal or via messages printed to the console. This gives the Deepfactor user the knowledge that some processes may not have been observed.
This document lists the various warning messages that are shown in the management portal or console when these situations occur, and possible remediations.
"Setuid bit set. You must run dfctl as root in this scenario."
"Setgid bit set. You must run dfctl as root in this scenario."
These messages are displayed when the command or container entrypoint being run by dfctl has either the setuid or setgid bit set. As Deepfactor uses LD_PRELOAD to insert its instrumentation library, this approach is incompatible with executables that have either the setuid or setgid bit set. In these scenarios, you can run dfctl as root to work around the operating system check that rejects LD_PRELOAD for setuid/setgid executables, or you can remove the setuid/setgid bit if not needed.
"OS distro version not supported"
This message is displayed when an unsupported distribution/version is detected. Deepfactor supports the Linux distributions listed in the following KB article:
"Entrypoint does not match libc environment"
This message is displayed when Deepfactor detects that the entrypoint executable defined in a container manifest does not match the libc style used by the container (glibc or musl). This check is performed against the "standard" libc shipped with a particular distribution (musl for Alpine containers and glibc otherwise). Deepfactor can operate in hybrid libc environments (for example, when a glibc based executable is installed in an Alpine container along with glibc itself); this warning message is displayed to ensure that the developer understands that such a configuration is being used.
Note: dfctl run (in non-container/Kubernetes deployments) does not support hybrid libc.
This message is displayed when the user has chosen to override libc detection by supplying the command line parameter --entrypoint-libc, but has chosen a libc variety that does not match the actual container entrypoint's requirements. To remedy this, the user should verify that the entrypoint executable's required libc (typically musl in Alpine containers and glibc otherwise) matches the supplied command line parameter or omit the --entrypoint-libc parameter and let Deepfactor automatically detect the required libc variety required.
"The libc is unknown for the provided entrypoint/command"
This message is displayed when Deepfactor does not recognize the type of libc library used by the command or container entrypoint. Deepfactor supports musl and glibc varieties of libc. The user can remedy this by verifying that the entrypoint is linked using musl or glibc. To do this, the user can use the system "ldd" command, or use /opt/deepfactor/bin/df-get-libc.sh. The Deepfactor df-get-libc.sh script uses ldd internally and will report the libc of an executable script/interpreter if applicable.
"Java agent is not loaded"
"DF Java Agent disabled in presence of another agent"
"DF JVMTI Agent disabled in presence of another agent"
These messages are displayed when the Deepfactor Java LSA (language specific agent) is not loaded when a Java application is being observed. This can happen for several reasons:
- The user chose to not load the agent via the --disable-stack-traces not specifying --enable-method-tracing flags
- Deepfactor detected a conflicting Java agent from another product being used in the environment
The first case is intended to remind the user that stack traces and usage information will not be available for the application. The second case is intended to inform the user that they will need to disable the alternate Java agent from the other product in order to make use of Deepfactor's stack tracing and method tracing capability.
"Please remove libdf.so from your pod container env and allow for mutating webhook to inject."
"libdf.so detected in pod container env LD_PRELOAD along with additional preload libraries."
These messages are displayed when Deepfactor detected an attempt to instrument/observe an application whose container image was already built with Deepfactor. To remedy this, the user can use a container image that does not already have Deepfactor loaded. An alternate remedy is to ensure that the container manifest does not specify an LD_PRELOAD value with Deepfactor's libdf.so.
"Unable to fetch image manifest"
This message is displayed when the mutating webhook cannot examine a container image manifest to determine entrypoint and libc characteristics or ImageID. This may be due to permission or communication issues when communicating with the container registry. To remedy this, the user can review the error description provided with the message. Common error messages and their remedies can be searched for in Deepfactor's KB articles.
"package manager not found or supported"
This message is displayed during execution of a containerized application in situations when Deepfactor cannot determine which OS package manager (dpkg, apk, rpm, etc) should be used to query OS package information. OS package information is used to provide dependency analysis and CVE information for packages used by the application. This message may indicate that a distroless container is being used, or that the user has removed the package manager from their container image. To remedy this, ensure that the standard package manager binary is functional within the container (dpkg for Ubuntu and Debian based containers, rpm for CentOS and RedHat based containers, and apk for Alpine based containers).
"sh test failed"
This message is displayed during execution of a containerized application in situations when Deepfactor detects that a suitable "sh" shell is not available, and that a suitable package manager was not detected (see previous message). Deepfactor tests OS distribution libc version compatibility with "sh" in scenarios where a package manager cannot be located or used. To remedy this, either:
- Ensure that the standard package manager binary is functional within the container (dpkg for Ubuntu and Debian based containers, rpm for CentOS and RedHat based containers, and apk for Alpine based containers).
- Or, if a package manager cannot be used, ensure that "/bin/sh" exists in the shell and is a standard Linux shell executable (bash, etc).