Deepfactor telemetry events

Types of events

Deepfactor intercepts application behavior and environmental context to provide security insights. To this end, the high-level telemetry categories listed below are sent between the application being observed and the Deepfactor management portal:

• Telemetry sent once for each process
• Telemetry sent once for each environment (container/VM)
• Telemetry sent once per API invocation
• Additional telemetry sent for web applications
• Telemetry sent periodically (heartbeat events)
• Language specific telemetry events

All telemetry sent from an application to the Deepfactor management portal is encrypted using TLS 1.3, via certificates installed by the customer (or using a default self-signed certificate generated during product installation). 

The telemetry data described in the following sections is stored in the Deepfactor management portal temporarily and is used by the management portal to create alerts based on application behavior and package vulnerabilities, subject to the alert policies chosen by the administrator. Deepfactor will send the telemetry described in this document to the management portal regardless of the alert policy currently in place for a given application; the management portal may choose to ignore certain fields if that data is not applicable to a given application. 

Telemetry sent once for each process

Deepfactor sends up to two telemetry events per process launch/termination:

• Process start event
• Process stop/termination event

The process start event contains the following information:

• Host information (system host name, uptime, number and type of CPUs)
• OS/distribution information (OS name/distribution and version)
• CSP information (CSP type (Amazon/Azure/GCP, for example)
• Container runtime (docker or other)
• Process information (memory maps, process ID, process parent ID, UID, GID)
• Kubernetes pod name and namespace
• Environment variables and their values

The process termination event contains the following information:

• Process exit code

Note: a process termination event is not guaranteed to be sent for each process (for example, if the process crashes before Deepfactor can send the telemetry event, no termination event will be recorded)

Telemetry sent once for each environment (container/VM)

Deepfactor creates a “host environment telemetry event” for each new pod/container/VM. This event contains the following:

• List of OS packages installed in the container/VM (package name, version, source)

Deepfactor will also send the package license information for each package as part of this event if license information is available.

Telemetry sent once per API invocation

For events generated as the result of the application invoking an observed API, various telemetry data is sent. The following subsections break down the data that is sent for each scenario.

Note: All API invocation-related telemetry events contain stack trace frame data, which can be augmented with symbols in the Deepfactor management portal to provide contextual information to assist developers in resolving bugs/issues (such as line number and source file name).

Telemetry Related to File Operations

File events contain the following information:

• Operation/API name/type (for example, “open”, “read”, “write”, “unlink”, “truncate”)
• Absolute pathname(s) being operated upon
• Mode/flag bits provided to the api (for open and create operations)

Telemetry Related to Network Operations

Network events contain the following information:

• Event type (“bind”, “connect”, “accept” or “DNS”)
• Address family
• Port and address information (remote and local port and address)
• For DNS, a list of FQDNs being resolved

Telemetry Related to Identity Change Operations

Identity change events contain the following information:

• Type of identity change (for example, “setuid”, “setgid”, “setegid”, “seteuid”, etc)
• Numeric 2-tuple containing the “from” and “to” identities

Telemetry Related to Process Operations

Process related events convey information about new processes being launched and contain the following information:

• Event type (exec, spawn, clone, etc)
• Pathname of the launched executable
• Command line arguments

Telemetry Related to Library (Dependency) Operations

Dependency information events provide information about shared libraries (.so files) and other dependencies (Java .jar/.zip files, for example)  loaded by the process and contain the following:

• Full pathname of the dependency being loaded
• SHA256 sum of the dependency binary
• Contextual information about dependency parent-child relationships (to preserve nested dependency hierarchy)
• Language (eg Java or C, if known)
• Dependency version if known

Telemetry Related to Web Applications

Web application telemetry events are sent in the following cases:

• When Deepfactor detects a new Web application (eg, an application that has opened a listening socket and has received one or more HTTP-style transaction on that socket), the following information is sent
  - L7 protocol (for example “HTTP/1.0”)
  - Address family
  - Remote and local socket and port information
  
  
• When Deepfactor detects a new URI request on a previously observed web application, the following information is sent
  - HTTP method (GET, POST, etc)
  - URI (eg, “/myapp/api/v1/foo?param1=bar”)
  - Local listening port corresponding to a previously discovered web application
  
  
• When Deepfactor detects an authorization/bearer token used to authenticate to a web application, the following information is sent
  - Authorization type (bearer token)
  - The bearer token’s value
  - Local listening port corresponding to a previously discovered web application
  

Telemetry Sent Periodically

Deepfactor sends periodic “heartbeat” telemetry events that contain the following information:

• List of file descriptors that are either currently open or have been opened and subsequently closed since the previous heartbeat event
• For files, the path of the file descriptor is included
• For sockets, any known local and remote port/address information is included
• Process memory and CPU usage
• Memory and CPU used by Deepfactor on behalf of the process
• Number of threads created and fork operations performed since the previous heartbeat event

Other API Related / Language Specific Telemetry Events

On occasion, Deepfactor may send additional telemetry events that do not fit into one of the previous categories. These include:

• Java method usage tracing telemetry, which includes the class name and method name of a Java method whose execution was observed during runtime. This information is sent only when method tracing is enabled while running your application with Deepfactor.

Size of events

The following table provides an approximate size of the raw telemetry data for a given event. Since telemetry events will contain data of various lengths (pathnames, URIs, etc), the following should only be used as a rough estimate as a customer’s specific telemetry events may be smaller or larger than these estimates.

When using these sizes to estimate the approximate amount of network traffic a particular application will generate, make sure to remember that these size estimates do not include network overhead due to TLS handshaking, TCP header overhead, or response messages from the Deepfactor management portal. For a sufficiently long-running application, this overhead should tend toward a small percentage of the overall traffic.

To give an idea as to how many of each of these events may be generated, the following is an example from a simple Java spring boot app launch and 5 URI requests from a client to the app’s served content. The application was then left idle for approximately 30 minutes. The application generated 743 telemetry events during the test period, sending a total of 659081 bytes (659.081 KB) of telemetry data (not including TCP/TLS network overhead).

Event count and size summary:
Component Start Event: count=3, average size=4263, total size=12789
Component Exit Event: count=2, average size=228, total size=456
Host Environment Event: count=3, average size=38060, total size=114180
Package Info Event: count=48, average size=349, total size=16785
API Event: count=8, average size=739, total size=5918
Heartbeat Event: count=248, average size=484, total size=120138
Network Event: count=9, average size=2590, total size=23310
Dependency Info Event: count=103, average size=376, total size=38764
Memory Warning Event: count=197, average size=446, total size=87997
File Event: count=120, average size=1816, total size=218009
Webapp Register Event: count=2, average size=313, total size=627