Entrypoint libc does not match OS distribution libc

Deepfactor Support

August 12, 2021 15:24

Deepfactor 2.1 or greater can automatically inspect the initial entrypoint, an executable or script-interpreter, and determine the libc of a dynamically linked program for compatibility.

Support Status in Deepfactor 2.1

dfctl run ... --docker-run :

automatic entrypoint libc detection

Deepfactor Dockerfile.df build:

automatic entrypoint libc detection by adding --build-arg "DF_IMAGE_ENTRYPOINT=

--build-arg "DF_IMAGE_ENTRYPOINT=`/opt/deepfactor/bin/df-get-entrypoint.sh $APP_IMAGE`"

Deepfactor Kubernetes mutating webhook:

entrypoint path or libc can be specified with an annotation in the pod deployment. The path may be absolute or relative.

df.k8-app.entrypoint.path: "java"

df.k8-app.entrypoint.libc: "glibc"

automatic entrypoint libc detection for image repositories is 'Tech Preview' in Deepfactor 2.2 Enable this preview feature by commenting in the following in your Deepfactor Kubernetes Admission Webhook in it's override.yaml. See 3. Install Deepfactor Mutating Admission Webhook

imageanalyzer:
  enabled: true

The content below in this article is no longer applicable since Deepfactor 2.1. The issues identified below can be resolved by upgrading to Deepfactor 2.1.0 or greater.

Deepfactor dfctl, Dockerfile templates, and K8s mutating webhook scripts do not inspect the initial target executable. This limitation does not impact df runtime if it is already loaded and --multilibc auto is enabled at dfctl/dockerffile/k8s run. See Hybrid libc environments for more information. Once Deepfactor runtime is loaded with multilibc support enabled, it will inspect an --exec*() target executable and automatically select the appropriate libc compatible Deepfactor runtime for the target executable/script. All three inspect the target OS distribution as follows.

dfctl run --docker-run test case

- alpine with java-glibc entrypoint

$ dfctl run -v -a alpine-java -c dfctl-docker-alpine-java-entry \ 2 
  --multilibc auto \ 
  --docker-run -it --rm --image adoptopenjdk/openjdk11:alpine-slim-libstdcpp java -version 
dfctl version: "1.8-867" "59576458f4ef686a3b5a42ced6b2edf8f9353a59" 
Detected libdf=musl, manifest libdf=auto 
Application version not specified. Using binary's md5sum as version 
Launching application with LD_PRELOAD=/opt/deepfactor/musl/libdf.so, manifest libc=auto 
Launching application /usr/bin/docker with args docker run -it -v /home/dfadmin/Sleep.class:/Sleep.class -v df-runtime:/opt/deepfactor/ -v /dev/shm/8bef038e-33ce-47d6-8998-a56ba67fa0fb-2.json:/opt/deepfactor/manifest/8bef038e-33ce-47d6-8998-a56ba67fa0fb-2.json -e DF_MANIFEST=/opt/deepfactor/manifest/8bef038e-33ce-47d6-8998-a56ba67fa0fb-2.json -e LD_PRELOAD=/opt/deepfactor/musl/libdf.so -e DF_INSTID=1b537475-62fb-4ca9-8240-bfccfceefbf0 adoptopenjdk/openjdk11:alpine-slim-libstdcpp-fix java -version 
java: libc dlopen: /usr/glibc-compat/lib/libc.so: invalid ELF header

Resolution:

- Add the --entrypoint-libc “glibc” or “musl” option in the dfctl run command [1]
OR
- Append sh -c '…' to container entrypoint or command when run [2]

[1] Add --entrypoint-libc “glibc” to the dfctl run command

$ docker build -t adoptopenjdk/openjdk11:alpine-slim-libstdcpp - << DKREOD
> FROM adoptopenjdk/openjdk11:alpine-slim
> RUN apk add libstdc++ 
> DKREOD ... 

$ ~/dfctl run -v -a alpine-java-glibc -c dfctl-docker-alpine-java-entry \ 
   --multilibc auto --entrypoint-libc glibc \ 
   --docker-run -it --rm --image adoptopenjdk/openjdk11:alpine-slim-libstdcpp java -version 
dfctl version: "1.8.1-DEEP-3748" "3ddf1500ada86b54102a7287ccbacc50a033d866-M" 
Application version not specified. Using binary's md5sum as version 
Launching application with LD_PRELOAD=/opt/deepfactor/glibc/libdf.so, manifest libc=auto 
Launching application /usr/bin/docker with args docker run -it -v /home/dfadmin/Sleep.class:/Sleep.class -v df-runtime:/opt/deepfactor/ -v /dev/shm/a55845a7-0b03-4525-a010-6ae1ce66fb66-1.json:/opt/deepfactor/manifest/a55845a7-0b03-4525-a010-6ae1ce66fb66-1.json -e DF_MANIFEST=/opt/deepfactor/manifest/a55845a7-0b03-4525-a010-6ae1ce66fb66-1.json -e LD_PRELOAD=/opt/deepfactor/glibc/libdf.so -e DF_INSTID=3135e02e-9c5b-419e-8531-86b2f3a926e1 adoptopenjdk/openjdk11:alpine-slim-libstdcpp-fix java -version 
openjdk version "11.0.11" 2021-04-20 
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9) 
OpenJDK 64-Bit Server VM AdoptOpenJDK-11.0.11+9 (build 11.0.11+9, mixed mode)

[2] e.g. Wrap the container entrypoint or command with shell: sh -c

$ dfctl run -v -a alpine-java -c dfctl-docker-alpine-sh-entry 
  --multilibc auto \
  --docker-run -it --rm --image adoptopenjdk/openjdk11:alpine-slim-libstdcpp sh -c 'java -version' 
dfctl version: "1.8-867" "59576458f4ef686a3b5a42ced6b2edf8f9353a59" 
Detected libdf=musl, manifest libdf=auto 
Application version not specified. Using binary's md5sum as version 
Launching application with LD_PRELOAD=/opt/deepfactor/musl/libdf.so, manifest libc=auto 
Launching application /usr/bin/docker with args docker run -it -v /home/dfadmin/Sleep.class:/Sleep.class --rm -v df-runtime:/opt/deepfactor/ -v /dev/shm/8bef038e-33ce-47d6-8998-a56ba67fa0fb-2.json:/opt/deepfactor/manifest/8bef038e-33ce-47d6-8998-a56ba67fa0fb-2.json -e DF_MANIFEST=/opt/deepfactor/manifest/8bef038e-33ce-47d6-8998-a56ba67fa0fb-2.json -e LD_PRELOAD=/opt/deepfactor/musl/libdf.so -e DF_INSTID=dc33b381-c2ec-4ac1-ae57-ffe7008289e5 adoptopenjdk/openjdk11:alpine-slim-libstdcpp-fix sh -c java -version 
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9) 
OpenJDK 64-Bit Server VM AdoptOpenJDK-11.0.11+9 (build 11.0.11+9, mixed mode)

Dockerfile build deployment example test app:

See footnote [4] for APP_IMAGE build.

$ export APP_IMAGE=adoptopenjdk11/alpine-slim:glibc-entry; 
$ export DF_COMPONENT="DfDemoSpringBoot"; 
$ docker build -t ${APP_IMAGE}-df -f Dockerfile.alpine.hybrid.1.8.1.df 
 --build-arg "DF_RUN_TOKEN=${DF_RUN_TOKEN}" 
 --build-arg "APP_IMAGE=${APP_IMAGE}" 
 --build-arg "DF_APP_NAME=${APP_IMAGE}" 
--build-arg "DF_COMPONENT=${DF_COMPONENT}-glibc-entry" 
--no-cache .
... 
Successfully tagged adoptopenjdk11/alpine-slim:glibc-entry-df

$ docker run -it --rm adoptopenjdk11/alpine-slim:glibc-entry-df 
java: libc dlopen: /usr/glibc-compat/lib/libc.so: invalid ELF header

Resolution
- append sh -c '…' to container entrypoint or command when run
OR
- manually configure the build Dockerfile.df and replace ${libc} with: musl or glibc
ENV LD_PRELOAD=/opt/deepfactor/${libc}/libdf.so

set optional query parameter entrypoint-libc=to the either “musl” or “glibc” to match the image's entrypoint when generating a Dockerfile.df e.g. [3]

[3] Example download of dockerfile.df with entrypoint-libc=glibc
- Dockerfile.alpine.hybrid.glibc-entry.1.8.1.df

$ df_instr_url='https://my-portal/api/services/v1/instrumentation/dockerfiles?'; 

$ curl -k -H "Authorization: Bearer $(cat ./my-portal.admin.api.token)" 
  ${df_instr_url}'app_name=$\{DF_APP_NAME\}&component_name=$\{DF_COMPONENT\}&alpine=true&multilibc=auto'
  | jq -r .data.file_data > Dockerfile.alpine.hybrid.1.8.1.df 
....
$ curl -k -H "Authorization: Bearer $(cat ./my-portal.admin.api.token)" 
   ${df_instr_url}'app_name=$\{DF_APP_NAME\}&component_name=$\{DF_COMPONENT\}&alpine=true&multilibc=auto'
'&entrypoint-libc=glibc'\
  | jq -r .data.file_data > Dockerfile.alpine.hybrid.glibc-entry.1.8.1.df 
... 
$ diff Dockerfile.alpine.hybrid.1.8.1.df Dockerfile.alpine.hybrid.glibc-entry.1.8.1.df 
56c56 
< ENV LD_PRELOAD=/opt/deepfactor/musl/libdf.so 
--- 
> ENV LD_PRELOAD=/opt/deepfactor/glibc/libdf.so

[4] APP_IMAGE adoptopenjdk11/alpine-slim:glibc-entry build

$ docker build -t adoptopenjdk11/alpine-slim:glibc-entry - << DKREOD 
FROM adoptopenjdk/openjdk11:alpine-slim

# libstdc++ required df runtime 
RUN apk add libstdc++ 

COPY ./DfDemo-0.0.1-SNAPSHOT.jar / 

# the alt. entrypoint commented out 'just works' because there will be an OS shell exec 
# ENTRYPOINT java -jar /DfDemo-0.0.1-SNAPSHOT.jar 

ENTRYPOINT [ "java", "-jar", "/DfDemo-0.0.1-SNAPSHOT.jar" ] 

DKREOD

K8s deployment test example:

$ kubectl apply -f df-adoptopenjdk-glibc-entry.yml 
pod/demo-hybrid-java-libc-app 
$ kubectl logs demo-hybrid-java-libc-app 
java: libc dlopen: /usr/glibc-compat/lib/libc.so: invalid ELF header

Example deployment image:

docker build -t my-registry/adoptopenjdk/openjdk11:alpine-slim-DfDemo-glibc-entry 
- << DKREOD 
FROM adoptopenjdk/openjdk11:alpine-slim 

# required df runtime dep 
RUN apk add libstdc++ 

COPY ./DfDemo-0.0.1-SNAPSHOT.jar / 

ENTRYPOINT [ "java", "-jar", "/DfDemo-0.0.1-SNAPSHOT.jar" ] 
DKREOD

Resolution:
- prefix sh -c '…' to deployment command [5]
OR
- set optional df.k8-app.entrypoint.libc
annotation in pod deployment to match the container entrypoint or command [6]

[5] alter deployment command

$ diff df-adoptopenjdk-glibc-entrypoint.yml df-adoptopenjdk-musl-entrypoint.yml 
14a15 
>   command: [ "sh", "-c", "java -jar /DfDemo-0.0.1-SNAPSHOT.jar" ]

[6] set optional df.k8-app.entrypoint.libc example


$ diff df-adoptopenjdk-glibc-entrypoint.yml df-adoptopenjdk-glibc-entrypoint-df-annotation.yml 
8a9 
>   df.k8-app.entrypoint.libc: "glibc"