A Rare libmagic Bug in Docker

We recently caught a rare bug:

One of our services is used to upload files to our storage service, and after uploading, it would check file format, and pass the extracted type to the next service. This file type inspection relies on python-magic. The bug occurs as it detected a docx file as zip file.

Attempts

Receiving this bug report, we tried the following steps to locate the bug:

1. check code on git
2. there is no recent code changes on git;
3. the master branch has been synced with the preprod branch;
4. we built the Docker image from the latest master branch;
5. check pip dependency
6. the docker image on the server has the pip dependencies with same version as we stated in the requirements as in the git repo;
7. check local and server environment
8. we rebuilt image from the same branch in the local environment;
9. we couldn't reproduce the bug from the local environment;
10. check docker image
11. we stoped the docker image on the server, and rebuilt the image from scratch;
12. the bug still existed on the server environment.

The same docx files had no issues when we uploaded from local test environment, but cannot pass from the server environment. However, not all the docx files had the same issues, as the majority had no issues to be uploaded.

At this point, everything looked very confusing:

1. same code,
2. same Dockerfile,
3. same pip dependencies
4. same input, but different results from different environment.

Luckily, at least we could exclude the problem from the above levels.

One of the team members suspected that there might be some compression when we uploaded the file to the server environment. However, if this is the case, why did not all docx files have the same issue?

Solution

We then checked gitlab repo of python-magic, and found a relevant issue:

This is dependent on the version of libmagic you are using and/or the magic definition file. e.g, with debian bullseye (libmagic 5.38) I get this:

This issue has exactly opposite issue as us: their issue occurred on MacOS, while ours occurred in Centos, but not on MacOS. Anyway, this gave us some clues, so we dug in to check the libmagic version in two Docker image with this command:

apt list --installed | grep magic

The local Docker image running on MacOS returned the following:

imagemagick-6-common/stable,now 8:6.9.11.60+dfsg-1.3 all [installed,automatic]  
imagemagick-6.q16/stable,now 8:6.9.11.60+dfsg-1.3 amd64 [installed,automatic]  
imagemagick/stable,now 8:6.9.11.60+dfsg-1.3 amd64 [installed]  
libmagic-mgc/stable,now 1:5.39-3 amd64 [installed,automatic]  
libmagic1/stable,now 1:5.39-3 amd64 [installed]  
libmagickcore-6-arch-config/stable,now 8:6.9.11.60+dfsg-1.3 amd64 [installed,automatic]  
libmagickcore-6-headers/stable,now 8:6.9.11.60+dfsg-1.3 all [installed,automatic]  
libmagickcore-6.q16-6-extra/stable,now 8:6.9.11.60+dfsg-1.3 amd64 [installed,automatic]  
libmagickcore-6.q16-6/stable,now 8:6.9.11.60+dfsg-1.3 amd64 [installed,automatic]  
libmagickcore-6.q16-dev/stable,now 8:6.9.11.60+dfsg-1.3 amd64 [installed,automatic]  
libmagickcore-dev/stable,now 8:6.9.11.60+dfsg-1.3 all [installed]  
libmagickwand-6-headers/stable,now 8:6.9.11.60+dfsg-1.3 all [installed,automatic]  
libmagickwand-6.q16-6/stable,now 8:6.9.11.60+dfsg-1.3 amd64 [installed,automatic]  
libmagickwand-6.q16-dev/stable,now 8:6.9.11.60+dfsg-1.3 amd64 [installed,automatic]  
libmagickwand-dev/stable,now 8:6.9.11.60+dfsg-1.3 all [installed]

While the server Docker image running on CentOS returned the following:

imagemagick-6-common/oldstable,oldstable,now 8:6.9.10.23+dfsg-2.1+deb10u1 all [installed,automatic]  
imagemagick-6.q16/oldstable,oldstable,now 8:6.9.10.23+dfsg-2.1+deb10u1 amd64 [installed,automatic]  
imagemagick/oldstable,oldstable,now 8:6.9.10.23+dfsg-2.1+deb10u1 amd64 [installed]  
libmagic-mgc/oldstable,now 1:5.35-4+deb10u2 amd64 [installed,automatic]  
libmagic1/oldstable,now 1:5.35-4+deb10u2 amd64 [installed]  
libmagickcore-6-arch-config/oldstable,oldstable,now 8:6.9.10.23+dfsg-2.1+deb10u1 amd64 [installed,automatic]  
libmagickcore-6-headers/oldstable,oldstable,now 8:6.9.10.23+dfsg-2.1+deb10u1 all [installed,automatic]  
libmagickcore-6.q16-6-extra/oldstable,oldstable,now 8:6.9.10.23+dfsg-2.1+deb10u1 amd64 [installed,automatic]  
libmagickcore-6.q16-6/oldstable,oldstable,now 8:6.9.10.23+dfsg-2.1+deb10u1 amd64 [installed,automatic]  
libmagickcore-6.q16-dev/oldstable,oldstable,now 8:6.9.10.23+dfsg-2.1+deb10u1 amd64 [installed,automatic]  
libmagickcore-dev/oldstable,oldstable,now 8:6.9.10.23+dfsg-2.1+deb10u1 all [installed]  
libmagickwand-6-headers/oldstable,oldstable,now 8:6.9.10.23+dfsg-2.1+deb10u1 all [installed,automatic]  
libmagickwand-6.q16-6/oldstable,oldstable,now 8:6.9.10.23+dfsg-2.1+deb10u1 amd64 [installed,automatic]  
libmagickwand-6.q16-dev/oldstable,oldstable,now 8:6.9.10.23+dfsg-2.1+deb10u1 amd64 [installed,automatic]  
libmagickwand-dev/oldstable,oldstable,now 8:6.9.10.23+dfsg-2.1+deb10u1 all [installed]

The versions of the  libmagic  do not match! The server side uses an older version of  libmagic. Although we specified to update  libmagic  in the DockerFile, the built Docker images still used different versions of  libmagic . This is the only obvious difference on these two environments.

We then updated the base Docker image from Python 3.8 to Python 3.9, and both images on server and local environment installed same version of  libmagic  as 8:6.9.11, and the bug was gone.

Takeaway

1. Isolating variables in debugging is important. This can help us to exclude the irrelevant factors;
2. We followed the certain steps to check our code, git, Docker image, although we didn't locate the bug in these stages, it helped us to exclude the bug from this stages;
3. Although Docker helped a lot to ensure the consistence of the environment and dependency, it doesn't ensure 100%. In our case, Docker running on CentOS pulling the base image of Python 3.8 used an old version of  libmagic , but Docker running on MacOS pulling the base image of Python 3.8 used the latest version of  libmagic .
4. In the future, apart from checking code, git, Docker image, we will also check the dependencies on the Docker level to make sure everything is consistent.
5. In the future, we would specify the library version in the DockerFile with

apt install package_name=package_version