Embedded Developer

I use Docker fairly often for a number of projects. I find it useful for getting access to tools that are not available on my host system (Mac running on Apple silicon) and also for duplicating the environments I use for CI on GitHub. I also use it to access some legacy tools that I can no longer install locally on my system but where the vendor has provided a Docker image containing those tools.

Docker Image

The docker containers I use most often are the espressif/idf containers, specifically the espressif/idf:release-v4.2 container. This is used to access the development tools for the ESP-IDF release 4.2 tools and libraries to support maintenance of legacy code whilst it is being ported to a newer version of the SDK.

Access to the tools is through the docker command:

docker run --platform linux/amd64 --rm -it -v $PWD:/project -w /project espressif/idf:release-v4.2

Running this command starts the container and sets the PATH etc allowing interactive access to the tools.

Detecting the Python interpreter
Checking "python" ...
Python 3.6.9
"python" has been detected
Adding ESP-IDF tools to PATH...
Using Python interpreter in /opt/esp/python_env/idf4.2_py3.6_env/bin/python
Checking if Python packages are up to date...
Python requirements from /opt/esp/idf/requirements.txt are satisfied.
Added the following directories to PATH:
  /opt/esp/idf/components/esptool_py/esptool
  /opt/esp/idf/components/espcoredump
  /opt/esp/idf/components/partition_table
  /opt/esp/idf/components/app_update
  /opt/esp/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin
  /opt/esp/tools/xtensa-esp32s2-elf/esp-2020r3-8.4.0/xtensa-esp32s2-elf/bin
  /opt/esp/tools/esp32ulp-elf/2.28.51-esp-20191205/esp32ulp-elf-binutils/bin
  /opt/esp/tools/esp32s2ulp-elf/2.28.51-esp-20191205/esp32s2ulp-elf-binutils/bin
  /opt/esp/tools/cmake/3.16.4/bin
  /opt/esp/tools/openocd-esp32/v0.11.0-esp32-20220706/openocd-esp32/bin
  /opt/esp/python_env/idf4.2_py3.6_env/bin
  /opt/esp/idf/tools
Done! You can now compile ESP-IDF projects.
Go to the project directory and run:

  idf.py build

root@a76f973ca31c:/project#

It is now possible to use all of the usual Espressif tools from the command prompt with the usual caveats for USB port access on Mac systems. Still, development remains possible even if deployment requires some additional steps.

Workflow

For a while now the working workflow has been as follows:

• Edit the code, build scripts in VS Code on Mac
• Build the code in the docker container running in a terminal session
• Deploy the code from a second terminal session where the latest libraries are installed and configured

This worked flawlessly for several months.

Until the last few days.

The Problem

A recent requirement change necessitated the modification of the source code for the application built using the workflow described above. All seemed to start well, the code was edited, the docker container started and the code hit the first compilation of the day for a syntax check.

This was closely followed by come code changes and a second compilation. All seemed well and the code was committed to source control and rebuilt. At this point I noticed something odd, the automatically generated build number did not increase. The system used for this repository changes the build number based upon the number of commits. The first two compilations would not have increased the build number as there was no commit. The build after the commit would normally generate an increment in the build number and it clearly was not doing so.

Let’s introduce a syntax error, deleting a semicolon should do it, and try rebuilding. The code compiled with no errors. How strange.

Investigating further using more to check the contents of the files on the host machine against the docker container revealed that the changes on the host system were not being reflected in the docker image.

Docker File System Access

Something odd is happening to the file system. Changes on the host are clearly not being reflected in the mounted volume in the docker container. Time to try a few things with the syntax error still in place:

• Exit the docker container and restart it and build the code – no change, the code still compiles
• Exit the docker container and run it none interactively – still no change
• Change the mount method from a volume to a mount option – the code still compiles
• Delete the docker image and restart (this will rebuild from scratch) – compilation gives a syntax error

So finally, the code change is reflected in the docker volume. Now we need to remove the syntax error by reverting the file change and we can recompile and move on. Doing this resulted in a compilation failure, the change had once again not been applied to the mounted volume.

At this point a colleague checked on their system that they could change files and see the changes reflected in the mounted volume and yes they could. Time for a comparison of the system settings and the obvious one to pick up is the File System settings. A quick check showed a difference between the two systems. Changing my system to match theirs and restarting everything resolved the issue.

The difference was in the File sharing implementation for your containers.

File Sharing Selection

My local system had this configured for gRPC FUSE. Changing to the above setting, VirtioFS, and restarting Docker Desktop and the docker container seems to have fixed the issue.

Conclusion

I still do not know why the change to the way the file system was accessed changed or why the system stopped reflecting changes to the files in the mounted volume. I don’t think I will ever find out but maybe the note will help others (maybe even myself in the future).