Contents
Introduction
Believe it or not but there are couple of reoccurring issues which seem to pop up right again right after they have been solved. Software packaging and package management in general is one of those.
This is tightly connected to house keeping and keeping a streamlined development and deployment process.
CD/CI (Continuous Development/Continuous Integration) certainly introduced a couple of very useful tools to automate compiling software and make build reproducible. Well, that was always the case when it comes to any software repository for Linux distributions or any of the BSDs. However, other software distributed, especially a lot of the crap distributed via github, does not comply with this and it introduces some trouble.
Package Management Rant
To make this clear up front. I’m not talking about software quality,which is a whole different topic, but just about the fucking packaging and integration into a larger ecosystem. These days “modern” package management often looks like late 90s “Windows shareware adventures”. It is truly wild west again. The dependency hell is expanding and depending what is used it is close to impossible to keep track of files installed and their dependencies. Having the same shared object in 10 different hard-coded versions can cause serious problems, not just for linking. Believe it or not but disk space is still a rare resource these days and to some degree so is memory (RAM). This is particularly true for edge deployments and basically any inference setup and applies to some training setups as well. Yes, people tend to forget that cloud servers/services cost money and not just a little bit and so does network traffic.
When working with deep-learning related software, there are two basic scenarios we come across. Either people are using Ubuntu, some nvidia docker image, or they are using some Linux distribution with conda environments. It doesn’t matter which option we chooses we still depend on other doing their jobs properly. If we are depending on latest software version, e.g. because a new generation of GPUs or other AI accelerators was released, then we are somewhat doomed to either wait for quite a long time or deal with it ourselves.
Ubuntu
Not that other Linux distributions don’t have any issues with keeping their packages up to date but there is a difference. Ubuntu is backed by a company and basically only LTS (long term support) versions are usable for deployment. Everything outside the Main repository is not maintained by Canonical, which means that anything from other repos (Universe, Restricted, Multiverse) is playing Russian roulette with respect to receiving updates. It is not always clear what package is supported how long and how well. In combination with some Ubuntu specific customizations and these awful automatic updates (default setting) may cause serious problems for maintaining production systems.
Sooner or later there is no way around building and maintaining custom packages with proper dependency management. This is a lot easier for e.g. software programmed in C++ than Python libraries with large dependencies.
Conda environments
When shit hits the fan, or just to minimize dependency collision of Python libraries, virtual environments such as venv or conda environments. This is a common practice to test various models released on github and python projects in general. Even single environments might take quite a long time to setup using conda. There is mamba but except for resolving dependencies faster it may introduce some additional problems as it seems to resolve a slightly different package version. Given how many {valid, invalid} package versions and combinations are required to be evaluated indicates what a mess software engineering is nowadays. It makes it really tough to define dependencies in custom packages and not break the dependency resolver.
Usually shit hits the fan when combining e.g. two repos. Think of e.g. a certain neural network for object tracking and another one for object detection. It could be any other combination, e.g. specific combinations of features of two or more packages with incompatible dependencies. In many cases this leaves us with three options:
- (manipulated dependency files or try to brute-force override conda using pip)
- rebuild all dependencies necessary (either compiling or just change the dependencies files before packaging)
- rewrite portions of the software
- run in different environments and deploy them with some form of inter-process communication (not recommended for production systems unless designed properly)
Given a complex enough piece of software, which can’t be modularized so it could run using some form of inter-process communication, a proper rewrite with proper dependency handling and packaging is inevitable and simply less work.
Docker Absurdities
What about Docker or other containers such as LXC and LXD? They’re supposed to solve all the world’s problems, right? If Ubuntu package management and conda environments are not enough, than there are containers, right?
Well, no. Not really. Reasons to use any kind of container or virtualization solution in general should be different than package/dependency management. While it is true different versions might be deployed using some container image, using container itself does not solve any of the issues mentioned above. However, if done incorrectly, then containers add to the problem instead of helping of helping to structure deployments better and reduce complexity.
Why do we use containers? They allow you to package a software application, libraries, and other runtime dependencies into a single image. This way, the application environment is both portable and consistent, and agnostic to the underlying host system software configuration. Containers also eliminate the need to install complex software environments and allow you to build applications on the system without any assistance from system administrators. Finally, containers are typically easier for users which can help speed up the adoption of new capabilities. However, a container itself provides nothing else than as a clean virtual image of an operating system such as Ubuntu. All the package management and installation problems remain the same. Just have a look the underlying docker container used at Kaggle:
These containers provide a base system only. If we have competing dependency requirements it solves nothing. Copy & pasting binaries or source code in them without packaging them also does not make things easier - it is just bad style and adds complexity with respect to handling e.g. docker files reliably.
Sane(r) Approach
In general, economic entities and hobbyists are short of a) time and b) money. Package management was solved a long time ago. None of this “modern nonsense” is necessary. Classical package management systems as known from various Linux distributions or the BSDs works. There is no need to fiddle around with other solutions or fall back into some weird copy&past schema.
When releasing software for production, packaging it properly with well-defined dependencies is the best way to do it. Perhaps there are a couple of dependencies which need to be re-packaged in-house and distributed using local repos but that is a price worth paying to cut down CD/CI costs and keep any operation running. It might be worth to even fork some versions or at least to develop in-house code in a way such as the number of dependencies are minimized. And it doesn’t matter if a library is packaged for system-wide deployment or for e.g. a conda environment. The basic principles remain the same. It helps to avoid “multiple versions of the same shared object” and stream-lines dependencies. A nice side effect is that this allows for cleaner containerized deployments as well.