Otke: Let the programming language be that tool

## Foreword

Sometimes we found ourselves browsing the vast Internet looking for the tool that will perfectly fit our needs. Sometimes we even do this without having sufficient clarity of our needs. Sometimes, that tool is just in front of us: the programming language and it's standard library.

This is a story of one of those times, and we call it Otke.

## Introduction

At Ektocomms we usually deal with CI pipelines which help us on building, testing an publishing artifacts for a given project. Ranging from well known tasks such as creating docker images, to triggering hardware in the loop tests in a laboratory. This usually involves a few dozens of tasks, arranged in a dependency graph, most of them entailing a long running process, and hereby producing caching worthy results. This is a typical scenario to think about introducing a build system such as Makefile, Bazel, Luigi, Taskfile.dev, to mention a few of them.

These are all great tools. But adopting them face us with big source code repositories and extensive documentation pages, usually explained because they implement a very rich feature set tackling a set of problems which easily exceeds our needs. In occasions we also find some friction between the proposed APIs of these tools and our expectation, rather because the tool focus on a generalized solution to build systems, so it needs to take extra cares, or because it is targeted for a niche that differs from our own.

At this point we may consider coding the tool we need. At Ektocomms we did the try, and it was indeed worthwhile.

## What we needed?

These are pretty much all the requirements we had:

• Dependency declaration between tasks.
• Minimal execution, in the sense of not executing the same task twice.
• Parallel execution of independent tasks.
• Being able to run the same pipeline in different environments:
  • The CI GitLab Runner, running in server at our lab.
  • Our own laptops.
  • The lab's server, via ssh.
• Ease of debugging and log inspection.
  • Running a single task in the foreground.
• Tagging tasks and running only those tasks matching a set of tags.

That was all, pretty much... Well, perhaps most importantly, we had the underlying desire for otke to be lightweight and easy to adapt in each of our projects, that guided some decisions that will be presented in the "Otke philosophy" section. Stay tuned.

## What we didn't need? (and do not yet)

• Distributed computing: running tasks on different machines.
• Cloud caching: taking advantage of the results from a pipeline run by a teammate.
• Cross platform support.

And a lot of other features commonly found on build systems.

## What we did then?

We wrote Otke a simple ~200 lines of code python script with no dependencies other than the python standard library which perfectly fits our needs.

Otke has three main components:

• The class Task defining the API that each task needs to implement. Being .do() the most important method, where the actual work is done.
• The function plan_batches() which takes the list of tasks (with their dependencies declarations), and return a list of task batches to be executed in order. Intuitively all the tasks from one batch can be run in parallel.
• The function run_batches() which actually run the tasks using a ThreadPoolExecutor.

This is how we use it:

usage: otke.py [-h] [-a] [-t [TAGS ...]] [--no-deps] [--log-foreground] [-d] [-j JOBS] [--shallow-run] [tasks ...]

positional arguments:
  tasks

options:
  -h, --help            show this help message and exit
  -a, --all
  -t [TAGS ...], --tags [TAGS ...]
  --no-deps             do not resolve dependencies
  --log-foreground      do log to stdout/stderr instead of log files
  -d, --detached        run in detached mode
  -j JOBS, --jobs JOBS  The maximum number of threads that can be used to execute tasks
  --shallow-run         shortcut for '--no-deps --log-foreground -j 1'

We redirect each task stdout to log files in some ./logs directory and then use some log navigation tool such as the great lnav to inspect them with with lnav ./logs. It also works for inspecting a remote directory over ssh.

You can go now and checkout otke by your own here. It won't take you more than few minutes to grasp it, then you can come back an continue reading.

## Otke philosophy

The Dependency Hell is not only about dealing with dependency conflicts, it's also about understanding your craft or not. With fewer dependencies you will likely make more sense of your software.

Package managers have made it easy for us to install third party packages to solve each of our needs, we just pip install some-lib and voilà. That's great, because it let us build on top of other's work, but our claim is that we may be abusing of it. Often, lightly adding thousands of lines of code to our source, which we rarely read. We may be also getting used to the idea that this is the "widely accepted" way to solve almost every problem.

But there is a trade-off between pip installing a possible solution and coding the solution your self. And, as it goes for every trade-off, the answer is not always one neither the other.

There must be room for building your own tools and it does not necessary mean starting from scratch. You can borrow code from others, understand it, adopt it and extend it to cover your specific needs. This is the way we want to distribute Otke. Closer to a code snippet than a software package. ¡Go on and copy paste what you need!

## A story you said...

We did, yes. We actually want to share you the whole story of how Otke came to be.

So, we found ourselves abusing of gitlab-ci.yaml at some projects. Having building and testing logic coupled with the GitLab CI platform. We certainly knew this was suboptimal, but that remained as such for some time as we dealt with deadlines.

In the meantime, we did try Make and some others build systems in order to decouple the pipeline from Gitlab CI and being able to run it on local machines. Some of us did search on it's own for the tool that will help us on assembling pipelines covering our needs, and being easy to adopt by the rest of the team.

At some point, one of us took courage and shared his frustration of not finding the right tool. This opened the door for the proposal to make our own tool. In that talk, we dissipated the fear that we were reinventing the wheel, triggered a process of collective awareness of the trade-off between adopting an existing solution or creating one of our own, and encouraged us to try out.

A first iteration of fifty lines of code gave birth to Otke, then deadlines kept coming and its development entered stand by for a while until we could come back to it. Few months later we proceeded by migrating the pipeline of one of our projects to Otke, adding all the features we couldn't afford to loose from our former pipeline executor, and thus making real use of it.

By the end of the day we did accomplish our objective. It took time and human effort, yes. We had to write the code, test it, find and fix bugs, name things, discuss which features were needed and prioritize them. But all this did not only gave us Otke, it also let us learn new stuff on the way, improved our internal communication, and led us to walk along a path where we were building, not guessing, feeling joy, not being overwhelmed.

My writing, ideas from many of us, thanks for reading,

Ger F.