Solomon Hykes, who started the original Docker project in 2010 and co-founded the Docker company in 2011, has spent the past five years trying to help solve the challenge of pipeline efficiency.

“Everyone we met was telling us that their delivery pipelines were a source of problems,” Hykes, co-founder at Dagger, a DevOps pipeline engine, told Dev Class. When developing an application it is easy to change the code and see the results, but when changing the pipeline “you have this very inefficient process … if there’s a significant change in the tech stack, say an AI feature, then a lot of times everything is ready except the pipeline, it slows people down … it’s all shell scripts, then you have another layer on top which is Jenkins or GitHub Actions or CircleCI, it’s this amalgamation of scripts in YAML tied together with duct tape.”

According to Hykes, “what the DevOps community is yearning for is an actual software platform to develop and run this little factory that is going to ship their app.”

That is the gap which Dagger aims to fill. “Every piece of your pipeline, build, test, deploy, the whole thing should run on containers,” Hykes told us. “We start from there and give you an API that lets you control these containers in a modern way, and then we give you SDKs for all popular languages, and we let each team control these pipelines using this API with the language that they already use to develop their application.” 

The Dagger Engine currently supports Go, Node.js and Python, which is some way from “all popular languages,” but each no doubt a welcome alternative to YAML; and according to Hykes, “the community started adding new SDKs, Java, Rust etc”. Under the covers functions are packaged into a GraphQL API. Integrations are developed using modules, and the project is open source on GitHub, under the Apache 2.0 license.

All that said, Dagger does not replace other CI (Continuous Integration) tools. “Dagger will run typically on the development machine or on top of your existing CI server so it’s additive, it doesn’t replace your CI,” Hykes said. Modernization of the pipeline becomes a case of gradually replacing existing scripts with a new Dagger function. “You remove 200 lines of old incomprehensible scripting magic and you replace it with 20 lines of Python or TypeScript … it’s not a revolution, its an incremental improvement.”

Does that mean though that Dagger becomes one more part of the stack to manage, on top of the existing ones, and risks adding rather than removing complexity?

“Of course. Whenever you add one more tool, you’re paying an extra price of complexity, there’s a tax. It’s up to us to demonstrate, we save you more trouble than you’ll add by taking on one more tool,” said Hykes.

One key aspect where Hykes reckons Dagger can help is with secret management: “In this API we designed, there’s native support for secrets. We bake in the concept that these secrets are handled in a special way and they should not be mixed with the code … the security situation in CI pipelines is horrendous and no one at the moment can truly fix it … the root of the problem is there’s no modern platform, it’s just scripts all the way.”

What is coming next for Dagger? “In a nutshell, it’s cross-language reusable modules,” Hykes said. “Right now, we’re turning these DevOps pipelines creators into developers and they’re happily writing software to describe their pipelines, but they’re doing it each in the silo of their favourite language.” With the new feature “they get all the benefits of their pipelines being code, including using each other’s code … the Dagger engine will take your perfect build function written in Go, and make it usable from another function written in TypeScript or in Python,” he said.

This feature in development is currently called Project Zenith but “it will be launched with a more pragmatic, descriptive name in the next one to two months,” Hykes told us.