Why CI/CD Tools like Jenkins/TeamCity Don't Support BYOL Bring Your Own Language for Writing Pipelines

Continuous Integration and Continuous Deployment (CI/CD) tools such as Jenkins and TeamCity fundamentally support the creation of pipelines that streamline and automate software development processes. One question often arises: why don’t these platforms natively support Bring Your Own Language (BYOL) approaches for writing pipelines? This article explores the reasons and implications behind this decision, highlighting technical constraints, maintenance challenges, and broader implications for CI/CD practices.

Technical Feasibility and Platform-Specific Libraries

Firstly, it's important to understand that CI/CD tools like Jenkins and TeamCity run their pipelines within a sandbox environment, directly interfacing with the underlying platform, typically a Java Virtual Machine (JVM). This design choice is driven by technical feasibility and maintainability. If a generic language were allowed, the tool would need to interpret and map it to the underlying platform, increasing complexity and potentially leading to unforeseen issues.

Most CI/CD tools like Jenkins adopt domain-specific languages (DSLs), often based on YAML or Groovy, to achieve a balance between flexibility and platform integration. These DSLs are designed to work seamlessly with the tool's core functionalities, making the process as predictable and efficient as possible. Any arbitrary programming language would introduce unnecessary overhead, making the pipeline less reliable and harder to maintain.

Technical Difficulties and Maintenance Challenges

Implementing a BYOL approach in CI/CD tools would require significant effort to ensure that the custom languages are properly interpreted and integrated into the platform. This would lead to a maintenance nightmare, as the tool would need to support a wide range of languages and ensure their compatibility with the underlying infrastructure.

The Scripted Pipelines feature in Jenkins, which allows the use of arbitrary scripting languages, has been criticized for issues such as complexity, lack of predictability, and difficulty in debugging. These problems are compounded when the script is written in a language that is not tightly integrated with the CI/CD tool.

Is There Value in Arbitrary Language Support?

The crux of the matter is whether the benefits of using a generic CI/CD tool are outweighed by the need to write custom pipeline scripts in arbitrary languages. The main argument against this is that a generic CI/CD tool's primary function is to orchestrate the flow of tasks rather than acting as a general-purpose programming platform.

For Jenkins, the Declarative Pipeline format is recommended as it is more predictable, easier to maintain, and better suited for the specific needs of pipeline management. This approach leverages the tool's native capabilities to ensure that the pipeline is robust and adheres to best practices. Tools like Jenkins X, with its jx binary, further enhance this by providing additional integrations and support forCI/CD workflows.

Conclusion: Focusing on Specific Needs

Instead of supporting arbitrary languages, CI/CD tools should aim to provide a solid, predictable, and specific set of tools for writing pipelines. This approach allows for a more controlled and reliable environment, where common pipeline tasks can be executed consistently and efficiently. While developers may have specific requirements that can be addressed with custom scripting, these should be handled by tools that can run on any machine, independent of the CI/CD tool.

By adhering to this principle, CI/CD tools can ensure that the pipeline processes are both robust and maintainable, ultimately enhancing the overall software development lifecycle.