I find myself regularly chaining programs together into software pipelines, and decided that having a pipeline management tool would be helpful. So I wrote one and posted the code here. To illustrate how the tool works, consider the following software pipeline:
Here we see that the steps “get weather forecast” and “get stock quote” depend on the completion of “generate timestamp”. Similarly, “compile a report” depends on completion of “get weather forecast” and “get stock quote”.
Future posts will detail how the pipeline manager works under the hood. For now I wanted to demonstrate how its Django-driven data model allows easy pipeline specification through the Django administrator interface:
We first define the steps of a pipeline:
Once all the steps have been added, we have:
We now link the steps by specifying dependencies:
Once all the dependencies have been specified, we have the following list. Note that the first step in the pipeline has a dependency on “None”:
We then add environment variables that will be available for use by the steps:
Once all the environment variables have been specified, we have:
We then attach the dependencies and environment variables to a pipeline:
Finally, we run the pipeline (not shown here) and can view progress through the Django administrator page’s “Run steps” view:
