Example¶
A quick usage overview:
$ tree
.
├── config.yaml
└── dummy_project
├── my_conf.yaml
└── run.py
config.yaml:
project_source: dummy_project
working_dir: tmp
exec_command:
- python3 run.py
result_dirs:
- results
base_config: dummy_project/my_conf.yaml
config_parameters:
- key: message
values: [A, B, C]
dummy_project/my_conf.yaml:
message: 'this is important'
run.py:
import os
import yaml
def main():
with open('my_conf.yaml') as fd:
config = yaml.load(fd)
os.makedirs('results')
with open('results/data.txt', 'w') as fd:
fd.write(config['message'])
if __name__ == '__main__':
main()
We can then run the pipeline:
$ project_manager build
Setting up environments: 100%|██████████████████████████| 3/3 [00:00<00:00, 477.57it/s]
$ project_manager run
run.message=A
> python3 run.py
run.message=B
> python3 run.py
run.message=C
> python3 run.py
$ project_manager gather
run.message=A
> data.txt
run.message=B
> data.txt
run.message=C
> data.txt
Here’s the result:
$ tree tmp/
tmp/
├── aggregated_results
│ └── results
│ ├── data.message=A.txt
│ ├── data.message=B.txt
│ └── data.message=C.txt
├── run.message=A
│ ├── my_conf.yaml
│ ├── results
│ │ └── data.txt
│ └── run.py
├── run.message=B
│ ├── my_conf.yaml
│ ├── results
│ │ └── data.txt
│ └── run.py
└── run.message=C
├── my_conf.yaml
├── results
│ └── data.txt
└── run.py
$ cat tmp/aggregated_results/results/*
ABC