Example

This document provides a brief overview of project_manager’s basic functionality.

Environment setup

tree

.
|____dummy_project
| |____my_conf.yaml
| |____run.py
|____config.yaml

cat config.yaml

project_source: dummy_project
working_dir: tmp

exec_command:
    - python3 run.py
result_files:
    - results

base_config: dummy_project/my_conf.yaml
config_parameters:
    - key: message
      values: [A, B, C]

cat dummy_project/my_conf.yaml

message: 'this is important'

cat dummy_project/run.py

import os
import yaml


def main():
    with open('my_conf.yaml') as fd:
        config = yaml.full_load(fd)

    os.makedirs('results')
    with open('results/data.txt', 'w') as fd:
        fd.write(config['message'])


if __name__ == '__main__':
    main()

Pipeline execution

Setup directory for each configuration

project_manager build -c config.yaml

Setting up environments: 100%|███████████████████| 3/3 [00:00<00:00, 238.72it/s]

Execute scripts for each configuration

project_manager run -c config.yaml

  0%|                                                     | 0/3 [00:00<?, ?it/s]run.message=A
 > python3 run.py
 33%|███████████████                              | 1/3 [00:00<00:00,  5.37it/s]run.message=C
 > python3 run.py
 67%|██████████████████████████████               | 2/3 [00:00<00:00,  5.76it/s]run.message=B
 > python3 run.py
100%|█████████████████████████████████████████████| 3/3 [00:00<00:00,  6.30it/s]

Gather results from each run

project_manager gather -c config.yaml

run.message=A
 > results/data.txt
run.message=C
 > results/data.txt
run.message=B
 > results/data.txt

Investigate results

tree tmp/

|____
|____aggregated_results
| |____results
| | |____data.message=B.txt
| | |____data.message=A.txt
| | |____data.message=C.txt
|____run.message=A
| |____my_conf.yaml
| |____results
| | |____data.txt
| |____run.py
|____run.message=C
| |____my_conf.yaml
| |____results
| | |____data.txt
| |____run.py
|____run.message=B
| |____my_conf.yaml
| |____results
| | |____data.txt
| |____run.py

cat tmp/aggregated_results/results/*

ABC