Luigi Python Package Setup Assistance
Budget: $10 – $30 USD
I'm looking for an expert who can help me set up the Luigi Python package for better visualization of my task flows. Currently, I manage 4-6 Python automations manually from the terminal and cron, and I aspire to streamline this process. None of these automations depend on each other, but I would like to have a clearer view of their operations.
Ideal skills for this project include:
- Proficiency in Python
- Experience with the Luigi package
- Knowledge of cron jobs
- Ability to provide clear visualizations of task flows
Luigi is a Python package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more. -> https://luigi.readthedocs.io/en/stable/index.html
Please note:
- luigi itself does not have precise time control, so if you need to set precise time use cron as in instruction
- don't forget to make scripts executable
- sudo chmod +x backup_remote.sh
- sudo chmod +x drop_local.sh
- sudo chmod +x restore_local.sh
- sudo pip install luigi
- it should match path in luigid.service at line 7
- ExecStart=/usr/local/bin/luigid --background --logdir /var/log
- then run luigi scheduler to manage tasks execution pipeline and logging
The normal steps will be but errors or dependencies may came up that I do not have time to correct, something very easy for someone with experience:
[1] Install luigi
pip install luigi
[2] Create luigi service
place file luigid.service to be available at this path
/etc/systemd/system/luigid.service
[3] Enable luigi service
sudo systemctl enable luigid.service
[4] Start service
sudo systemctl start luigid.service
[5] Check status
sudo systemctl status luigid.service
[6] Open visualiser
http://localhost:8082/
[7] Create configuration for remote db connection remote_db_config with lines like in .env
[8] Create configuration for local db connection local_db_config with lines like in .env
[9] Create cron for backup, place absolute path where py scripts and configuratioin files are located
0 1 * * * cd /home/ && backup_remote.sh
[10] Create cron for drop and replace, place absolute path where py scripts are located
0 2 * * * cd /home/ && drop_local.sh && restore_local.sh
Test and live!
Please provide a bid if you can assist with this setup.
Ideal skills for this project include:
- Proficiency in Python
- Experience with the Luigi package
- Knowledge of cron jobs
- Ability to provide clear visualizations of task flows
Luigi is a Python package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more. -> https://luigi.readthedocs.io/en/stable/index.html
Please note:
- luigi itself does not have precise time control, so if you need to set precise time use cron as in instruction
- don't forget to make scripts executable
- sudo chmod +x backup_remote.sh
- sudo chmod +x drop_local.sh
- sudo chmod +x restore_local.sh
- sudo pip install luigi
- it should match path in luigid.service at line 7
- ExecStart=/usr/local/bin/luigid --background --logdir /var/log
- then run luigi scheduler to manage tasks execution pipeline and logging
The normal steps will be but errors or dependencies may came up that I do not have time to correct, something very easy for someone with experience:
[1] Install luigi
pip install luigi
[2] Create luigi service
place file luigid.service to be available at this path
/etc/systemd/system/luigid.service
[3] Enable luigi service
sudo systemctl enable luigid.service
[4] Start service
sudo systemctl start luigid.service
[5] Check status
sudo systemctl status luigid.service
[6] Open visualiser
http://localhost:8082/
[7] Create configuration for remote db connection remote_db_config with lines like in .env
[8] Create configuration for local db connection local_db_config with lines like in .env
[9] Create cron for backup, place absolute path where py scripts and configuratioin files are located
0 1 * * * cd /home/ && backup_remote.sh
[10] Create cron for drop and replace, place absolute path where py scripts are located
0 2 * * * cd /home/ && drop_local.sh && restore_local.sh
Test and live!
Please provide a bid if you can assist with this setup.