Project:RunImporter

Overview

This section details how to import items or test new changes to the importer locally, on the staging portal, and on the main portal.

Import Locally

Development, testing, and debugging can be done locally on a virtual machine. Instructions for setting up the VM and connecting to it can be found here.

To set up the repositories for local development:

1. clone the portal-compose GitHub repository and follow the steps detailed in the ReadMe section for the installation and configuration.
2. clone the docker-compose GitHub repository and follow the steps detailed in the ReadMe section to set up the configuration.

After implementing any changes to the importer:

1. navigate to the portal-compose repo cd portal-compose
2. Start the containers (if they are not already up) (which might take a while the first time) sudo docker compose up -d
3. Access the importer container sudo docker exec -it mardi-importer bash
4. Run the importer python3 /mardi_importer/mardi_importer/scripts/import.py --mode <mode>
5. If the run was successful, you should be able to load a local instance on localhost:8080 in your browser. Make sure to forward the port first eg. shh -L 8080:127.0.0.1:8080 user@vm_ip_address

During development, if you need to pull any new changes to the portal-compose repo, you can restart your containers by following these steps:

1. cd portal-compose
2. sudo docker compose down
3. sudo docker volume prune --all
4. sudo docker compose up -d

Import to the staging portal

After local development, changes can be tested on the staging portal where the rest of the team can also check them and provide feedback.

1. If changes have been made to the script, they need to be pushed and merged to the docker-importer repo so that the docker image includes the changes. Make sure the commits are signed.
2. Trigger the workflow on the mardi03 GitHub repo (instructions are on git). (?)
3. Execute the importer script from within the mardi-importer container. To do this, there are two options:
   • Executing the script on the mardi03 server:
     1. connect to mardi03 via ssh shh mardi03
     2. navigate to the portal-compose directory: cd portal-compose
     3. access the importer container: docker exec -it mardi-importer bash
     4. execute the importer script that's in /mardi_importer/mardi_importer/scripts
   • Executing the script in Portainer:
     1. log in to Portainer (user: admin)
     2. Under 'Environments' click 'local' and then click 'containers' in the side panel on the left
     3. Search for the mardi-importer container
     4. If changes have been made to the code, click on "Recreate" with the option to re-pull the image, to pull the latest changes from the docker-importer repo.
     5. Click '>_ Console' and 'Connect' to access the container.
     6. execute the importer script that's in /mardi_importer/mardi_importer/scripts

Import to the portal

It is recommended to have executed a trial run involving any new changes on the staging server before uploading to the main portal. Assuming changes are already merged to the main branch on the docker-importer GitHub repo and included in the mardi-importer container, the execution steps are similar to those above.

1. Trigger the workflow on the mardi02 GitHub repo (?).
2. Execute the script on the mardi02 server:
   1. connect to mardi03 via ssh shh mardi02
   2. navigate to the portal-compose directory: cd portal-compose
   3. access the importer container: docker exec -it mardi-importer bash
   4. execute the importer script that's in /mardi_importer/mardi_importer/scripts