Open Pension – Manage Service

We use docker, with docker compose, to manage all the services we use. There are three main services that help us to get the data properly:

• Fetcher – The fetcher service download for us all the source files from the pension companies.
• Processor – After we got the files we need convert them into JSON object which can be use for processing.
• Enrichment – The JSON objects we got from the fetcher used don’t have fully qualified value. Each company can have various way to be represented – Teva, T.E.V.A, teva, tv25 etc. etc. etc. The service takes those variations and converts them to a unify texts.

The other services are peripheral services:

• BLOP – The data go into the BLOP service which is a postgress and Flask that holds the data in a SQL tables.
• Gateway – The gateway service provides a single graphql endpoint which allow us to communicate with the other services.
• Front – A vuejs app that serves all the data from the blop service.
• Backoffice – Gives us dashboard for managing other services:
  • Content such as blogs, enrichment tables, user authentication and more
  • Manage downloaded files, manually upload files for processing
  • Watch logs from other services
  • Fix uploaded files
• Kafka – give us the option to notify services of events that occurred in other services.
• Logs – Allow us to log events which occurred in service: a file has been processed, file downloaded.

Pre requirements:

1. Install Docker (make sure you have docker compose).

Installation:

1. Run docker-compose up -d.
2. Make sure you have NodeJS
3. Install Angular CLI
4. move to client directory: cd client
5. run: npm i
6. run: npm run build -- --watch

Tests

Client

• Run npm run lint to check for lint mistakes.
• Run npm run test to execute the unit tests via Karma.
• Run npm run e2e to execute the end-to-end tests via Protractor.

Server

In order to execute any python commands you have to do it inside the service container docker-compose exec [service-name] /bin/sh this will open the shell inside the container

• Run pycodestyle --show-source --max-line-length=120 --exclude=pension/migrations --show-pep8 . to check for lint mistakes.
• Run isort . --recursive --check-only to check for import mistakes.
• Run python manage.py test to run the unit tetst.

Github