We use docker, with docker compose, to manage all the services we use. There are three main services that help us to get the data properly:
- Fetcher – The fetcher service download for us all the source files from the pension companies.
- Processor – After we got the files we need convert them into JSON object which can be use for processing.
- Enrichment – The JSON objects we got from the fetcher used don’t have fully qualified value. Each company can have various way to be represented – Teva, T.E.V.A, teva, tv25 etc. etc. etc. The service takes those variations and converts them to a unify texts.
The other services are peripheral services:
- BLOP – The data go into the
BLOP
service which is apostgress
andFlask
that holds the data in a SQL tables. - Gateway – The gateway service provides a single graphql endpoint which allow us to communicate with the other services.
- Front – A
vuejs
app that serves all the data from the blop service. - Backoffice – Gives us dashboard for managing other services:
- Content such as blogs, enrichment tables, user authentication and more
- Manage downloaded files, manually upload files for processing
- Watch logs from other services
- Fix uploaded files
- Kafka – give us the option to notify services of events that occurred in other services.
- Logs – Allow us to log events which occurred in service: a file has been processed, file downloaded.
Pre requirements:
- Install Docker (make sure you have docker compose).
Installation:
- Run
docker-compose up -d
. - Make sure you have NodeJS
- Install Angular CLI
- move to client directory:
cd client
- run:
npm i
- run:
npm run build -- --watch
Tests
Client
- Run
npm run lint
to check for lint mistakes. - Run
npm run test
to execute the unit tests via Karma. - Run
npm run e2e
to execute the end-to-end tests via Protractor.
Server
docker-compose exec [service-name] /bin/sh
this will open the shell inside the container
In order to execute any python commands you have to do it inside the service container - Run
pycodestyle --show-source --max-line-length=120 --exclude=pension/migrations --show-pep8 .
to check for lint mistakes. - Run
isort . --recursive --check-only
to check for import mistakes. - Run
python manage.py test
to run the unit tetst.
Comments