We use docker, with docker compose, to manage all the services we use. There are three main services that help us to get the data properly:
- Fetcher – The fetcher service download for us all the source files from the pension companies.
- Processor – After we got the files we need convert them into JSON object which can be use for processing.
- Enrichment – The JSON objects we got from the fetcher used don’t have fully qualified value. Each company can have various way to be represented – Teva, T.E.V.A, teva, tv25 etc. etc. etc. The service takes those variations and converts them to a unify texts.
The other services are peripheral services:
- BLOP – The data go into the
BLOPservice which is a
Flaskthat holds the data in a SQL tables.
- Gateway – The gateway service provides a single graphql endpoint which allow us to communicate with the other services.
- Front – A
vuejsapp that serves all the data from the blop service.
- Backoffice – Gives us dashboard for managing other services:
- Content such as blogs, enrichment tables, user authentication and more
- Manage downloaded files, manually upload files for processing
- Watch logs from other services
- Fix uploaded files
- Kafka – give us the option to notify services of events that occurred in other services.
- Logs – Allow us to log events which occurred in service: a file has been processed, file downloaded.
- Install Docker (make sure you have docker compose).
docker-compose up -d.
- Make sure you have NodeJS
- Install Angular CLI
- move to client directory:
npm run build -- --watch
npm run lintto check for lint mistakes.
npm run testto execute the unit tests via Karma.
npm run e2eto execute the end-to-end tests via Protractor.
In order to execute any python commands you have to do it inside the service container
docker-compose exec [service-name] /bin/sh this will open the shell inside the container
pycodestyle --show-source --max-line-length=120 --exclude=pension/migrations --show-pep8 .to check for lint mistakes.
isort . --recursive --check-onlyto check for import mistakes.
python manage.py testto run the unit tetst.