diff --git a/examples/minimal-deployment/.gitignore b/examples/minimal-deployment/.gitignore new file mode 100644 index 00000000..485dee64 --- /dev/null +++ b/examples/minimal-deployment/.gitignore @@ -0,0 +1 @@ +.idea diff --git a/examples/minimal-deployment/.gitlab/issue_templates/bug.md b/examples/minimal-deployment/.gitlab/issue_templates/bug.md new file mode 100644 index 00000000..f46f50ae --- /dev/null +++ b/examples/minimal-deployment/.gitlab/issue_templates/bug.md @@ -0,0 +1,38 @@ +## Summary + +(Summarize the bug encountered concisely) + + +## Steps to reproduce + +(How one can reproduce the issue - this is very important) + + +## Example Project + +(If possible, please create an example project here on GitLab.com that exhibits the problematic behaviour, and link to it here in the bug report) + +(If you are using an older version of GitLab, this will also determine whether the bug has been fixed in a more recent version) + + +## What is the current bug behavior? + +(What actually happens) + + +## What is the expected correct behavior? + +(What you should see instead) + + +## Relevant logs and/or screenshots + +(Paste any relevant logs - please use code blocks (```) to format console output, +logs, and code as it's very hard to read otherwise.) + + +## Possible fixes + +(If you can, link to the line of code that might be responsible for the problem) + +/label ~meeting diff --git a/examples/minimal-deployment/.gitlab/issue_templates/feature.md b/examples/minimal-deployment/.gitlab/issue_templates/feature.md new file mode 100644 index 00000000..cb9b0517 --- /dev/null +++ b/examples/minimal-deployment/.gitlab/issue_templates/feature.md @@ -0,0 +1,16 @@ +## Description + +(Describe the feature that you're requesting concisely) + + +## Explanation + +(Explain why the feature is necessary) + + +## Dependencies, issues to be resolved beforehand + +(List issues or dependencies that need to be resolved before this feature can be implemented) + + +/label ~meeting diff --git a/examples/minimal-deployment/README.md b/examples/minimal-deployment/README.md new file mode 100644 index 00000000..5df69962 --- /dev/null +++ b/examples/minimal-deployment/README.md @@ -0,0 +1,139 @@ +# Minimal deployment +This is an example for how a deployment for StApps can look like. The project also shows why we decided to use Docker +for the deployment on all universities. With help of Docker's virtualization you don't have to install and configure +anything beyond `docker` and `docker compose` itself. + +A complete StApps deployment consists of following projects: +``` +|*****| |*******| |*********| |**********| +| app | <----> | proxy | <----> | backend | <----> | database | +|*****| |*******| |*********| |**********| + ^ + | + |************| + | connectors | + |************| +``` + +This project shows a very fast way to deploy [backend](https://gitlab.com/openstapps/backend), +[database](https://gitlab.com/openstapps/database) and [api](https://gitlab.com/openstapps/api) (to copy some data from +an existing deployment of the TU Berlin). + + +With a running backend and database you should be able to run the [app](https://gitlab.com/openstapps/app) +with your deployment. + +# Step by step to your own deployment +At first you need to install [docker](https://docs.docker.com/install/) and +[docker-compose](https://docs.docker.com/compose/install/). Then clone or download this repository. To download the +docker images from GitLab you have to be authenticated against GitLabs container registry. Please execute +`docker login registry.gitlab.com`. The requested credentials are the same as for GitLab. +`docker login` stores the credentials the users home directory. If you plan to execute docker commands only from root +or via sudo you should run `docker login` with root or sudo as well. + + +On Linux machines you should be able to proceed on [Execution on Linux](#execution-on-linux) otherwise follow me step +by step. + + +Be sure to execute all the following commands in the repository and with root privileges or access to the Docker daemon. + +## Start the database +Our current database is Elasticsearch and most configuration is contained in the Docker image. Since Docker is not a +full virtualization it uses the hosts kernel. That's why on some systems you may need to increase your virtual memory to +get Elasticsearch to work. +([documentation for increasing your virtual memory](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/vm-max-map-count.html)) + + +To start the database +```sh +# let the elasticsearch user and group be the owner of the folder +chown 100:101 database + +# start the database container +docker-compose up -d database + +# have a look at the logs of the database +docker-compose logs database +``` + +As stated in the `docker-compose.yml` the database will expose port `9200` to other services in the same deployment +(all services in the file). + +## Start the backend +The backend is providing an interface to search and bulk index data into the database. Simply run: + +```sh +# start the backend container +docker-compose up -d backend + +# have a look at the logs of the backend +docker-compose logs backend +``` + + +The backend will find the database on `http://database:9200` inside the deployment network. As stated in the +`docker-compose.yml` the backend will expose port 3000 on the host. You should be able to request the backend for it's +configuration on http://localhost:3000 (or http://MY-IP-ADDRESS:3000). + + +Now you have a running backend that can be queried by the app. You could already try to the install the +[app](https://gitlab.com/openstapps/app) and use it with your local deployment. That is pretty boring, because the +backend doesn't provide any data by now. + +There is no public deployment for the app version 2.0.0 by now. But you could copy some data from there. + +To import some `SCPlace`'s you could run: +```shell +docker-compose run --rm api copy --appVersion 2.0.0 place http://backend:3000 20 +``` +This will copy data using the [api](https://gitlab.com/openstapps/api) from the deployment of the TU Berlin to your +deployment. + +# Execution on Linux +On Linux you can execute a simple shell script that will run you through +steps above and copy some data in your deployment: + +```shell +sh start.sh +``` + +# Where to go from here +## Writing your own connector +You now have your own deployment and can import some data from your university into the backend. To write your own +program to import data you should checkout the [minimal-connector](https://gitlab.com/openstapps/minimal-connector/) +This is a small program that shows the usage of the [api](https://gitlab.com/openstapps/api/) and imports some example +data from TypeScript code. Feel free to fork the [minimal-connector](https://gitlab.com/openstapps/minimal-connector/) +and write your own connector with it. + +## Add your connector +You may want to integrate your connectors in the `docker-compose.yml` to +deploy your whole StApps-backend-environment in a single command. Use the +`systemd` directory in this project as an example of how to deploy +your connector(s) and configure `systemd` services and timers. + +The service will try to restart a named container. That means for successful +execution the container needs to be started once before with the appropriate +command. +```shell +docker-compose up minimal-connector +``` + +To enable the service that runs the connector periodically execute: +```shell +systemctl --now enable /absolute/path/minimal-deployment/minimal-connector/minimal-connector.service /absolute/path/minimal-deployment/minimal-connector/minimal-connector.timer +``` +This command will immediately start the service on execution. + +## Round off your deployment with the [proxy](https://gitlab.com/openstapps/proxy) +The backend is exposed on the port 3000 now. This means anyone can import data into your backend and you can only run +one version of the backend at a time. Have a look at the [proxy](https://gitlab.com/openstapps/proxy/) to secure and +round off your deployment. The proxy will add a layer of security, allowing you to run multiple deployments for +different app version on one server and provide some static data like images for your university. + + +## Explore docker capabilities +Docker is a great tool with many great features. Have a look at the +[docker cli documentation](https://docs.docker.com/engine/reference/commandline/cli/) or our +[docker cheat sheet](https://gitlab.com/openstapps/projectmanagement/blob/master/project-docs/DOCKER_CHEAT_SHEET.md) to learn +more about it. diff --git a/examples/minimal-deployment/database/.gitignore b/examples/minimal-deployment/database/.gitignore new file mode 100644 index 00000000..f54b6b96 --- /dev/null +++ b/examples/minimal-deployment/database/.gitignore @@ -0,0 +1 @@ +nodes/ diff --git a/examples/minimal-deployment/docker-compose.yml b/examples/minimal-deployment/docker-compose.yml new file mode 100644 index 00000000..6e1630b1 --- /dev/null +++ b/examples/minimal-deployment/docker-compose.yml @@ -0,0 +1,44 @@ +version: '3.7' +services: + database: + image: registry.gitlab.com/openstapps/database:master + volumes: + - ./database:/usr/share/elasticsearch/data + expose: + - "9200" + restart: unless-stopped + + backend: + image: registry.gitlab.com/openstapps/backend/default:core-0.31 + environment: + ES_ADDR: "http://database:9200" + NODE_CONFIG_ENV: "elasticsearch" + ALLOW_NO_TRANSPORT: "true" + expose: + - 3000 + ports: + - 3000:3000 + links: + - "database" + labels: + - stapps.version=1.0.0 + restart: unless-stopped + depends_on: + - database + + api: + image: registry.gitlab.com/openstapps/api/copy:core-0.23 + links: + - "backend" + + minimal-connector: + image: registry.gitlab.com/openstapps/minimal-connector:core-0.23 + container_name: minimal-connector-0.23 + command: ["http://backend:3000", "minimal-connector", "f-u"] + + app: + image: registry.gitlab.com/openstapps/app/executable:core-0.23 + expose: + - 8100 + ports: + - 8100:8100 diff --git a/examples/minimal-deployment/start.sh b/examples/minimal-deployment/start.sh new file mode 100644 index 00000000..3c00bcff --- /dev/null +++ b/examples/minimal-deployment/start.sh @@ -0,0 +1,96 @@ +SUDO='' + +if [ $(id -u) != 0 ]; then + SUDO='sudo' +fi + +if ! hash docker 2>/dev/null; then + echo "\"docker\" executable not found. Please install docker." + exit 1 +fi + +if ! hash docker-compose 2>/dev/null; then + echo "\"docker-compose\" executable not found. Please install docker-compose." + exit 1 +fi + +# check kernel parameter +# check if vm.max_map_count is at least 262144 on host +# this is needed for elasticsearch >5.x +ACTUAL_MAX_MAP_COUNT=$(sysctl -n vm.max_map_count) +MINIMAL_MAX_MAP_COUNT=262144 + +if [ "$ACTUAL_MAX_MAP_COUNT" -lt "$MINIMAL_MAX_MAP_COUNT" ]; then + echo "vm.max_map_count is too low. This is a kernel parameter used by java which is used by elasticsearch. Elasticsearch won't run if vm.max_map_count is too low." + while true; do + read -p "Do you wish to increase vm.max_map_count to 262144? " yn + case $yn in + [Yy]* ) $SUDO sysctl -w "vm.max_map_count=262144"; echo "To make this change permanent after restart: Add \"vm.max_map_count=262144\" to \"/etc/sysctl.conf\""; break;; + [Nn]* ) exit 1;; + * ) echo "Please answer yes or no.";; + esac + done +fi + +$SUDO chown 100:101 database + +cat <