commit d3a9f3853990341abadbd6d0de8a841832180aa7 Author: Anselm Stordeur Date: Tue Jan 22 13:39:04 2019 +0100 feat: add minimal deployment diff --git a/.gitignore b/.gitignore new file mode 100644 index 00000000..74d3e87d --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +database/ diff --git a/README.md b/README.md new file mode 100644 index 00000000..e366c040 --- /dev/null +++ b/README.md @@ -0,0 +1,122 @@ +# Minimal deployment +This is an example for how a deployment for StApps can look like. The project also shows why we decided to use Docker +for the deployment on all universities. With help of Docker's virtualization you don't have to install and configure +anything beyond `docker` and `docker compose` itself. + +A complete StApps deployment consists of following projects: +``` +|*****| |*******| |*********| |**********| +| app | <----> | proxy | <----> | backend | <----> | database | +|*****| |*******| |*********| |**********| + ^ + | + |************| + | connectors | + |************| +``` + +This project shows a very fast way to deploy [backend](https://gitlab.com/openstapps/backend), +[database](https://gitlab.com/openstapps/database) and [api](https://gitlab.com/openstapps/api) (to copy some data from +an existing deployment of the TU Berlin). + + +With a running backend and database you should be able to run the [app](https://gitlab.tubit.tu-berlin.de/stapps/app) +with your deployment. + +# Step by step to your own deployment +At first you need to install [docker](https://docs.docker.com/install/) and +[docker-compose](https://docs.docker.com/compose/install/). Then clone or download this repository. To download the +docker images from GitLab you have to be authenticated against GitLabs container registry. Please execute +`docker login gitlab-registry.tubit.tu-berlin.de`. The requested credentials are the same as for GitLab. +`docker login` stores the credentials the users home directory. If you plan to execute docker commands only from root +or via sudo you should run `docker login` with root or sudo as well. + + +On Linux machines you should be able to proceed on [Execution on Linux](#execution-on-linux) otherwise follow me step +by step. + + +Be sure to execute all the following commands in the repository and with root privileges or access to the Docker daemon. + +## Start the database +Our current database is Elasticsearch and most configuration is contained in the Docker image. Since Docker is not a +full virtualization it uses the hosts kernel. That's why on some systems you may need to increase your virtual memory to +get Elasticsearch to work. +([documentation for increasing your virtual memory](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/vm-max-map-count.html)) + + +To start the database +```sh +# create a directory for all data +mkdir -p database + +# let the database be the owner of the folder +chown 1000:1000 database + +# start the database +docker-compose up -d database + +# have a look at the logs of the database +docker-compose logs database +``` + +As stated in the `docker-compose.yml` the database will expose port `9200` to other services in the same deployment +(all services in the file). + +## Start the backend +The backend is providing an interface to search and bulk index data into the database. Simply run: + +```sh +# start the backend +docker-compose up -d backend + +# have a look at the logs of the backend +docker-compose logs backend +``` + + +The backend will find the database on `http://database:9200` inside the deployment network. As stated in the +`docker-compose.yml` the backend will expose port 3000 on the host. You should be able to request the backend for it's +configuration on http://localhost:3000. + + +Now you have a running backend that can be queried by the app. You could already try to the install the +[app](https://gitlab.com/openstapps/app) and use it with your local deployment. That is pretty boring, because the +backend doesn't provide any data by now. + +There is no public deployment for the app version 2.0.0 by now. But you could copy some data from there. + +To import some `SCPlace`'s you could run: +```sh +docker-compose run --rm api copy --appVersion 2.0.0 place http://backend:3000 20 +``` +This will copy data using the [api](https://gitlab.com/openstapps/api) from the deployment of the TU Berlin to your +deployment. + +# Execution on Linux +On Linux you can execute a simple shell script that will run you through +steps above and copy some data in your deployment: + +```sh +sh start.sh +``` + +# Where to go from here +## Writing your own connector +You now have your own deployment and can import some data from your university into the backend. To write your own +program to import data you should checkout the [minimal-connector](https://gitlab.com/openstapps/minimal-connector/) +This is a small program that shows the usage of the [api](https://gitlab.com/openstapps/api/) and imports some example +data from TypeScript code. Feel free to copy the [minimal-connector](https://gitlab.com/openstapps/minimal-connector/) +and write your own connector with it. + +## Round off your deployment with the [proxy](https://gitlab.com/openstapps/proxy) +The backend is exposed on the port 3000 now. This means anyone can import data into your backend and you can only run +one version of the backend at a time. Have a look at the [proxy](https://gitlab.com/openstapps/proxy/) to secure and +round off your deployment. The proxy will allow you to run multiple deployments for different app version on one server +and provide some static data like images for your university. + +## Explore docker capabilities +Docker is a great tool with many great features. Have a look at the +[docker cli documentation](https://docs.docker.com/engine/reference/commandline/cli/) or our +[docker cheat sheet](https://gitlab.tubit.tu-berlin.de/stapps/projectmanagement/blob/master/DOCKER.md) to learn +more about it. diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 00000000..403cb989 --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,30 @@ +version: '2' +services: + database: + image: registry.gitlab.com/openstapps/database:master + volumes: + - ./database:/usr/share/elasticsearch/data + expose: + - "9200" + + backend: + image: registry.gitlab.com/openstapps/backend/default:master + environment: + ES_PORT_9200_TCP_ADDR: database + ES_PORT_9200_TCP_PORT: 9200 + NODE_CONFIG_ENV: "elasticsearch" + ALLOW_NO_TRANSPORT: "true" + expose: + - 3000 + ports: + - 3000:3000 + links: + - "database" + labels: + - stapps.version=1.0.0 + + api: + image: registry.gitlab.com/openstapps/api/copy:v0.0.3 + links: + - "backend" + diff --git a/start.sh b/start.sh new file mode 100755 index 00000000..3e432a82 --- /dev/null +++ b/start.sh @@ -0,0 +1,89 @@ +SUDO='' + +if [ $(id -u) != 0 ]; then + SUDO='sudo' +fi + +if ! hash docker 2>/dev/null; then + echo "\"docker\" executable not found. Please install docker." + exit 1 +fi + +if ! hash docker-compose 2>/dev/null; then + echo "\"docker-compose\" executable not found. Please install docker-compose." + exit 1 +fi + +# check kernel parameter +# check if vm.max_map_count is at least 262144 on host +# this is needed for elasticsearch >5.x +ACTUAL_MAX_MAP_COUNT=$(sysctl -n vm.max_map_count) +MINIMAL_MAX_MAP_COUNT=262144 + +if [ "$ACTUAL_MAX_MAP_COUNT" -lt "$MINIMAL_MAX_MAP_COUNT" ]; then + echo "vm.max_map_count is too low. This is a kernel parameter used by java which is used by elasticsearch. Elasticsearch won't run if vm.max_map_count is too low." + while true; do + read -p "Do you wish to increase vm.max_map_count to 262144? " yn + case $yn in + [Yy]* ) $SUDO sysctl -w "vm.max_map_count=262144"; echo "To make this change permanent after restart: Add \"vm.max_map_count=262144\" to \"/etc/sysctl.conf\""; break;; + [Nn]* ) exit 1;; + * ) echo "Please answer yes or no.";; + esac + done +fi + +mkdir -p database +$SUDO chown 1000:1000 database + +cat <