Theaninova/openstapps
1
0
Fork 0
mirror of https://gitlab.com/openstapps/openstapps.git synced 2025-12-13 01:36:22 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'minimal-deployment/master'

Browse Source
This commit is contained in:
Thea Schöbl
2023-03-14 17:36:27 +01:00
parent d8baa99fc6 bbed8bf182
commit 93b651b35b
9 changed files with 353 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
examples/minimal-deployment/.gitignore vendored Normal file
View File

@@ -0,0 +1 @@
.idea

38
examples/minimal-deployment/.gitlab/issue_templates/bug.md Normal file
View File

@@ -0,0 +1,38 @@
## Summary
(Summarize the bug encountered concisely)
## Steps to reproduce
(How one can reproduce the issue - this is very important)
## Example Project
(If possible, please create an example project here on GitLab.com that exhibits the problematic behaviour, and link to it here in the bug report)
(If you are using an older version of GitLab, this will also determine whether the bug has been fixed in a more recent version)
## What is the current bug behavior?
(What actually happens)
## What is the expected correct behavior?
(What you should see instead)
## Relevant logs and/or screenshots
(Paste any relevant logs - please use code blocks (```) to format console output,
logs, and code as it's very hard to read otherwise.)
## Possible fixes
(If you can, link to the line of code that might be responsible for the problem)
/label ~meeting

16
examples/minimal-deployment/.gitlab/issue_templates/feature.md Normal file
View File

@@ -0,0 +1,16 @@
## Description
(Describe the feature that you're requesting concisely)
## Explanation
(Explain why the feature is necessary)
## Dependencies, issues to be resolved beforehand
(List issues or dependencies that need to be resolved before this feature can be implemented)
/label ~meeting

139
examples/minimal-deployment/README.md Normal file
View File

@@ -0,0 +1,139 @@
# Minimal deployment
This is an example for how a deployment for StApps can look like. The project also shows why we decided to use Docker
for the deployment on all universities. With help of Docker's virtualization you don't have to install and configure
anything beyond `docker` and `docker compose` itself.
A complete StApps deployment consists of following projects:
```
|*****| |*******| |*********| |**********|
| app | <----> | proxy | <----> | backend | <----> | database |
|*****| |*******| |*********| |**********|
^
|
|************|
| connectors |
|************|
```
This project shows a very fast way to deploy [backend](https://gitlab.com/openstapps/backend),
[database](https://gitlab.com/openstapps/database) and [api](https://gitlab.com/openstapps/api) (to copy some data from
an existing deployment of the TU Berlin).
With a running backend and database you should be able to run the [app](https://gitlab.com/openstapps/app)
with your deployment.
# Step by step to your own deployment
At first you need to install [docker](https://docs.docker.com/install/) and
[docker-compose](https://docs.docker.com/compose/install/). Then clone or download this repository. To download the
docker images from GitLab you have to be authenticated against GitLabs container registry. Please execute
`docker login registry.gitlab.com`. The requested credentials are the same as for GitLab.
`docker login` stores the credentials the users home directory. If you plan to execute docker commands only from root
or via sudo you should run `docker login` with root or sudo as well.
On Linux machines you should be able to proceed on [Execution on Linux](#execution-on-linux) otherwise follow me step
by step.
Be sure to execute all the following commands in the repository and with root privileges or access to the Docker daemon.
## Start the database
Our current database is Elasticsearch and most configuration is contained in the Docker image. Since Docker is not a
full virtualization it uses the hosts kernel. That's why on some systems you may need to increase your virtual memory to
get Elasticsearch to work.
([documentation for increasing your virtual memory](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/vm-max-map-count.html))
To start the database
```sh
# let the elasticsearch user and group be the owner of the folder
chown 100:101 database
# start the database container
docker-compose up -d database
# have a look at the logs of the database
docker-compose logs database
```
As stated in the `docker-compose.yml` the database will expose port `9200` to other services in the same deployment
(all services in the file).
## Start the backend
The backend is providing an interface to search and bulk index data into the database. Simply run:
```sh
# start the backend container
docker-compose up -d backend
# have a look at the logs of the backend
docker-compose logs backend
```
The backend will find the database on `http://database:9200` inside the deployment network. As stated in the
`docker-compose.yml` the backend will expose port 3000 on the host. You should be able to request the backend for it's
configuration on http://localhost:3000 (or http://MY-IP-ADDRESS:3000).
Now you have a running backend that can be queried by the app. You could already try to the install the
[app](https://gitlab.com/openstapps/app) and use it with your local deployment. That is pretty boring, because the
backend doesn't provide any data by now.
There is no public deployment for the app version 2.0.0 by now. But you could copy some data from there.
To import some `SCPlace`'s you could run:
```shell
docker-compose run --rm api copy --appVersion 2.0.0 place <url-of-public-deployment> http://backend:3000 20
```
This will copy data using the [api](https://gitlab.com/openstapps/api) from the deployment of the TU Berlin to your
deployment.
# Execution on Linux
On Linux you can execute a simple shell script that will run you through
steps above and copy some data in your deployment:
```shell
sh start.sh
```
# Where to go from here
## Writing your own connector
You now have your own deployment and can import some data from your university into the backend. To write your own
program to import data you should checkout the [minimal-connector](https://gitlab.com/openstapps/minimal-connector/)
This is a small program that shows the usage of the [api](https://gitlab.com/openstapps/api/) and imports some example
data from TypeScript code. Feel free to fork the [minimal-connector](https://gitlab.com/openstapps/minimal-connector/)
and write your own connector with it.
## Add your connector
You may want to integrate your connectors in the `docker-compose.yml` to
deploy your whole StApps-backend-environment in a single command. Use the
`systemd` directory in this project as an example of how to deploy
your connector(s) and configure `systemd` services and timers.
The service will try to restart a named container. That means for successful
execution the container needs to be started once before with the appropriate
command.
```shell
docker-compose up minimal-connector
```
To enable the service that runs the connector periodically execute:
```shell
systemctl --now enable /absolute/path/minimal-deployment/minimal-connector/minimal-connector.service /absolute/path/minimal-deployment/minimal-connector/minimal-connector.timer
```
This command will immediately start the service on execution.
## Round off your deployment with the [proxy](https://gitlab.com/openstapps/proxy)
The backend is exposed on the port 3000 now. This means anyone can import data into your backend and you can only run
one version of the backend at a time. Have a look at the [proxy](https://gitlab.com/openstapps/proxy/) to secure and
round off your deployment. The proxy will add a layer of security, allowing you to run multiple deployments for
different app version on one server and provide some static data like images for your university.
## Explore docker capabilities
Docker is a great tool with many great features. Have a look at the
[docker cli documentation](https://docs.docker.com/engine/reference/commandline/cli/) or our
[docker cheat sheet](https://gitlab.com/openstapps/projectmanagement/blob/master/project-docs/DOCKER_CHEAT_SHEET.md) to learn
more about it.

1
examples/minimal-deployment/database/.gitignore vendored Normal file
View File

@@ -0,0 +1 @@
nodes/

44
examples/minimal-deployment/docker-compose.yml Normal file
View File

@@ -0,0 +1,44 @@
version: '3.7'
services:
database:
image: registry.gitlab.com/openstapps/database:master
volumes:
- ./database:/usr/share/elasticsearch/data
expose:
- "9200"
restart: unless-stopped
backend:
image: registry.gitlab.com/openstapps/backend/default:core-0.31
environment:
ES_ADDR: "http://database:9200"
NODE_CONFIG_ENV: "elasticsearch"
ALLOW_NO_TRANSPORT: "true"
expose:
- 3000
ports:
- 3000:3000
links:
- "database"
labels:
- stapps.version=1.0.0
restart: unless-stopped
depends_on:
- database
api:
image: registry.gitlab.com/openstapps/api/copy:core-0.23
links:
- "backend"
minimal-connector:
image: registry.gitlab.com/openstapps/minimal-connector:core-0.23
container_name: minimal-connector-0.23
command: ["http://backend:3000", "minimal-connector", "f-u"]
app:
image: registry.gitlab.com/openstapps/app/executable:core-0.23
expose:
- 8100
ports:
- 8100:8100

96
examples/minimal-deployment/start.sh Normal file
View File

@@ -0,0 +1,96 @@
SUDO=''
if [ $(id -u) != 0 ]; then
SUDO='sudo'
fi
if ! hash docker 2>/dev/null; then
echo "\"docker\" executable not found. Please install docker."
exit 1
fi
if ! hash docker-compose 2>/dev/null; then
echo "\"docker-compose\" executable not found. Please install docker-compose."
exit 1
fi
# check kernel parameter
# check if vm.max_map_count is at least 262144 on host
# this is needed for elasticsearch >5.x
ACTUAL_MAX_MAP_COUNT=$(sysctl -n vm.max_map_count)
MINIMAL_MAX_MAP_COUNT=262144
if [ "$ACTUAL_MAX_MAP_COUNT" -lt "$MINIMAL_MAX_MAP_COUNT" ]; then
echo "vm.max_map_count is too low. This is a kernel parameter used by java which is used by elasticsearch. Elasticsearch won't run if vm.max_map_count is too low."
while true; do
read -p "Do you wish to increase vm.max_map_count to 262144? " yn
case $yn in
[Yy]* ) $SUDO sysctl -w "vm.max_map_count=262144"; echo "To make this change permanent after restart: Add \"vm.max_map_count=262144\" to \"/etc/sysctl.conf\""; break;;
[Nn]* ) exit 1;;
* ) echo "Please answer yes or no.";;
esac
done
fi
$SUDO chown 100:101 database
cat <<EOF
Please execute the following command in another shell, but in the same directory.
This will start backend (and database, which is its dependency) in interactive mode so that you can see the logging
output of the backend container in your shell.
$SUDO docker-compose up backend
After you've done this and no more log messages appear **PRESS ENTER** to import data from the b-tu backend.
Useful commands:
* If you want to see logs from database container in real time, you can use the following command:
$SUDO docker-compose logs -f database
* In case you don't need the containers to be running anymore, you can also stop them using docker-compose command (example for database container):
$SUDO docker-compose stop database
* If you want to test the functionality of the "minimal-connector", use the following command.
$SUDO docker-compose up minimal-connector
* In case you want to see how the data looks like in the app, start its corresponding container.
$SUDO docker-compose up app
EOF
read _
while true; do
read -p "How much do you want to import? (A)ll, (M)inimal or (N)othing? " yn
case $yn in
[Aa]* ) CHOICE="a"; break ;;
[Mm]* ) CHOICE="m"; break ;;
[Nn]* ) CHOICE="n"; break ;;
* ) echo "Please choose A, M or N!";;
esac
done
if [ "$CHOICE" = "n" ]; then
exit;
fi
echo "This is currently not supported, because there is no public deployment for the app v2.0.0 by now."
exit 1
$SUDO docker-compose run --rm api copy --appVersion 2.0.0 place https://stappsbe01.innocampus.tu-berlin.de http://backend:3000 20
$SUDO docker-compose run --rm api copy --appVersion 2.0.0 dish https://stappsbe01.innocampus.tu-berlin.de http://backend:3000 20
if [ "$CHOICE" = "m" ]; then
exit;
fi
$SUDO docker-compose run --rm api copy --appVersion 2.0.0 article https://stappsbe01.innocampus.tu-berlin.de http://backend:3000 20
$SUDO docker-compose run --rm api copy --appVersion 2.0.0 floor https://stappsbe01.innocampus.tu-berlin.de http://backend:3000 20
$SUDO docker-compose run --rm api copy --appVersion 2.0.0 organization https://stappsbe01.innocampus.tu-berlin.de http://backend:3000 20
$SUDO docker-compose run --rm api copy --appVersion 2.0.0 person https://stappsbe01.innocampus.tu-berlin.de http://backend:3000 20
$SUDO docker-compose run --rm api copy --appVersion 2.0.0 catalog https://stappsbe01.innocampus.tu-berlin.de http://backend:3000 20
$SUDO docker-compose run --rm api copy --appVersion 2.0.0 event https://stappsbe01.innocampus.tu-berlin.de http://backend:3000 20
$SUDO docker-compose run --rm api copy --appVersion 2.0.0 date https://stappsbe01.innocampus.tu-berlin.de http://backend:3000 20

10
examples/minimal-deployment/systemd/minimal-connector.service Normal file
View File

@@ -0,0 +1,10 @@
[Unit]
Description=Restarts the "minimal-connector-0.23" container
[Service]
Type=oneshot
User=root
ExecStart=/usr/bin/env docker restart minimal-connector-0.23
[Install]
WantedBy=multi-user.target

8
examples/minimal-deployment/systemd/minimal-connector.timer Normal file
View File

@@ -0,0 +1,8 @@
[Unit]
Description=Run the minimal-connector.service every 5 minutes. If you need other timer configurations, please refer to: https://wiki.archlinux.org/index.php/Systemd/Timers
[Timer]
OnCalendar=*:0/5
[Install]
WantedBy=timers.target