Theaninova/openstapps
1
0
Fork 0
mirror of https://gitlab.com/openstapps/openstapps.git synced 2026-01-22 01:22:54 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

refactor: use core supplied mappings

Browse Source
This commit is contained in:
Wieland Schöbl
2021-09-03 15:17:15 +00:00
committed by Rainer Killinger
parent 614a1b1e9b
commit 43a89ec4f2
22 changed files with 1622 additions and 1762 deletions
Download Patch File Download Diff File Expand all files Collapse all files

38
README.md
View File

@@ -4,8 +4,10 @@ This project is a reference implementation for a StApps backend. It provides an
perform full text search, sorts and filters. It also delivers the configuration needed by the app. The API is specified
within the [@openstapps/core](https://gitlab.com/openstapps/core).
If you want to perform requests, index data or search within JavaScript or TypeScript you should consider using
[@openstapps/api](https://gitlab.com/openstapps/api)
If you want to perform requests, index data or search within JavaScript or TypeScript you should consider using our client
[@openstapps/api](https://gitlab.com/openstapps/api).
Or generate your own client using the openapi/swagger definitions you can get form the [API documentation](https://openstapps.gitlab.io/backend).
# Usage
This backend is not a standalone software. It needs a database like Elasticsearch to work.
@@ -16,38 +18,15 @@ you with everything you need to run this backend.
# Local usage for development purposes
## Requirements
* Elasticsearch (5.5)
* Node.js (~10) / NPM
* Elasticsearch (5.6)
* Node.js (~14) / NPM
* Docker
## Generating Elasticsearch Mapping
The mappings will be generated automatically on the first start. If there are any errors, the backend will inform you and stop
the execution, however it will do its best to complete the mappings. You can then either resolve these errors in the `core-tools` or the `core`, depending on where it originated.
If you need a quick solution, you can also take the generated output file and manually correct the errors, then rename it to `[coreVersion]_template_[type].json` (replace any spaces with a `_`)
and restart the backend (make sure that you don't have `ES_FORCE_MAPPING_UPDATE` set to `true`). This time it will take your file. *The filenames and the path will also be displayed in the log of the backend.*
### Manually Resolving Errors
There are multiple types of errors the backend can run into. Manual error resolving requires you to be familiar with Elasticsearch
mappings.
An error will be represented in the output through an Elasticsearch type written in CAPS. Refer to either the console output
or the `[coreVersion]_error_report.txt` for more info. If you feel lucky you can try to replace every error (`"type": "MISSING_PREMAP"`,
`"type": "PARSE_ERROR"`, `"type": "TYPE_CONFLICT"`) with
```json
"dynamic": true,
"properties": {}
```
This should ONLY be used as a temporary workaround and might compromise other features.
### Startup Behaviour
*This might be important if you work on the Core*
The backend is using the `core-tools` to automatically generate Elasticsearch Mappings and Aggregations from the current `core` version.
By default, the backend creates a local copy of the generated mappings and aggregations in `src/storage/elasticsearch/templates/[coreVersion]_template_[type].json` and `src/storage/elasticsearch/templates/[coreVersion]_aggregations.json`.
On each start, it first checks if the aggregation file exists, this is because it does not know which of the types actually exist for the current core version. If the file does exist, it will just use the existing files and *not* generate a new mapping to cut down the time
it takes to start the backend. When you are working on the Core, you might not want to have this behaviour, you can then either delete
the generated file at each start or run the backend with the environment variable `ES_FORCE_MAPPING_UPDATE=true`. This will cause it to generate the mapping
each time starts regardless of whether there are already files there.
The backend is using Elasticsearch Mappings and Aggregations from its currently used `core` dependency.
## Start Database (Elasticsearch)
Elasticsearch needs some configuration and plugins to be able to work
@@ -95,7 +74,6 @@ The list of environment variables includes:
* `NODE_ENV` when set to `production`, there will be a reduced amount of output from the logger
* `PORT` when this is not set, the backend will default to port 3000
* `ES_ADDR` the Elasticsearch address, if not set it will default the Elasticsearch address to `http://localhost:9200`
* `ES_FORCE_MAPPING_UPDATE` when this variable is set to `true`, the backend will always generate a new Elasticsearch mapping from the core regardless of whether there is already a version present. This should only really be used when you are working on the core.
* `ALLOW_NO_TRANSPORT` if set to true, the backend will allow starting without an Email configured that receives critical errors.
* `ES_DEBUG` setting this to `true` will result in Elasticsearch logging to be **VERY** extensive, in almost all situation this should no be enabled.
* `PROMETHEUS_MIDDLEWARE` if set to `true` will enable metrics collection with [Express Prometheus Middleware](https://www.npmjs.com/package/express-prometheus-middleware)