backend (a reference implementation of a StApps backend)

This project is a reference implementation for a StApps backend. It provides an HTTP API to index data into a database, perform full text search, sorts and filters. It also delivers the configuration needed by the app. The API is specified within the @openstapps/core.

If you want to perform requests, index data or search within JavaScript or TypeScript you should consider using @openstapps/api

Usage

This backend is not a standalone software. It needs a database like Elasticsearch to work.

If you just want to use the backend you should consider using minimal-deployment. The minimal-deployment will provide you with everything you need to run this backend.

Local usage for development purposes

Requirements

• Elasticsearch (5.5)
• Node.js (~10) / NPM
• Docker

Generating Elasticsearch Mapping

The mappings will be generated automatically on the first start. If there are any errors, the backend will inform you and stop the execution, however it will do its best to complete the mappings. You can then either resolve these errors in the core-tools or the core, depending on where it originated. If you need a quick solution, you can also take the generated output file and manually correct the errors, then rename it to [coreVersion]_template_[type].json (replace any spaces with a _) and restart the backend (make sure that you don't have ES_FORCE_MAPPING_UPDATE set to true). This time it will take your file. The filenames and the path will also be displayed in the log of the backend.

Manually Resolving Errors

There are multiple types of errors the backend can run into. Manual error resolving requires you to be familiar with Elasticsearch mappings. An error will be represented in the output through an Elasticsearch type written in CAPS. Refer to either the console output or the [coreVersion]_error_report.txt for more info. If you feel lucky you can try to replace every error ("type": "MISSING_PREMAP", "type": "PARSE_ERROR", "type": "TYPE_CONFLICT") with

"dynamic": true,
"properties": {}

This should ONLY be used as a temporary workaround and might compromise other features.

Startup Behaviour

This might be important if you work on the Core

The backend is using the core-tools to automatically generate Elasticsearch Mappings and Aggregations from the current core version.

By default, the backend creates a local copy of the generated mappings and aggregations in src/storage/elasticsearch/templates/[coreVersion]_template_[type].json and src/storage/elasticsearch/templates/[coreVersion]_aggregations.json. On each start, it first checks if the aggregation file exists, this is because it does not know which of the types actually exist for the current core version. If the file does exist, it will just use the existing files and not generate a new mapping to cut down the time it takes to start the backend. When you are working on the Core, you might not want to have this behaviour, you can then either delete the generated file at each start or run the backend with the environment variable ES_FORCE_MAPPING_UPDATE=true. This will cause it to generate the mapping each time starts regardless of whether there are already files there.

Start Database (Elasticsearch)

Elasticsearch needs some configuration and plugins to be able to work with the backend. To save you some work we provide a docker image which only needs to be executed to work with the backend.

Run docker run -d -p 9200:9200 registry.gitlab.com/openstapps/database:master

Elasticsearch should be running at port 9200 now. If you have problems with getting elasticsearch to work, have a look in the README of the image first.

Start backend

Run npm install and npm run build, then start with npm start. The server should now be accepting connections at http://localhost:3000.

Environment Variables

To select a database implementation you have to set the NODE_CONFIG_ENV variable. At the time only NODE_CONFIG_ENV=elasticsearch is supported. Set NODE_ENV=production to run backend for production usages. In production the backend expects some kind of monitoring to be set via the environment. At the time only SMTP is being implemented. The backend wouldn't start if you don't provide SMTP authentification. Alternatively you can set ALLOW_NO_TRANSPORT=true. To set up an SMTP configuration have a look at @openstapps/logger.

The list of environment variables includes:

• NODE_ENV when set to production, there will be a reduced amount of output from the logger
• PORT when this is not set, the backend will default to port 3000
• ES_ADDR the Elasticsearch address, if not set it will default the Elasticsearch address to http://localhost:9200
• ES_FORCE_MAPPING_UPDATE when this variable is set to true, the backend will always generate a new Elasticsearch mapping from the core regardless of whether there is already a version present. This should only really be used when you are working on the core.
• ALLOW_NO_TRANSPORT if set to true, the backend will allow starting without an Email configured that receives critical errors.
• ES_DEBUG setting this to true will result in Elasticsearch logging to be VERY extensive, in almost all situation this should no be enabled.

Config files

Each university can have it's specific config for the general backend and app and for all databases.

All config files can be found in ./config/. There is a default.ts which is used by default. You can create an university specific file with following naming scheme: default-<university license plate>.ts

A university specific file will only overwrite all properties of the default.ts that are set in the file itself. To start the backend using your configuration you have to provide the NODE_APP_INSTANCE environment variable with your university license plate.

To set a database you have to provide the NODE_CONFIG_ENV environment variable with the name of the database. At the time only Elasticsearch is implemented.

To create your university specific config file for the elasticsearch you have to create a file with following naming scheme: elasticsearch-<university license plate>.ts.

Debugging

Set ES_DEBUG=true to enable verbose Elasticsearch tracing information. This can be useful to debug some issues between backend and elasticsearch.

Contributing