Compare commits

...

55 Commits

Author SHA1 Message Date
e4b29cad73 feat: update components 2023-11-27 11:44:13 +01:00
8cfedd7aa1 refactor: rework components 2023-11-15 21:57:15 +01:00
10e3b21ad4 refactor: rework components 2023-11-15 20:56:52 +01:00
0a5cf19b8a feat: update components 2023-11-15 20:43:36 +01:00
4833155721 refactor: rework components 2023-11-15 19:19:25 +01:00
d3fe9a2f85 refactor: update to Angular 17 2023-11-08 15:44:00 +01:00
6dc01b538c feat: improve monorepo dev experience 2023-11-08 14:20:58 +01:00
fdec5a5baa feat: pipeline improvements 2023-11-07 18:51:55 +01:00
51602ffa0f feat: pipeline improvements 2023-11-07 18:00:17 +01:00
8a421cb2fb feat: generator updates 2023-11-07 17:14:58 +01:00
9ef77ab3ed feat: generator updates 2023-11-07 15:23:38 +01:00
4e181f881b feat: e2e connector 2023-11-07 13:55:38 +01:00
780916eb35 feat: copy connector 2023-11-07 12:48:21 +01:00
62d5ea4275 feat: more fixes 2023-11-01 16:10:12 +01:00
4bdd4b20d0 fix: build issues 2023-11-01 15:22:32 +01:00
c7555e1918 refactor: revert json schema changes 2023-11-01 14:51:06 +01:00
d18a579cb8 refactor: revert json schema changes 2023-11-01 14:44:45 +01:00
0de613969e feat: json-schema updates\nfeat: new route proposal 2023-11-01 14:31:12 +01:00
8466976b3c feat: improve monorepo dev experience 2023-10-30 17:31:40 +01:00
f65fb52def feat: improve monorepo dev experience 2023-10-27 22:46:07 +02:00
c6ab4ae48b feat: improve monorepo dev experience 2023-10-27 22:45:44 +02:00
f618725598 feat: full database clone feature in api-cli 2023-10-20 15:13:26 +02:00
theld
06b8ca109e feat: job portal
fix: disable function scoping lint rule
fix: outdated contributing docs for adding SCThings
2023-10-20 15:13:26 +02:00
f3ba8af051 fix: prevent duplicate publishes 2023-10-20 14:19:07 +02:00
2e0020b5c8 fix: non-publish branches fail on main 2023-10-20 14:15:22 +02:00
d17e8abea3 fix: docs fail to generate 2023-10-20 14:04:08 +02:00
ef092a8bff fix: library login fails 2023-10-17 18:00:55 +02:00
656f2266e3 feat: add scripts to enable login flow from localhost 2023-10-17 14:50:34 +02:00
Rainer Killinger
5c8c151917 docs: update changelogs for release
ci: publish release
2023-10-15 21:50:05 +02:00
Rainer Killinger
c020f075be fix: missing dependencies 2023-10-15 21:48:18 +02:00
98f21ac23b fix: distance disappears after 10 seconds 2023-10-15 16:41:52 +00:00
c460a3dbc0 fix: safari crashes with long opening hours change times 2023-10-15 16:28:42 +00:00
Rainer Killinger
5db3b7948a docs: update changelogs for release
ci: publish release
2023-10-13 18:22:48 +02:00
e7607360e2 feat: section dynamic slide buttons 2023-10-12 11:47:59 +02:00
1f62b5c5b0 feat: backend-supplied id cards on profile page
feat: SCIdCard thing
2023-10-12 11:47:59 +02:00
905ebf8c59 refactor: update capacitor build files 2023-10-12 11:14:28 +02:00
b210e0d9c3 fix: change leaflet tiles to be inverted as a whole 2023-10-12 10:18:18 +02:00
3a79d57c8d fix: detail view should not reuse things without references 2023-10-09 08:53:28 +00:00
9e6d4fe054 fix: week overview button broken on webkit 2023-10-06 17:57:02 +02:00
7b96500a75 fix: backend tests failing since the start of this semester 2023-10-06 15:36:00 +02:00
e0b7e616b3 fix: live update distance in list items 2023-09-21 09:38:52 +00:00
38fb7a398d fix: long branch names prevent review deployment 2023-09-21 09:55:35 +02:00
a99e08cd68 refactor: change opening hours handling
fix: opening hours not updating
feat: lazy-load opening hours module
feat: add e2e tests for opening hours
refactor: migrate opening hours to on-push change detection
feat: show exact minutes in opening hours starting one hour before next change
2023-09-21 09:55:35 +02:00
a5c9d22016 feat: add external directions referral
feat: change map page
feat: add error handling and timeout to location fetching in directions

resolves #124
resolves #122
2023-09-15 18:52:28 +00:00
3c49c4cf6d fix: route stack service causes needless network traffic 2023-09-06 15:29:47 +02:00
f2c4ee308f feat: share loaded data to detail views across routes 2023-09-06 14:52:02 +02:00
bd09b36620 refactor: use observable chains in rating component 2023-08-29 10:20:35 +00:00
ca146b7761 refactor: separate dashboard schedule nav e2e tests 2023-08-28 13:48:34 +02:00
Thea Schöbl
001f978bf9 feat: cleanup profile page 2023-08-25 14:43:25 +00:00
Thea Schöbl
57a5b6061b fix: type errors in easy-ast when generating docs 2023-08-23 08:45:47 +00:00
Rainer Killinger
4fb5941c56 ci: prepare e2e jobs for non nonexistent cache 2023-08-22 17:09:14 +02:00
Rainer Killinger
314e6a6e86 ci: move most pipelines to GitLab OSS large runner 2023-08-08 17:31:14 +02:00
Thea Schöbl
e1cc33bba2 refactor: rename "recurring" to "week overview"
refactor: replace dashboard calendar icon
resolves #128

refactor: route dashboard "next unit" to date series instead of events
resolves #126
2023-08-07 15:51:23 +00:00
Rainer Killinger
9abd397578 refactor: update changelog
ci: publish release
2023-08-03 16:30:40 +02:00
Rainer Killinger
69fe8c6ac8 ci: polish publishing via ci pipelines 2023-08-03 16:30:35 +02:00
688 changed files with 15505 additions and 12647 deletions

View File

@@ -1,5 +0,0 @@
---
'@openstapps/app': major
---
Update to Angular 16.1

View File

@@ -1,8 +0,0 @@
---
'@openstapps/core-tools': major
---
Removed pack tool
Use a bundler like `tsup` in conjunction with an
`index.ts` that aggregates all exports instead.

View File

@@ -1,11 +0,0 @@
---
'@openstapps/api-plugin': major
'@openstapps/api-cli': major
'@openstapps/api': major
---
Split API into API, API-CLI and API-Plugin
Plugins are now required to use `api-plugin`.
Consumers of `api` can benefit from a slimmer package with
no NodeJS dependencies.

View File

@@ -1,31 +0,0 @@
---
'@openstapps/backend': major
---
Migrate config system to cosmiconfig
Configs are now written using type-checked JavaScript
```js
// @ts-check
// This file is now type-safe just like TypeScript, but
// without the compilation step
/** @type {import('@openstapps/package').Type} */
const foo = {};
```
You can write config files in JavaScript, JSON, or other
cosmiconfig-supported formats.
The config files now are
- `backendrc.{js,json,...}`
- `elasticsearchrc.{js,json,...}`
- `prometheusrc.{js,json,...}`
You can also split them into multiple files,
as well as using the `markdown.js` helper in
`config/default/tools` to include markdown files
as text (see `config/f-u/about-pages`)

View File

@@ -0,0 +1,7 @@
---
'@openstapps/backend': minor
'@openstapps/core': minor
'@openstapps/app': minor
---
Add job portal feature

View File

@@ -1,51 +0,0 @@
---
'@openstapps/projectmanagement': major
'@openstapps/prettier-config': major
'@openstapps/es-mapping-generator': major
'@openstapps/backend-config': major
'@openstapps/eslint-config': major
'@openstapps/minimal-connector': major
'@openstapps/collection-utils': major
'@openstapps/minimal-plugin': major
'@openstapps/tsconfig': major
'@openstapps/api-plugin': major
'@openstapps/core-tools': major
'@openstapps/gitlab-api': major
'@openstapps/easy-ast': major
'@openstapps/api-cli': major
'@openstapps/backend': major
'@openstapps/logger': major
'@openstapps/proxy': major
'@openstapps/core': major
'@openstapps/app': major
'@openstapps/api': major
---
Move project to a turbo monorepo & pnpm
Internal dependencies are now defined using `"@openstapps/package": "workspace:*"`
- Removed extraneous files from packages
- `.npmrc`
- `.npmignore`
- `.mailmap`
- `.gitignore`
- `CONTRIBUTING.md`
- `LICENSE` (Project license file is added upon publishing, see [pnpm.io](https://pnpm.io/cli/publish))
- `package-lock.json`
- `.editorconfig`
- `.eslintrc.json` (moved eslint config to `package.json`)
- `.eslintignore`
- `.gitlab-ci.yml` (Most workflows are workspace-level)
- `.gitlab/**` (issue templates etc. are now workspace-level)
- `.dockerignore` (Docker files are determined by which files are deployed with `pnpm deploy`, as per `package.json/files`)
- TSConfig has been moved to its own package (You can now use `"extends": "@openstapps/tsconfig"`)
- Removed ESLint and Prettier peer dependency hell by injecting them through the `.pnpmfile.cjs`
- Added syncpack for keeping dependency versions in sync (and consistent key ordering in `package.json`)
- Replaced conventional changelog with changesets
- Apps with binaries now use a top level `app.js`
```js
#!/usr/bin/env node
import './lib/app.js';
```

View File

@@ -1,13 +0,0 @@
---
'@openstapps/core': major
---
Migrate away from [ts-optchain](https://www.npmjs.com/package/ts-optchain)
The package has been deprecated with the last releast being 4 years ago.
TypeScript as well as ECMAScript have native support for optional
chaining with the `?.` operator now.
You will need to update any packages that rely on the translator
module.

View File

@@ -1,25 +0,0 @@
---
'@openstapps/projectmanagement': patch
'@openstapps/prettier-config': patch
'@openstapps/es-mapping-generator': patch
'@openstapps/eslint-config': patch
'@openstapps/minimal-connector': patch
'@openstapps/minimal-plugin': patch
'@openstapps/core-tools': patch
'@openstapps/gitlab-api': patch
'@openstapps/backend': patch
'@openstapps/logger': patch
'@openstapps/proxy': patch
'@openstapps/core': patch
'@openstapps/api': patch
---
Migrated changelogs to changeset format
```js
import fs from 'fs';
const path = 'packages/logger/CHANGELOG.md';
fs.writeFileSync(path, fs.readFileSync(path, 'utf8').replace(/^#+\s+\[/gm, '## ['));
```

View File

@@ -1,8 +0,0 @@
---
'@openstapps/backend': minor
'@openstapps/api-cli': major
---
Migrate integration tests from docker-compose solution to a shell script
`api-cli` no longer builds as a Docker container as a result.

View File

@@ -0,0 +1,5 @@
---
'@openstapps/api-cli': minor
---
Add "\*" option to copy command that allows for a full database clone

View File

@@ -1,34 +0,0 @@
---
'@openstapps/projectmanagement': major
'@openstapps/prettier-config': major
'@openstapps/es-mapping-generator': major
'@openstapps/backend-config': major
'@openstapps/eslint-config': major
'@openstapps/minimal-connector': major
'@openstapps/collection-utils': major
'@openstapps/minimal-plugin': major
'@openstapps/tsconfig': major
'@openstapps/api-plugin': major
'@openstapps/core-tools': major
'@openstapps/gitlab-api': major
'@openstapps/easy-ast': major
'@openstapps/api-cli': major
'@openstapps/backend': major
'@openstapps/logger': major
'@openstapps/proxy': major
'@openstapps/core': major
'@openstapps/app': major
'@openstapps/api': major
---
Migrate to ESM
CommonJS is no longer supported in any capacity. To use the new
version, you will need to migrate your package to ESM.
We recommend using `tsup` and `Node 18`.
```json
{
"type": "module"
}
```

View File

@@ -1,30 +0,0 @@
---
'@openstapps/projectmanagement': major
'@openstapps/prettier-config': major
'@openstapps/es-mapping-generator': major
'@openstapps/backend-config': major
'@openstapps/eslint-config': major
'@openstapps/minimal-connector': major
'@openstapps/collection-utils': major
'@openstapps/minimal-plugin': major
'@openstapps/tsconfig': major
'@openstapps/api-plugin': major
'@openstapps/core-tools': major
'@openstapps/gitlab-api': major
'@openstapps/easy-ast': major
'@openstapps/api-cli': major
'@openstapps/backend': major
'@openstapps/logger': major
'@openstapps/proxy': major
'@openstapps/core': major
'@openstapps/app': major
'@openstapps/api': major
---
Migrate package to Node 18
- Consumers of this package will need to migrate to Node 18 or
higher.
- Packages have been migrated from promisified `readFile` or
`readFileSync` towards `fs/promises`
- Packages use native `flatMap` now

View File

@@ -1,5 +0,0 @@
---
'@openstapps/gitlab-api': minor
---
Migrate from request-promise-native to got

View File

@@ -1,5 +0,0 @@
---
'@openstapps/projectmanagement': patch
---
Moved project-docs to the [GitLab Wiki](https://gitlab.com/openstapps/openstapps/-/wikis/home) repo.

View File

@@ -1,31 +0,0 @@
---
'@openstapps/projectmanagement': minor
'@openstapps/prettier-config': minor
'@openstapps/es-mapping-generator': minor
'@openstapps/backend-config': minor
'@openstapps/eslint-config': minor
'@openstapps/minimal-connector': minor
'@openstapps/collection-utils': minor
'@openstapps/minimal-plugin': minor
'@openstapps/tsconfig': minor
'@openstapps/api-plugin': minor
'@openstapps/core-tools': minor
'@openstapps/gitlab-api': minor
'@openstapps/easy-ast': minor
'@openstapps/api-cli': minor
'@openstapps/backend': minor
'@openstapps/logger': minor
'@openstapps/proxy': minor
'@openstapps/core': minor
'@openstapps/app': minor
'@openstapps/api': minor
---
Migrate tests to C8/Chai/Mocha
- `@testdeck` OOP testing has been removed.
- Tests have been unified
- CommonJS module mocking has been replaced through
refactoring of tests, as ES Modules cannot be mocked
(do yourself a favor and don't try to mock them)
- C8 now replaces NYC as a native coverage tool

View File

@@ -1,5 +0,0 @@
---
'@openstapps/es-mapping-generator': patch
---
Remove @openstapps/logger dependency

View File

@@ -1,59 +0,0 @@
{
"mode": "pre",
"tag": "next",
"initialVersions": {
"@openstapps/backend": "2.0.0",
"@openstapps/database": "2.0.0",
"@openstapps/proxy": "2.0.0",
"@openstapps/backend-config": "2.0.0",
"@openstapps/eslint-config": "2.0.0",
"@openstapps/prettier-config": "2.0.0",
"@openstapps/projectmanagement": "2.0.0",
"@openstapps/tsconfig": "2.0.0",
"@openstapps/minimal-connector": "2.0.0",
"@openstapps/minimal-deployment": "2.0.0",
"@openstapps/minimal-plugin": "2.0.0",
"@openstapps/app": "2.0.0",
"@openstapps/app-release-template": "2.0.0",
"@openstapps/api": "2.0.0",
"@openstapps/api-cli": "2.0.0",
"@openstapps/api-plugin": "2.0.0",
"@openstapps/collection-utils": "2.0.0",
"@openstapps/core": "2.0.0",
"@openstapps/core-tools": "2.0.0",
"@openstapps/easy-ast": "2.0.0",
"@openstapps/es-mapping-generator": "2.0.0",
"@openstapps/gitlab-api": "2.0.0",
"@openstapps/logger": "2.0.0"
},
"changesets": [
"cool-jars-kiss",
"cuddly-bobcats-roll",
"dull-news-appear",
"five-ears-wash",
"fuzzy-walls-greet",
"giant-crabs-cheer",
"gorgeous-flowers-reflect",
"healthy-steaks-shop",
"honest-sheep-train",
"late-zoos-breathe",
"moody-parrots-develop",
"neat-hats-trade",
"new-pianos-joke",
"proud-wolves-end",
"quick-houses-count",
"rare-squids-bake",
"soft-donuts-fail",
"sour-coins-visit",
"spicy-snails-sort",
"stale-garlics-share",
"strange-hounds-repair",
"tall-ducks-dream",
"tame-mayflies-hug",
"tame-rings-dream",
"thick-weeks-compete",
"thin-camels-give",
"tidy-buses-reflect",
"tough-hairs-provide"
]
}

View File

@@ -1,8 +0,0 @@
---
'@openstapps/projectmanagement': major
'@openstapps/node-base': major
---
Move images to separate packages
Removed builder image due to migration to Kaniko

View File

@@ -1,9 +0,0 @@
---
'@openstapps/eslint-config': minor
---
Update rules
- `unicorn/no-array-reduce` is now off
- `unicorn/no-non-null-assertion` is now off
- Removed prettier rules (use prettier formatting instead)

View File

@@ -1,7 +0,0 @@
---
'@openstapps/es-mapping-generator': major
---
Remove put-es-templates
The `put-es-templates` functionality has been removed.

View File

@@ -1,24 +0,0 @@
---
'@openstapps/projectmanagement': patch
'@openstapps/prettier-config': patch
'@openstapps/es-mapping-generator': patch
'@openstapps/backend-config': patch
'@openstapps/eslint-config': patch
'@openstapps/minimal-connector': patch
'@openstapps/collection-utils': patch
'@openstapps/minimal-plugin': patch
'@openstapps/tsconfig': patch
'@openstapps/api-plugin': patch
'@openstapps/core-tools': patch
'@openstapps/gitlab-api': patch
'@openstapps/easy-ast': patch
'@openstapps/api-cli': patch
'@openstapps/backend': patch
'@openstapps/logger': patch
'@openstapps/proxy': patch
'@openstapps/core': patch
'@openstapps/app': patch
'@openstapps/api': patch
---
Migrate away from @openstapps/configuration

View File

@@ -1,5 +0,0 @@
---
'@openstapps/projectmanagement': patch
---
Update release configs

View File

@@ -1,30 +0,0 @@
---
'@openstapps/projectmanagement': patch
'@openstapps/prettier-config': patch
'@openstapps/app-release-template': patch
'@openstapps/es-mapping-generator': patch
'@openstapps/backend-config': patch
'@openstapps/eslint-config': patch
'@openstapps/minimal-deployment': patch
'@openstapps/minimal-connector': patch
'@openstapps/collection-utils': patch
'@openstapps/minimal-plugin': patch
'@openstapps/tsconfig': patch
'@openstapps/node-builder': patch
'@openstapps/api-plugin': patch
'@openstapps/core-tools': patch
'@openstapps/gitlab-api': patch
'@openstapps/app-builder-image': patch
'@openstapps/easy-ast': patch
'@openstapps/database': patch
'@openstapps/node-base': patch
'@openstapps/api-cli': patch
'@openstapps/backend': patch
'@openstapps/logger': patch
'@openstapps/proxy': patch
'@openstapps/core': patch
'@openstapps/app': patch
'@openstapps/api': patch
---
Update to TypeScript 5.1.6

View File

@@ -1,5 +0,0 @@
---
'@openstapps/app': patch
---
Migrate collection helpers to use @openstapps/collection-utils

View File

@@ -1,8 +0,0 @@
---
'@openstapps/app': minor
---
Migrate to Ionic 7
- Migrate uses of `<ion-label>` with inputs to new syntax
- Fix infinite loop in schedule date picker (`datetime.confirm()` to `datetime.cancel()`)

View File

@@ -1,7 +0,0 @@
---
'@openstapps/api': minor
---
Migrate HttpClient to Node 18's native `fetch` API
HttpClient can now be used both in NodeJS and browsers.

View File

@@ -1,7 +0,0 @@
---
'@openstapps/es-mapping-generator': major
'@openstapps/backend': major
'@openstapps/core': major
---
Migrate es mapping types from es-mapping-generator to .d.ts next to generated mappings

View File

@@ -1,5 +0,0 @@
---
'@openstapps/eslint-config': patch
---
Add license header enforcement rule (unfinished)

View File

@@ -1,5 +0,0 @@
---
'@openstapps/core': minor
---
Include openapi.json in built package

View File

@@ -1,26 +0,0 @@
---
'@openstapps/projectmanagement': minor
'@openstapps/minimal-connector': minor
'@openstapps/minimal-plugin': minor
'@openstapps/collection-utils': minor
'@openstapps/core-tools': minor
'@openstapps/gitlab-api': minor
'@openstapps/backend': minor
'@openstapps/logger': minor
'@openstapps/core': minor
'@openstapps/api': minor
---
Migrate away from `@krlwlfrt/async-pool`
```ts
import {mapAsyncLimit} from '@openstapps/collection-utils';
await mapAsyncLimit(
[1, 2, 3],
async it => {
await someNetworkRequest(it);
},
5,
);
```

View File

@@ -1,5 +0,0 @@
---
'@openstapps/app': minor
---
Migrate away from JIT compilation

View File

@@ -1,5 +0,0 @@
---
'@openstapps/logger': patch
---
Fix an issue with chalk terminal colors in tests

View File

@@ -1,9 +0,0 @@
---
'@openstapps/core-tools': major
---
Migrate openapi generation to output single file
The OpenAPI generator now outputs a single file instead
of a directory with the `openapi.json` and copied schema
files.

View File

@@ -1,6 +0,0 @@
---
'@openstapps/app': patch
---
Migrate unit tests to karma-coverage and junit reports.
Fixes an issue where coverage reports would not be generated.

View File

@@ -1,5 +0,0 @@
---
'@openstapps/projectmanagement': minor
---
Migrate moment.js to date-fns

View File

@@ -1,23 +0,0 @@
---
'@openstapps/projectmanagement': major
'@openstapps/es-mapping-generator': major
'@openstapps/minimal-connector': major
'@openstapps/collection-utils': major
'@openstapps/minimal-plugin': major
'@openstapps/api-plugin': major
'@openstapps/core-tools': major
'@openstapps/gitlab-api': major
'@openstapps/easy-ast': major
'@openstapps/api-cli': major
'@openstapps/backend': major
'@openstapps/logger': major
'@openstapps/proxy': major
'@openstapps/core': major
'@openstapps/api': major
---
Migrate build system to `tsup`
All packages now use an `index.ts` file to expose contents.
You will need to migrate paths from `import foo from '@scope/package/lib/foo` to `import foo from '@scope/package'`

View File

@@ -1,9 +0,0 @@
---
'@openstapps/core-tools': major
'@openstapps/easy-ast': major
---
Migrate easy-ast to separate package
The `easy-ast` part of `core-tools` has been moved to its own package.
For migration, simply use the new package.

View File

@@ -1,5 +0,0 @@
---
'@openstapps/core-tools': patch
---
Migrate from away from lodash

2
.gitignore vendored
View File

@@ -51,6 +51,8 @@ node_modules/
.pnpm-store/
jspm_packages/
.browser-data
# TypeScript v1 declaration files
typings/

View File

@@ -7,14 +7,18 @@
# ```
# To your pipeline.
# https://docs.gitlab.com/ee/ci/pipelines/merge_request_pipelines.html#use-rules-to-add-jobs
.limit_pipelines:
rules:
- if: $CI_PIPELINE_SOURCE == 'merge_request_event' || $CI_COMMIT_BRANCH == 'main' || $CI_COMMIT_BRANCH == 'develop'
when: on_success
- when: never
include:
- local: /backend/backend/.gitlab-ci.yml
- local: /frontend/app/.gitlab-ci.yml
- local: /.gitlab/schedules.gitlab-ci.yml
- local: /.gitlab/publishing.gitlab-ci.yml
rules:
- if: '$CI_COMMIT_MESSAGE =~ /ci: publish release$/'
when: always
variables:
TURBO_CACHE_BYPASS:
@@ -28,7 +32,7 @@ variables:
default:
image: registry.gitlab.com/openstapps/openstapps/node-builder
tags:
- performance
- saas-linux-xlarge-amd64
interruptible: true
before_script:
- corepack enable
@@ -76,13 +80,14 @@ build:
rules: &deploy-rules
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
variables:
DEPLOY_ID: $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
DEPLOY_ID: $CI_MERGE_REQUEST_IID
- if: $CI_COMMIT_BRANCH == 'main'
variables:
DEPLOY_ID: production
- if: $CI_COMMIT_BRANCH == 'develop'
variables:
DEPLOY_ID: staging
- !reference [.limit_pipelines, rules]
stop review:
stage: build
@@ -121,15 +126,15 @@ unit:
coverage_format: cobertura
path: coverage.xml
rules:
- if: $CI_PIPELINE_SOURCE == 'merge_request_event' || $CI_COMMIT_BRANCH == 'main' || $CI_COMMIT_BRANCH == 'develop'
- !reference [.limit_pipelines, rules]
audit:
stage: audit
allow_failure: true
needs: []
script:
- pnpm audit --prod
- pnpm audit --prod --audit-level critical
rules:
- if: $CI_PIPELINE_SOURCE == 'merge_request_event' || $CI_COMMIT_BRANCH == 'develop'
allow_failure: true
- if: $CI_COMMIT_BRANCH == 'main'
allow_failure: false
- !reference [.limit_pipelines, rules]

View File

@@ -1,6 +1,11 @@
.limit_publishing:
- if: $CI_PIPELINE_SOURCE != "schedule"
when: on_success
.limit_publish_pipelines:
rules:
- if: '$CI_COMMIT_BRANCH == "develop" && $CI_COMMIT_MESSAGE =~ /ci: publish prerelease/ && $CI_PIPELINE_SOURCE != "schedule"'
variables:
PUBLISH_TAG: next
- if: '$CI_COMMIT_BRANCH == "main" && $CI_COMMIT_MESSAGE =~ /ci: publish release/ && $CI_PIPELINE_SOURCE != "schedule"'
variables:
PUBLISH_TAG: latest
deploy:
stage: publish
@@ -12,7 +17,8 @@ deploy:
paths:
- ./.deploy
- ./frontend/app/www
rules: !reference [.limit_publishing]
rules:
- !reference [.limit_publish_pipelines, rules]
publish image:
stage: publish
@@ -29,8 +35,7 @@ publish image:
--context "${CI_PROJECT_DIR}/${DEPLOY_DIR}"
--dockerfile "${CI_PROJECT_DIR}/${DEPLOY_DIR}/Dockerfile"
--destination "${CI_REGISTRY_IMAGE}/${IMAGE_NAME}:$(grep -o '"version": "[^"]*' "${DEPLOY_DIR}/package.json" | cut -d'"' -f4)"
--destination "${CI_REGISTRY_IMAGE}/${IMAGE_NAME}:latest"
rules: !reference [.limit_publishing]
--destination "${CI_REGISTRY_IMAGE}/${IMAGE_NAME}:${PUBLISH_TAG}"
parallel:
matrix:
- IMAGE_NAME: database
@@ -47,6 +52,8 @@ publish image:
DEPLOY_DIR: .deploy/minimal-plugin
- IMAGE_NAME: app
DEPLOY_DIR: frontend/app
rules:
- !reference [.limit_publish_pipelines, rules]
publish packages:
stage: publish
@@ -57,15 +64,10 @@ publish packages:
script:
- pnpm install
- pnpm build
- pnpm publish -r --publish-branch ${PUBLISH_BRANCH} --tag ${PUBLISH_TAG} --no-git-checks # TODO: Git checks...
- pnpm config set '//registry.npmjs.org/:_authToken' "${NPM_AUTH_TOKEN}"
- pnpm publish -r --publish-branch ${CI_COMMIT_BRANCH} --tag ${PUBLISH_TAG} --no-git-checks # TODO: Git checks...
rules:
- if: $CI_COMMIT_BRANCH == 'main'
variables:
PUBLISH_BRANCH: main
PUBLISH_TAG: latest
- variables:
PUBLISH_BRANCH: $CI_COMMIT_BRANCH
PUBLISH_TAG: next
- !reference [.limit_publish_pipelines, rules]
publish docs:
stage: publish
@@ -77,3 +79,5 @@ publish docs:
artifacts:
paths:
- public
rules:
- !reference [.limit_publish_pipelines, rules]

View File

@@ -1,6 +1,6 @@
.limit_base_image_publishing:
- if: $CI_PIPELINE_SOURCE == "schedule" && $CI_COMMIT_BRANCH == 'main'
when: always
.limit_scheduled_pipelines:
rules:
- if: $CI_COMMIT_BRANCH == 'main' && $CI_PIPELINE_SOURCE == "schedule"
base image:
image: docker
@@ -18,10 +18,9 @@ base image:
docker build
-t "${CI_REGISTRY_IMAGE}/${IMAGE_NAME}:$(grep -o '"version": "[^"]*' "${DEPLOY_DIR}/package.json" | cut -d'"' -f4)"
-t "${CI_REGISTRY_IMAGE}/${IMAGE_NAME}:latest" "${CI_PROJECT_DIR}/${DEPLOY_DIR}" &&
docker push "${CI_REGISTRY_IMAGE}/${IMAGE_NAME}"
docker push "${CI_REGISTRY_IMAGE}/${IMAGE_NAME}" --all-tags
cache: {} # disable irrelevant cache for this job
before_script: [] # do not run irrelevant before script for this job
rules: !reference [.limit_base_image_publishing]
parallel:
matrix:
- IMAGE_NAME: node-base
@@ -30,4 +29,7 @@ base image:
DEPLOY_DIR: images/node-builder
- IMAGE_NAME: app-builder
DEPLOY_DIR: images/app-builder
- IMAGE_NAME: app-cypress
DEPLOY_DIR: images/app-cypress
rules:
- !reference [.limit_scheduled_pipelines, rules]

View File

@@ -1,7 +1,7 @@
{
"extension": ["ts"],
"extension": ["ts", "js"],
"node-option": ["loader=ts-node/esm"],
"reporter": "mocha-junit-reporter",
"reporter-option": ["mochaFile=coverage/report-junit.xml"],
"spec": ["test/**/*.spec.ts"]
"spec": ["test/**/*.spec.{ts,js}"]
}

View File

@@ -40,12 +40,6 @@ const config = {
packages: ['**'],
isIgnored: true,
},
{
label: 'ES Mapping Generator Special Dependencies',
dependencies: ['typescript', 'typedoc', 'ts-node', '@types/node', 'got'],
packages: ['@openstapps/es-mapping-generator'],
isIgnored: true,
},
{
label: 'Packages should use workspace version',
dependencies: ['@openstapps/**'],

View File

@@ -37,13 +37,13 @@ Adding new types requires changes at multiple locations for it to work correctly
- Add your SCThing and SCThingWithoutReferences to `src/things/your-thing-name.ts` and make them extend `SCThingWithoutReferences` and `SCThing` respectively
- Add your SCThingMeta to `src/things/your-thing-name.ts` and make it extend `SCThingMeta`
- Add your SCThingMeta to `SCClasses` in `src/meta.ts`
- Add your SCThing to `SCThingsWithoutDiff` in `src/meta.ts`
- Add your SCThing to `SCIndexableThings ` in `src/meta.ts`
- Add your SCThingWithoutReferences to `SCAssociatedThingWithoutReferences` in `src/meta.ts`
- Add your SCThing to `SCAssociatedThing` in `src/meta.ts`
- Add your SCThing to the `SCThingType` enum in `src/things/abstract/thing.ts`
- Add an example file for your SCThing in `test/resources/YourThingName.json`
- Add the following lines for your SCThing in `test/type.spec.ts`:
- Make sure your SCThing (but not SCThingWithoutReferences!) includes the `@indexable` and `@validatable` JSDoc annotations, otherwise neither JSON Schemas nor Elasticsearch mappings will be generated
```typescript
/**
* Types of properties of SCYourThingName

View File

@@ -19,4 +19,4 @@ integration:
junit:
- backend/backend/coverage/integration-report-junit.xml
rules:
- if: $CI_PIPELINE_SOURCE == 'merge_request_event' || $CI_COMMIT_BRANCH == 'main' || $CI_COMMIT_BRANCH == 'develop'
- !reference [.limit_pipelines, rules]

View File

@@ -1,5 +1,174 @@
# @openstapps/backend
## 3.0.1
### Patch Changes
- Fix missing dependency
## 3.0.0
### Major Changes
- 64caebaf: Migrate config system to cosmiconfig
Configs are now written using type-checked JavaScript
```js
// @ts-check
// This file is now type-safe just like TypeScript, but
// without the compilation step
/** @type {import('@openstapps/package').Type} */
const foo = {};
```
You can write config files in JavaScript, JSON, or other
cosmiconfig-supported formats.
The config files now are
- `backendrc.{js,json,...}`
- `elasticsearchrc.{js,json,...}`
- `prometheusrc.{js,json,...}`
You can also split them into multiple files,
as well as using the `markdown.js` helper in
`config/default/tools` to include markdown files
as text (see `config/f-u/about-pages`)
- 64caebaf: Move project to a turbo monorepo & pnpm
Internal dependencies are now defined using `"@openstapps/package": "workspace:*"`
- Removed extraneous files from packages
- `.npmrc`
- `.npmignore`
- `.mailmap`
- `.gitignore`
- `CONTRIBUTING.md`
- `LICENSE` (Project license file is added upon publishing, see [pnpm.io](https://pnpm.io/cli/publish))
- `package-lock.json`
- `.editorconfig`
- `.eslintrc.json` (moved eslint config to `package.json`)
- `.eslintignore`
- `.gitlab-ci.yml` (Most workflows are workspace-level)
- `.gitlab/**` (issue templates etc. are now workspace-level)
- `.dockerignore` (Docker files are determined by which files are deployed with `pnpm deploy`, as per `package.json/files`)
- TSConfig has been moved to its own package (You can now use `"extends": "@openstapps/tsconfig"`)
- Removed ESLint and Prettier peer dependency hell by injecting them through the `.pnpmfile.cjs`
- Added syncpack for keeping dependency versions in sync (and consistent key ordering in `package.json`)
- Replaced conventional changelog with changesets
- Apps with binaries now use a top level `app.js`
```js
#!/usr/bin/env node
import './lib/app.js';
```
- 64caebaf: Migrate to ESM
CommonJS is no longer supported in any capacity. To use the new
version, you will need to migrate your package to ESM.
We recommend using `tsup` and `Node 18`.
```json
{
"type": "module"
}
```
- 64caebaf: Migrate package to Node 18
- Consumers of this package will need to migrate to Node 18 or
higher.
- Packages have been migrated from promisified `readFile` or
`readFileSync` towards `fs/promises`
- Packages use native `flatMap` now
- 0a7e6af1: Migrate es mapping types from es-mapping-generator to .d.ts next to generated mappings
- 64caebaf: Migrate build system to `tsup`
All packages now use an `index.ts` file to expose contents.
You will need to migrate paths from `import foo from '@scope/package/lib/foo` to `import foo from '@scope/package'`
### Minor Changes
- 64caebaf: Migrate integration tests from docker-compose solution to a shell script
`api-cli` no longer builds as a Docker container as a result.
- 64caebaf: Migrate tests to C8/Chai/Mocha
- `@testdeck` OOP testing has been removed.
- Tests have been unified
- CommonJS module mocking has been replaced through
refactoring of tests, as ES Modules cannot be mocked
(do yourself a favor and don't try to mock them)
- C8 now replaces NYC as a native coverage tool
- 64caebaf: Migrate away from `@krlwlfrt/async-pool`
```ts
import {mapAsyncLimit} from '@openstapps/collection-utils';
await mapAsyncLimit(
[1, 2, 3],
async it => {
await someNetworkRequest(it);
},
5,
);
```
### Patch Changes
- 64caebaf: Migrated changelogs to changeset format
```js
import fs from 'fs';
const path = 'packages/logger/CHANGELOG.md';
fs.writeFileSync(path, fs.readFileSync(path, 'utf8').replace(/^#+\s+\[/gm, '## ['));
```
- 98546a97: Migrate away from @openstapps/configuration
- 23481d0d: Update to TypeScript 5.1.6
- Updated dependencies [98546a97]
- Updated dependencies [64caebaf]
- Updated dependencies [98546a97]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [1f62b5c5]
- Updated dependencies [98546a97]
- Updated dependencies [23481d0d]
- Updated dependencies [0a7e6af1]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [98546a97]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [98546a97]
- @openstapps/core-tools@3.0.0
- @openstapps/logger@3.0.0
- @openstapps/core@3.0.0
## 3.0.0-next.4
### Patch Changes
- 23481d0d: Update to TypeScript 5.1.6
- Updated dependencies [23481d0d]
- @openstapps/core-tools@3.0.0-next.4
- @openstapps/logger@3.0.0-next.4
- @openstapps/core@3.0.0-next.4
## 3.0.0-next.0
### Major Changes

View File

@@ -56,6 +56,19 @@ const menus = [
},
},
},
{
icon: 'work',
route: '/jobs',
title: 'job postings',
translations: {
de: {
title: 'Jobangebote',
},
en: {
title: 'job postings',
},
},
},
],
title: 'overview',
route: '/overview',

View File

@@ -3,7 +3,7 @@ import about from './about.js';
import imprint from './imprint.js';
import privacy from './privacy.js';
/** @type {import('@openstapps/core').SCMap<import('@openstapps/core').SCAboutPage>} */
/** @type {Record<string, import('@openstapps/core').SCAboutPage>} */
const aboutPages = {
'about': about,
'about/imprint': imprint,

View File

@@ -70,6 +70,10 @@ const config = {
authProvider: 'paia',
url: 'https://hds.hebis.de/paia/core',
},
/** TODO: idCards: {
authProvider: 'default',
url: 'TODO',
} */
},
},
aboutPages,

View File

@@ -5,18 +5,18 @@ if [ -z $GITLAB_CI ]; then
fi
( STAPPS_LOG_LEVEL=31 STAPPS_EXIT_LEVEL=8 NODE_CONFIG_ENV=elasticsearch NODE_ENV=integration-test ALLOW_NO_TRANSPORT=true ES_ADDR=http://$ES_HOST:9200 node app.js ) & backend_pid=$!
( openstapps-api e2e http://$BACKEND_HOST:3000 --reportPath coverage/integration-report-junit.xml --waiton tcp:$BACKEND_HOST:3000 --samples node_modules/@openstapps/core/test/resources/indexable ) & api_cli_pid=$!
( e2e-connector http://$BACKEND_HOST:3000 --reportPath coverage/integration-report-junit.xml --waiton tcp:$BACKEND_HOST:3000 --samples node_modules/@openstapps/core/test/resources/indexable ) & e2e_pid=$!
## Check output codes
# api-cli output defines passing the test
# backend should not exit early
wait $api_cli_pid
api_cli_exit=$?
wait $e2e_pid
e2e_exit=$?
wait $backend_pid
backend_exit=$?
if [ "$api_cli_exit" -eq "0" ]; then
if [ "$e2e_exit" -eq "0" ]; then
echo "FINISHED";
exit;
fi

View File

@@ -1,7 +1,7 @@
{
"name": "@openstapps/backend",
"description": "A reference implementation for a StApps backend",
"version": "3.0.0-next.0",
"version": "3.0.1",
"private": true,
"type": "module",
"license": "AGPL-3.0-only",
@@ -31,7 +31,6 @@
"build": "tsup-node",
"build:docker": "docker build -t openstapps:backend ../../.deploy/backend",
"deploy": "pnpm --prod --filter=@openstapps/backend deploy ../../.deploy/backend",
"dev": "tsup --watch --onSuccess \"pnpm run start\"",
"format": "prettier . -c --ignore-path ../../.gitignore",
"format:fix": "prettier --write . --ignore-path ../../.gitignore",
"lint": "tsc --noEmit && eslint --ext .ts src/",
@@ -43,9 +42,9 @@
"test:unit": "cross-env NODE_CONFIG_ENV=elasticsearch ALLOW_NO_TRANSPORT=true STAPPS_LOG_LEVEL=0 mocha --exit"
},
"dependencies": {
"@elastic/elasticsearch": "8.4.0",
"@elastic/elasticsearch": "8.10.0",
"@openstapps/core": "workspace:*",
"@openstapps/core-tools": "workspace:*",
"@openstapps/core-validator": "workspace:*",
"@openstapps/logger": "workspace:*",
"@types/body-parser": "1.19.2",
"@types/cors": "2.8.13",
@@ -56,9 +55,12 @@
"@types/nodemailer": "6.4.7",
"@types/promise-queue": "2.2.0",
"@types/uuid": "8.3.4",
"ajv": "8.12.0",
"ajv-formats": "2.1.1",
"body-parser": "1.20.2",
"cors": "2.8.5",
"cosmiconfig": "8.1.3",
"deepmerge": "4.3.1",
"express": "4.18.2",
"express-prom-bundle": "6.6.0",
"express-promise-router": "4.1.1",
@@ -74,7 +76,7 @@
"uuid": "8.3.2"
},
"devDependencies": {
"@openstapps/api-cli": "workspace:*",
"@openstapps/e2e-connector": "workspace:*",
"@openstapps/eslint-config": "workspace:*",
"@openstapps/prettier-config": "workspace:*",
"@openstapps/tsconfig": "workspace:*",
@@ -98,18 +100,8 @@
"sinon-express-mock": "2.2.1",
"supertest": "6.3.3",
"ts-node": "10.9.1",
"tsup": "6.7.0",
"typescript": "5.1.6"
},
"tsup": {
"entry": [
"src/cli.ts"
],
"sourcemap": true,
"clean": true,
"target": "es2022",
"format": "esm",
"outDir": "lib"
"tsup": "7.2.0",
"typescript": "5.2.2"
},
"prettier": "@openstapps/prettier-config",
"eslintConfig": {

View File

@@ -23,8 +23,7 @@ import {Logger} from '@openstapps/logger';
import cors from 'cors';
import {Express} from 'express';
import morgan from 'morgan';
import path from 'path';
import {DEFAULT_TIMEOUT, isTestEnvironment, mailer, plugins, validator} from './common.js';
import {DEFAULT_TIMEOUT, isTestEnvironment, mailer, plugins} from './common.js';
import {getPrometheusMiddleware} from './middleware/prometheus.js';
import {MailQueue} from './notification/mail-queue.js';
import {bulkAddRouter} from './routes/bulk-add-route.js';
@@ -39,7 +38,7 @@ import {virtualPluginRoute} from './routes/virtual-plugin-route.js';
import {BulkStorage} from './storage/bulk-storage.js';
import {DatabaseConstructor} from './storage/database.js';
import {backendConfig} from './config.js';
import {fileURLToPath} from 'url';
import {validator} from './validator.js';
/**
* Configure the backend
@@ -143,20 +142,8 @@ export async function configureApp(app: Express, databases: {[name: string]: Dat
request.on('data', chunkGatherer).on('end', endCallback);
});
// validate config file
const directory = path.dirname(fileURLToPath(import.meta.url));
await validator.addSchemas(
path.join(directory, '..', 'node_modules', '@openstapps', 'core', 'lib', 'schema'),
);
// validate the config file
const configValidation = validator.validate(backendConfig, 'SCConfigFile');
// validation failed
if (configValidation.errors.length > 0) {
throw new Error(
`Validation of config file failed. Errors were: ${JSON.stringify(configValidation.errors)}`,
);
if (!validator.validate(backendConfig, 'SCConfigFile')) {
throw new Error(`Validation of config file failed. Errors were: ${JSON.stringify(validator.errors)}`);
}
// check if a database name was given

View File

@@ -14,7 +14,6 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import {SCPluginMetaData} from '@openstapps/core';
import {Validator} from '@openstapps/core-tools';
import {BackendTransport} from './notification/backend-transport.js';
/**
@@ -22,11 +21,6 @@ import {BackendTransport} from './notification/backend-transport.js';
*/
export const mailer = BackendTransport.getTransportInstance();
/**
* A validator instance to check if something is a valid JSON object (e.g. a request or a thing)
*/
export const validator = new Validator();
/**
* Provides information if the backend is executed in the "test" (non-production) environment
*/

View File

@@ -19,12 +19,12 @@ import {
SCRoute,
SCValidationErrorResponse,
} from '@openstapps/core';
import {ValidationError} from '@openstapps/core-tools/src/types/validator.js';
import {Logger} from '@openstapps/logger';
import {Application, Router} from 'express';
import PromiseRouter from 'express-promise-router';
import {isTestEnvironment, validator} from '../common.js';
import {isTestEnvironment} from '../common.js';
import {isHttpMethod} from './http-types.js';
import {validator} from '../validator.js';
/**
* Creates a router from a route class and a handler function which implements the logic
@@ -56,11 +56,8 @@ export function createRoute<REQUESTTYPE, RETURNTYPE>(
// create a route handler for the given HTTP method
route[verb](async (request, response) => {
try {
// validate request
const requestValidation = validator.validate(request.body, routeClass.requestBodyName);
if (requestValidation.errors.length > 0) {
const error = new SCValidationErrorResponse(requestValidation.errors, isTestEnvironment);
if (!validator.validate(request.body, routeClass.requestBodyName as never)) {
const error = new SCValidationErrorResponse(validator.errors as unknown[], isTestEnvironment);
response.status(error.statusCode);
response.json(error);
await Logger.error(error);
@@ -68,17 +65,13 @@ export function createRoute<REQUESTTYPE, RETURNTYPE>(
return;
}
// hand over request to handler with path parameters
const handlerResponse = await handler(request.body, request.app, request.params);
const handlerResponse = await handler(request.body as REQUESTTYPE, request.app, request.params);
// validate response generated by handler
const responseErrors: ValidationError[] = validator.validate(
handlerResponse,
routeClass.responseBodyName,
).errors;
if (responseErrors.length > 0) {
const validationError = new SCValidationErrorResponse(responseErrors, isTestEnvironment);
if (!validator.validate(handlerResponse, routeClass.responseBodyName)) {
const validationError = new SCValidationErrorResponse(
validator.errors as unknown[],
isTestEnvironment,
);
// The validation error is not caused by faulty user input, but through an error that originates somewhere in
// the backend, therefore we use this "stacked" error.
const internalServerError = new SCInternalServerErrorResponse(validationError, isTestEnvironment);

View File

@@ -13,12 +13,12 @@
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import {SCInternalServerErrorResponse, SCPluginMetaData, SCValidationErrorResponse} from '@openstapps/core';
import {Request} from 'express';
import got from 'got';
import {isTestEnvironment, validator} from '../common.js';
import {isTestEnvironment} from '../common.js';
import {backendConfig} from '../config.js';
import {validator} from '../validator.js';
/**
* Generic route function used to proxy actual requests to plugins
@@ -28,10 +28,9 @@ import {backendConfig} from '../config.js';
*/
export async function virtualPluginRoute(request: Request, plugin: SCPluginMetaData): Promise<object> {
try {
const requestValidation = validator.validate(request.body, plugin.requestSchema);
if (requestValidation.errors.length > 0) {
if (!validator.validate(request.body, plugin.requestSchema)) {
// noinspection ExceptionCaughtLocallyJS
throw new SCValidationErrorResponse(requestValidation.errors, isTestEnvironment);
throw new SCValidationErrorResponse(validator.errors as unknown[], isTestEnvironment);
}
// send the request to the plugin (forward the body) and save the response
const response = await got.post(plugin.route.replaceAll(/^\//gi, ''), {
@@ -43,10 +42,9 @@ export async function virtualPluginRoute(request: Request, plugin: SCPluginMetaD
responseType: 'json',
});
const responseBody = response.body;
const responseValidation = validator.validate(responseBody, plugin.responseSchema);
if (responseValidation.errors.length > 0) {
if (!validator.validate(responseBody, plugin.responseSchema)) {
// noinspection ExceptionCaughtLocallyJS
throw new SCValidationErrorResponse(responseValidation.errors, isTestEnvironment);
throw new SCValidationErrorResponse(validator.errors as unknown[], isTestEnvironment);
}
return responseBody as object;
} catch (error) {

View File

@@ -31,7 +31,7 @@ import {parseAggregations} from './aggregations.js';
import * as Monitoring from './monitoring.js';
import {buildQuery} from './query/query.js';
import {buildSort} from './query/sort.js';
import {aggregations, putTemplate} from './templating.js';
import {putTemplate} from './templating.js';
import {
ElasticsearchConfig,
ElasticsearchQueryDisMaxConfig,
@@ -46,6 +46,7 @@ import {
} from './util/index.js';
import {noUndefined} from './util/no-undefined.js';
import {retryCatch, RetryOptions} from './util/retry.js';
import config from '@openstapps/core/elasticsearch.json' assert {type: 'json'};
/**
* A database interface for elasticsearch
@@ -370,7 +371,7 @@ export class Elasticsearch implements Database {
};
const response: SearchResponse<SCThings> = await this.client.search({
aggs: aggregations,
...config.search,
query: buildQuery(parameters, this.config, esConfig),
from: parameters.from,
index: ACTIVE_INDICES_ALIAS,

View File

@@ -15,19 +15,7 @@
*/
import {Client} from '@elastic/elasticsearch';
import {SCThingType} from '@openstapps/core';
import type {AggregationSchema} from '@openstapps/core/lib/mappings/aggregations.json.js';
import type {ElasticsearchTemplateCollection} from '@openstapps/core/lib/mappings/mappings.json.js';
import {readFileSync} from 'fs';
import path from 'path';
const mappingsPath = path.resolve('node_modules', '@openstapps', 'core', 'lib', 'mappings');
export const mappings = JSON.parse(
readFileSync(path.resolve(mappingsPath, 'mappings.json'), 'utf8'),
) as ElasticsearchTemplateCollection;
export const aggregations = JSON.parse(
readFileSync(path.resolve(mappingsPath, 'aggregations.json'), 'utf8'),
) as AggregationSchema;
import config from '@openstapps/core/elasticsearch.json' assert {type: 'json'};
/**
* Prepares all indices
@@ -40,7 +28,7 @@ export async function putTemplate(client: Client, type: SCThingType) {
const sanitizedType = `template_${type.replaceAll(/\s/g, '_')}`;
return client.indices.putTemplate({
body: mappings[sanitizedType],
body: config.mappings[sanitizedType],
name: sanitizedType,
});
}

View File

@@ -0,0 +1,3 @@
import {Validator} from '@openstapps/core-validator';
export const validator = new Validator();

View File

@@ -25,12 +25,12 @@ import bodyParser from 'body-parser';
import sinon from 'sinon';
import {expect} from 'chai';
import {Application} from 'express';
import {validator} from '../../src/common.js';
import {createRoute} from '../../src/routes/route.js';
import express, {Express} from 'express';
import supertest from 'supertest';
import {Logger} from '@openstapps/logger';
import {DEFAULT_TEST_TIMEOUT} from '../common.js';
import {validator} from '../../src/validator.js';
interface ReturnType {
foo: boolean;
@@ -78,8 +78,7 @@ describe('Create route', async function () {
it('should complain (throw an error) if used method is other than defined in the route creation', async function () {
const methodNotAllowedError = new SCMethodNotAllowedErrorResponse();
// @ts-expect-error not assignable
sandbox.stub(validator, 'validate').returns({errors: []});
sandbox.stub(validator, 'validate').returns(true);
let error: any = {};
sandbox.stub(Logger, 'warn').callsFake(error_ => {
error = error_;
@@ -97,8 +96,7 @@ describe('Create route', async function () {
});
it('should provide a route which returns handler response and success code', async function () {
// @ts-expect-error not assignable
sandbox.stub(validator, 'validate').returns({errors: []});
sandbox.stub(validator, 'validate').returns(true);
const router = createRoute<any, any>(routeClass, handler);
app.use(router);
@@ -115,8 +113,7 @@ describe('Create route', async function () {
app.use(router);
const startApp = supertest(app);
const validatorStub = sandbox.stub(validator, 'validate');
// @ts-expect-error not assignable
validatorStub.withArgs(body, routeClass.requestBodyName).returns({errors: [new Error('Foo Error')]});
validatorStub.withArgs(body, routeClass.requestBodyName).returns(false);
const response = await startApp
.post(routeClass.urlPath)
@@ -131,12 +128,8 @@ describe('Create route', async function () {
const router = createRoute<any, any>(routeClass, handler);
await app.use(router);
const startApp = supertest(app);
// @ts-expect-error not assignable
const validatorStub = sandbox.stub(validator, 'validate').returns({errors: []});
validatorStub
.withArgs(bodySuccess, routeClass.responseBodyName)
// @ts-expect-error not assignable
.returns({errors: [new Error('Foo Error')]});
const validatorStub = sandbox.stub(validator, 'validate').returns(false);
validatorStub.withArgs(bodySuccess, routeClass.responseBodyName).returns(false);
const response = await startApp.post(routeClass.urlPath).send();
@@ -177,8 +170,7 @@ describe('Create route', async function () {
await app.use(router);
const startApp = supertest(app);
// @ts-expect-error not assignable
sandbox.stub(validator, 'validate').returns({errors: []});
sandbox.stub(validator, 'validate').returns(false);
const response = await startApp.post(routeClass.urlPath).send();
@@ -213,8 +205,7 @@ describe('Create route', async function () {
await app.use(router);
const startApp = supertest(app);
// @ts-expect-error not assignable
sandbox.stub(validator, 'validate').returns({errors: []});
sandbox.stub(validator, 'validate').returns(false);
const response = await startApp.post(routeClass.urlPath).send();

View File

@@ -22,11 +22,12 @@ import got, {Options} from 'got';
import nock from 'nock';
import sinon from 'sinon';
import {mockReq} from 'sinon-express-mock';
import {plugins, validator} from '../../src/common.js';
import {plugins} from '../../src/common.js';
import {virtualPluginRoute} from '../../src/routes/virtual-plugin-route.js';
import {DEFAULT_TEST_TIMEOUT, FooError} from '../common.js';
import {registerAddRequest} from './plugin-register-route.spec.js';
import {testApp} from '../tests-setup.js';
import {validator} from '../../src/validator.js';
use(chaiAsPromised);
@@ -71,8 +72,7 @@ describe('Virtual plugin routes', async function () {
// spy the post method of got
// @ts-expect-error not assignable
const gotStub = sandbox.stub(got, 'post').returns({body: {}});
// @ts-expect-error not assignable
sandbox.stub(validator, 'validate').returns({errors: []});
sandbox.stub(validator, 'validate').returns(true);
const request_ = mockReq(request);
await virtualPluginRoute(request_, plugin);

View File

@@ -34,7 +34,6 @@ import mockedEnv from 'mocked-env';
import sinon, {SinonStub} from 'sinon';
import {removeInvalidAliasChars} from '../../../src/storage/elasticsearch/util/alias.js';
import {MailQueue} from '../../../src/notification/mail-queue.js';
import {aggregations} from '../../../src/storage/elasticsearch/templating.js';
import {Elasticsearch} from '../../../src/storage/elasticsearch/elasticsearch.js';
import {bulk, DEFAULT_TEST_TIMEOUT, getTransport, getIndex} from '../../common.js';
import fs from 'fs';
@@ -50,6 +49,7 @@ import {
} from '../../../src/storage/elasticsearch/util/index.js';
import cron from 'node-cron';
import {query} from './query.js';
import {search} from '@openstapps/core/elasticsearch.json' assert {type: 'json'};
use(chaiAsPromised);
@@ -131,7 +131,7 @@ describe('Elasticsearch', function () {
expect(indexUID.length).to.be.equal(INDEX_UID_LENGTH);
// test starting and ending character
expect(indexUID[0]).to.be.equal(bulk.uid[0]);
expect(indexUID[indexUID.length - 1]).to.be.equal(bulk.uid[INDEX_UID_LENGTH - 1]);
expect(indexUID.at(-1)).to.be.equal(bulk.uid[INDEX_UID_LENGTH - 1]);
});
it('should provide index name from the provided data', function () {
@@ -679,7 +679,7 @@ describe('Elasticsearch', function () {
await es.search(parameters);
expect(searchStub.firstCall.firstArg).to.be.deep.equal({
aggs: aggregations,
...search,
query,
allow_no_indices: true,
sort: [{'name.sort': 'desc'}],

View File

@@ -29,7 +29,7 @@ export const query: QueryDslQueryContainer = {
should: [],
},
},
weight: 1.1,
weight: 1.05,
},
{
filter: {
@@ -42,14 +42,14 @@ export const query: QueryDslQueryContainer = {
},
{
term: {
'academicTerms.acronym.raw': 'WS 2022/23',
'academicTerms.acronym.raw': 'WS 2023/24',
},
},
],
should: [],
},
},
weight: 1.05,
weight: 1.1,
},
{
filter: {
@@ -69,7 +69,7 @@ export const query: QueryDslQueryContainer = {
should: [],
},
},
weight: 1.1,
weight: 1.05,
},
{
filter: {
@@ -82,14 +82,14 @@ export const query: QueryDslQueryContainer = {
},
{
term: {
'academicTerms.acronym.raw': 'WiSe 2022/23',
'academicTerms.acronym.raw': 'WiSe 2023/24',
},
},
],
should: [],
},
},
weight: 1.05,
weight: 1.1,
},
{
filter: {

View File

@@ -0,0 +1,10 @@
import {defineConfig} from 'tsup';
export default defineConfig({
entry: ['src/cli.ts'],
sourcemap: true,
clean: true,
target: 'esnext',
format: 'esm',
outDir: 'lib',
});

View File

@@ -0,0 +1,256 @@
# @openstapps/minimal-connector
## 3.0.0
### Major Changes
- 64caebaf: Move project to a turbo monorepo & pnpm
Internal dependencies are now defined using `"@openstapps/package": "workspace:*"`
- Removed extraneous files from packages
- `.npmrc`
- `.npmignore`
- `.mailmap`
- `.gitignore`
- `CONTRIBUTING.md`
- `LICENSE` (Project license file is added upon publishing, see [pnpm.io](https://pnpm.io/cli/publish))
- `package-lock.json`
- `.editorconfig`
- `.eslintrc.json` (moved eslint config to `package.json`)
- `.eslintignore`
- `.gitlab-ci.yml` (Most workflows are workspace-level)
- `.gitlab/**` (issue templates etc. are now workspace-level)
- `.dockerignore` (Docker files are determined by which files are deployed with `pnpm deploy`, as per `package.json/files`)
- TSConfig has been moved to its own package (You can now use `"extends": "@openstapps/tsconfig"`)
- Removed ESLint and Prettier peer dependency hell by injecting them through the `.pnpmfile.cjs`
- Added syncpack for keeping dependency versions in sync (and consistent key ordering in `package.json`)
- Replaced conventional changelog with changesets
- Apps with binaries now use a top level `app.js`
```js
#!/usr/bin/env node
import './lib/app.js';
```
- 64caebaf: Migrate to ESM
CommonJS is no longer supported in any capacity. To use the new
version, you will need to migrate your package to ESM.
We recommend using `tsup` and `Node 18`.
```json
{
"type": "module"
}
```
- 64caebaf: Migrate package to Node 18
- Consumers of this package will need to migrate to Node 18 or
higher.
- Packages have been migrated from promisified `readFile` or
`readFileSync` towards `fs/promises`
- Packages use native `flatMap` now
- 64caebaf: Migrate build system to `tsup`
All packages now use an `index.ts` file to expose contents.
You will need to migrate paths from `import foo from '@scope/package/lib/foo` to `import foo from '@scope/package'`
### Minor Changes
- 64caebaf: Migrate tests to C8/Chai/Mocha
- `@testdeck` OOP testing has been removed.
- Tests have been unified
- CommonJS module mocking has been replaced through
refactoring of tests, as ES Modules cannot be mocked
(do yourself a favor and don't try to mock them)
- C8 now replaces NYC as a native coverage tool
- 64caebaf: Migrate away from `@krlwlfrt/async-pool`
```ts
import {mapAsyncLimit} from '@openstapps/collection-utils';
await mapAsyncLimit(
[1, 2, 3],
async it => {
await someNetworkRequest(it);
},
5,
);
```
### Patch Changes
- 64caebaf: Migrated changelogs to changeset format
```js
import fs from 'fs';
const path = 'packages/logger/CHANGELOG.md';
fs.writeFileSync(path, fs.readFileSync(path, 'utf8').replace(/^#+\s+\[/gm, '## ['));
```
- 98546a97: Migrate away from @openstapps/configuration
- 23481d0d: Update to TypeScript 5.1.6
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [98546a97]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [1f62b5c5]
- Updated dependencies [98546a97]
- Updated dependencies [23481d0d]
- Updated dependencies [64caebaf]
- Updated dependencies [0a7e6af1]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [98546a97]
- Updated dependencies [64caebaf]
- @openstapps/api@3.0.0
- @openstapps/logger@3.0.0
- @openstapps/core@3.0.0
## 3.0.0-next.4
### Patch Changes
- 23481d0d: Update to TypeScript 5.1.6
- Updated dependencies [23481d0d]
- @openstapps/logger@3.0.0-next.4
- @openstapps/core@3.0.0-next.4
- @openstapps/api@3.0.0-next.4
## 3.0.0-next.0
### Major Changes
- 64caebaf: Move project to a turbo monorepo & pnpm
Internal dependencies are now defined using `"@openstapps/package": "workspace:*"`
- Removed extraneous files from packages
- `.npmrc`
- `.npmignore`
- `.mailmap`
- `.gitignore`
- `CONTRIBUTING.md`
- `LICENSE` (Project license file is added upon publishing, see [pnpm.io](https://pnpm.io/cli/publish))
- `package-lock.json`
- `.editorconfig`
- `.eslintrc.json` (moved eslint config to `package.json`)
- `.eslintignore`
- `.gitlab-ci.yml` (Most workflows are workspace-level)
- `.gitlab/**` (issue templates etc. are now workspace-level)
- `.dockerignore` (Docker files are determined by which files are deployed with `pnpm deploy`, as per `package.json/files`)
- TSConfig has been moved to its own package (You can now use `"extends": "@openstapps/tsconfig"`)
- Removed ESLint and Prettier peer dependency hell by injecting them through the `.pnpmfile.cjs`
- Added syncpack for keeping dependency versions in sync (and consistent key ordering in `package.json`)
- Replaced conventional changelog with changesets
- Apps with binaries now use a top level `app.js`
```js
#!/usr/bin/env node
import './lib/app.js';
```
- 64caebaf: Migrate to ESM
CommonJS is no longer supported in any capacity. To use the new
version, you will need to migrate your package to ESM.
We recommend using `tsup` and `Node 18`.
```json
{
"type": "module"
}
```
- 64caebaf: Migrate package to Node 18
- Consumers of this package will need to migrate to Node 18 or
higher.
- Packages have been migrated from promisified `readFile` or
`readFileSync` towards `fs/promises`
- Packages use native `flatMap` now
- 64caebaf: Migrate build system to `tsup`
All packages now use an `index.ts` file to expose contents.
You will need to migrate paths from `import foo from '@scope/package/lib/foo` to `import foo from '@scope/package'`
### Minor Changes
- 64caebaf: Migrate tests to C8/Chai/Mocha
- `@testdeck` OOP testing has been removed.
- Tests have been unified
- CommonJS module mocking has been replaced through
refactoring of tests, as ES Modules cannot be mocked
(do yourself a favor and don't try to mock them)
- C8 now replaces NYC as a native coverage tool
- 64caebaf: Migrate away from `@krlwlfrt/async-pool`
```ts
import {mapAsyncLimit} from '@openstapps/collection-utils';
await mapAsyncLimit(
[1, 2, 3],
async it => {
await someNetworkRequest(it);
},
5,
);
```
### Patch Changes
- 64caebaf: Migrated changelogs to changeset format
```js
import fs from 'fs';
const path = 'packages/logger/CHANGELOG.md';
fs.writeFileSync(path, fs.readFileSync(path, 'utf8').replace(/^#+\s+\[/gm, '## ['));
```
- 98546a97: Migrate away from @openstapps/configuration
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [98546a97]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [98546a97]
- Updated dependencies [64caebaf]
- Updated dependencies [0a7e6af1]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [98546a97]
- Updated dependencies [64caebaf]
- @openstapps/api@3.0.0-next.0
- @openstapps/logger@3.0.0-next.0
- @openstapps/core@3.0.0-next.0
## [0.2.0](https://gitlab.com/openstapps/minimal-connector/compare/v0.1.0...v0.2.0) (2019-02-07)
## [0.1.0](https://gitlab.com/openstapps/minimal-connector/compare/v0.0.2...v0.1.0) (2019-01-30)
## [0.0.2](https://gitlab.com/openstapps/minimal-connector/compare/v0.0.1...v0.0.2) (2018-12-03)
## [0.0.1](https://gitlab.com/openstapps/minimal-connector/compare/d332f6e...v0.0.1) (2018-12-03)
### Features
- add minimal connector ([d332f6e](https://gitlab.com/openstapps/minimal-connector/commit/d332f6e))

View File

@@ -0,0 +1,5 @@
FROM registry.gitlab.com/openstapps/openstapps/node-base
ADD . .
ENTRYPOINT ["node", "lib/cli.js"]

View File

@@ -0,0 +1,3 @@
# @openstapps/copy-connector
A default connector that copies data from one backend to another

View File

@@ -0,0 +1,2 @@
#!/usr/bin/env node
import './lib/cli.js';

View File

@@ -0,0 +1,77 @@
{
"name": "@openstapps/copy-connector",
"description": "This is a minimal connector which serves as an example",
"version": "3.0.0",
"private": true,
"type": "module",
"license": "GPL-3.0-only",
"repository": "git@gitlab.com:openstapps/minimal-connector.git",
"author": "Anselm Stordeur <anselmstordeur@gmail.com>",
"contributors": [
"Jovan Krunić <jovan.krunic@gmail.com>",
"Karl-Philipp Wulfert <krlwlfrt@gmail.com>",
"Michel Jonathan Schmitz <michel.schmitz1992@gmail.com>",
"Rainer Killinger <git@killinger.co>"
],
"bin": "app.js",
"files": [
"app.js",
"lib",
"Dockerfile",
"README.md",
"CHANGELOG.md"
],
"scripts": {
"build": "tsup-node",
"deploy": "pnpm --prod --filter=@openstapps/copy-connector deploy ../../.deploy/copy-connector",
"format": "prettier . -c --ignore-path ../../.gitignore",
"format:fix": "prettier --write . --ignore-path ../../.gitignore",
"lint": "tsc --noEmit && eslint --ext .ts src/",
"lint:fix": "eslint --fix --ext .ts src/",
"test": "c8 --exclude src/cli.ts mocha --exit"
},
"dependencies": {
"@openstapps/api": "workspace:*",
"@openstapps/core": "workspace:*",
"@openstapps/logger": "workspace:*",
"@types/cli-progress": "3.11.5",
"cli-progress": "3.12.0",
"commander": "10.0.0"
},
"devDependencies": {
"@openstapps/eslint-config": "workspace:*",
"@openstapps/prettier-config": "workspace:*",
"@openstapps/tsconfig": "workspace:*",
"@types/chai": "4.3.5",
"@types/chai-as-promised": "7.1.5",
"@types/chai-spies": "1.0.6",
"@types/mocha": "10.0.1",
"@types/node": "18.15.3",
"c8": "7.14.0",
"chai": "4.3.7",
"chai-as-promised": "7.1.1",
"chai-spies": "1.1.0",
"conventional-changelog-cli": "2.2.2",
"mocha": "10.2.0",
"mocha-junit-reporter": "2.2.0",
"nock": "13.3.1",
"ts-node": "10.9.1",
"tsup": "7.2.0",
"typescript": "5.2.2"
},
"tsup": {
"entry": [
"src/cli.ts"
],
"sourcemap": true,
"clean": true,
"format": "esm",
"outDir": "lib"
},
"prettier": "@openstapps/prettier-config",
"eslintConfig": {
"extends": [
"@openstapps"
]
}
}

View File

@@ -0,0 +1,50 @@
/*
* Copyright (C) 2018, 2019 StApps
* This program is free software: you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the Free
* Software Foundation, version 3.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
* more details.
*
* You should have received a copy of the GNU General Public License along with
* this program. If not, see <https://www.gnu.org/licenses/>.
*/
import {Logger} from '@openstapps/logger';
import {Command} from 'commander';
import {version} from '../package.json';
import {copy} from './copy.js';
import {SCThingType, STAPPS_CORE_VERSION} from '@openstapps/core';
import {Client, ConnectorClient, HttpClient} from '@openstapps/api';
new Command()
.command('run <backendURL> <origin> <licensePlate>')
.argument('<backendUrl>', 'The URL of the StApps deployment', 'http://localhost:3000')
.argument(
'<foreignBackendUrl>',
'The URL of the backend to copy the data from',
'https://mobile.server.uni-frankfurt.de',
)
.argument('<types>', 'The type (RegExp full match)', '.*')
.argument('<batchSize>', 'Batch Size', 100)
.version(version)
.action(async (backendUrl: string, foreignBackendUrl: string, types: string, batchSize: number) => {
const client = new HttpClient();
const apiIn = new Client(client, foreignBackendUrl, STAPPS_CORE_VERSION);
const apiOut = new ConnectorClient(client, backendUrl);
const indexResponse = await apiIn.handshake(STAPPS_CORE_VERSION);
const origin = new URL(foreignBackendUrl).host.replaceAll(/[^a-zA-Z0-9-]/g, '-');
const licensePlate = indexResponse.backend.namespace;
const source = `${licensePlate}-${origin}`;
for (const type of Object.values(SCThingType)) {
if (!new RegExp(`^${types}$`).test(type)) continue;
await copy(apiIn, apiOut, source, type, batchSize);
}
Logger.ok('Done');
})
.addHelpCommand()
.parse(process.argv);

View File

@@ -14,60 +14,30 @@
*/
import {SCSearchRequest, SCThingType} from '@openstapps/core';
import {Bar} from 'cli-progress';
import {Client, ConnectorClient, OutOfRangeError, HttpClientInterface} from '@openstapps/api';
/**
* Options to set up copying data from one backend to another
*/
export interface CopyOptions {
/**
* Batch size to copy at once
*/
batchSize: number;
/**
* URL of the backend to copy from
*/
from: string;
/**
* Source identifier
*/
source: string;
/**
* URL of the backend to copy to
*/
to: string;
/**
* StAppsCore type to copy
*/
type: SCThingType;
/**
* StApps version identifier to copy data for
*/
version: string;
}
import {Client, ConnectorClient, OutOfRangeError} from '@openstapps/api';
/**
* Copy data for a StAppsCore type from one backend to another
* @param client HTTP client
* @param options Map of options
* @param apiIn The API for the backend to copy from
* @param apiOut The API for the backend to copy to
* @param source The source identifier for the bulk
* @param type The SCThingType to copy
* @param batchSize The batch size for the copy operation
*/
export async function copy(client: HttpClientInterface, options: CopyOptions): Promise<void> {
const apiIn = new Client(client, options.from, options.version);
const apiOut = new ConnectorClient(client, options.to);
// open a bulk
const bulk = await apiOut.bulk(options.type, options.source);
export async function copy(
apiIn: Client,
apiOut: ConnectorClient,
source: string,
type: SCThingType,
batchSize: number,
): Promise<void> {
const bulk = await apiOut.bulk(type, source);
let searchRequest: SCSearchRequest = {
filter: {
arguments: {
field: 'type',
value: options.type,
value: type,
},
type: 'value',
},
@@ -76,7 +46,7 @@ export async function copy(client: HttpClientInterface, options: CopyOptions): P
let searchResponse = await apiIn.search(searchRequest);
searchRequest.size = options.batchSize;
searchRequest.size = batchSize;
const progressBar = new Bar({});
progressBar.start(searchResponse.pagination.total, 0);

View File

@@ -27,12 +27,18 @@ import {
import chai from 'chai';
import chaiAsPromised from 'chai-as-promised';
import chaiSpies from 'chai-spies';
import {ApiError, HttpClient, HttpClientRequest, HttpClientResponse} from '@openstapps/api';
import {
ApiError,
HttpClient,
HttpClientRequest,
HttpClientResponse,
Client,
ConnectorClient,
} from '@openstapps/api';
import {copy} from '../src/copy.js';
/**
* Recursive Partial
*
* @see https://stackoverflow.com/a/51365037
*/
export type RecursivePartial<T> = {
@@ -134,14 +140,13 @@ describe('Copy', function () {
},
);
await copy(httpClient, {
batchSize: 5,
from: 'http://foo.bar',
source: 'stapps-copy',
to: 'http://localhost',
type: SCThingType.Dish,
version: 'foo.bar.foobar',
});
await copy(
new Client(httpClient, 'http://foo.bar'),
new ConnectorClient(httpClient, 'http://localhost'),
'stapps-copy',
SCThingType.Dish,
5,
);
});
it('should fail to copy', async function () {
@@ -216,13 +221,12 @@ describe('Copy', function () {
},
);
await copy(httpClient, {
batchSize: 5,
from: 'http://foo.bar',
source: 'stapps-copy',
to: 'http://localhost',
type: SCThingType.Dish,
version: 'foo.bar.foobar',
}).should.be.rejectedWith(ApiError);
await copy(
new Client(httpClient, 'http://foo.bar'),
new ConnectorClient(httpClient, 'http://localhost'),
'stapps-copy',
SCThingType.Dish,
5,
).should.be.rejectedWith(ApiError);
});
});

View File

@@ -0,0 +1,13 @@
# @openstapps/database
## 3.0.0
### Patch Changes
- 23481d0d: Update to TypeScript 5.1.6
## 3.0.0-next.4
### Patch Changes
- 23481d0d: Update to TypeScript 5.1.6

View File

@@ -1,10 +1,13 @@
{
"name": "@openstapps/database",
"version": "2.0.0",
"version": "3.0.0",
"private": true,
"files": [
"config",
"Dockerfile",
"README.md"
]
],
"scripts": {
"start": "docker run --rm -t -p 9200:9200 -p 9300:9300 $(docker build -q .)"
}
}

View File

@@ -0,0 +1,234 @@
# @openstapps/api-cli
## 3.0.0
### Major Changes
- 64caebaf: Split API into API, API-CLI and API-Plugin
Plugins are now required to use `api-plugin`.
Consumers of `api` can benefit from a slimmer package with
no NodeJS dependencies.
- 64caebaf: Move project to a turbo monorepo & pnpm
Internal dependencies are now defined using `"@openstapps/package": "workspace:*"`
- Removed extraneous files from packages
- `.npmrc`
- `.npmignore`
- `.mailmap`
- `.gitignore`
- `CONTRIBUTING.md`
- `LICENSE` (Project license file is added upon publishing, see [pnpm.io](https://pnpm.io/cli/publish))
- `package-lock.json`
- `.editorconfig`
- `.eslintrc.json` (moved eslint config to `package.json`)
- `.eslintignore`
- `.gitlab-ci.yml` (Most workflows are workspace-level)
- `.gitlab/**` (issue templates etc. are now workspace-level)
- `.dockerignore` (Docker files are determined by which files are deployed with `pnpm deploy`, as per `package.json/files`)
- TSConfig has been moved to its own package (You can now use `"extends": "@openstapps/tsconfig"`)
- Removed ESLint and Prettier peer dependency hell by injecting them through the `.pnpmfile.cjs`
- Added syncpack for keeping dependency versions in sync (and consistent key ordering in `package.json`)
- Replaced conventional changelog with changesets
- Apps with binaries now use a top level `app.js`
```js
#!/usr/bin/env node
import './lib/app.js';
```
- 64caebaf: Migrate integration tests from docker-compose solution to a shell script
`api-cli` no longer builds as a Docker container as a result.
- 64caebaf: Migrate to ESM
CommonJS is no longer supported in any capacity. To use the new
version, you will need to migrate your package to ESM.
We recommend using `tsup` and `Node 18`.
```json
{
"type": "module"
}
```
- 64caebaf: Migrate package to Node 18
- Consumers of this package will need to migrate to Node 18 or
higher.
- Packages have been migrated from promisified `readFile` or
`readFileSync` towards `fs/promises`
- Packages use native `flatMap` now
- 64caebaf: Migrate build system to `tsup`
All packages now use an `index.ts` file to expose contents.
You will need to migrate paths from `import foo from '@scope/package/lib/foo` to `import foo from '@scope/package'`
### Minor Changes
- 64caebaf: Migrate tests to C8/Chai/Mocha
- `@testdeck` OOP testing has been removed.
- Tests have been unified
- CommonJS module mocking has been replaced through
refactoring of tests, as ES Modules cannot be mocked
(do yourself a favor and don't try to mock them)
- C8 now replaces NYC as a native coverage tool
### Patch Changes
- 98546a97: Migrate away from @openstapps/configuration
- 23481d0d: Update to TypeScript 5.1.6
- Updated dependencies [98546a97]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [98546a97]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [1f62b5c5]
- Updated dependencies [64caebaf]
- Updated dependencies [98546a97]
- Updated dependencies [23481d0d]
- Updated dependencies [64caebaf]
- Updated dependencies [0a7e6af1]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [98546a97]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [98546a97]
- @openstapps/core-tools@3.0.0
- @openstapps/api@3.0.0
- @openstapps/eslint-config@3.0.0
- @openstapps/logger@3.0.0
- @openstapps/core@3.0.0
## 3.0.0-next.4
### Patch Changes
- 23481d0d: Update to TypeScript 5.1.6
- Updated dependencies [23481d0d]
- @openstapps/eslint-config@3.0.0-next.4
- @openstapps/core-tools@3.0.0-next.4
- @openstapps/logger@3.0.0-next.4
- @openstapps/core@3.0.0-next.4
- @openstapps/api@3.0.0-next.4
## 3.0.0-next.0
### Major Changes
- 64caebaf: Split API into API, API-CLI and API-Plugin
Plugins are now required to use `api-plugin`.
Consumers of `api` can benefit from a slimmer package with
no NodeJS dependencies.
- 64caebaf: Move project to a turbo monorepo & pnpm
Internal dependencies are now defined using `"@openstapps/package": "workspace:*"`
- Removed extraneous files from packages
- `.npmrc`
- `.npmignore`
- `.mailmap`
- `.gitignore`
- `CONTRIBUTING.md`
- `LICENSE` (Project license file is added upon publishing, see [pnpm.io](https://pnpm.io/cli/publish))
- `package-lock.json`
- `.editorconfig`
- `.eslintrc.json` (moved eslint config to `package.json`)
- `.eslintignore`
- `.gitlab-ci.yml` (Most workflows are workspace-level)
- `.gitlab/**` (issue templates etc. are now workspace-level)
- `.dockerignore` (Docker files are determined by which files are deployed with `pnpm deploy`, as per `package.json/files`)
- TSConfig has been moved to its own package (You can now use `"extends": "@openstapps/tsconfig"`)
- Removed ESLint and Prettier peer dependency hell by injecting them through the `.pnpmfile.cjs`
- Added syncpack for keeping dependency versions in sync (and consistent key ordering in `package.json`)
- Replaced conventional changelog with changesets
- Apps with binaries now use a top level `app.js`
```js
#!/usr/bin/env node
import './lib/app.js';
```
- 64caebaf: Migrate integration tests from docker-compose solution to a shell script
`api-cli` no longer builds as a Docker container as a result.
- 64caebaf: Migrate to ESM
CommonJS is no longer supported in any capacity. To use the new
version, you will need to migrate your package to ESM.
We recommend using `tsup` and `Node 18`.
```json
{
"type": "module"
}
```
- 64caebaf: Migrate package to Node 18
- Consumers of this package will need to migrate to Node 18 or
higher.
- Packages have been migrated from promisified `readFile` or
`readFileSync` towards `fs/promises`
- Packages use native `flatMap` now
- 64caebaf: Migrate build system to `tsup`
All packages now use an `index.ts` file to expose contents.
You will need to migrate paths from `import foo from '@scope/package/lib/foo` to `import foo from '@scope/package'`
### Minor Changes
- 64caebaf: Migrate tests to C8/Chai/Mocha
- `@testdeck` OOP testing has been removed.
- Tests have been unified
- CommonJS module mocking has been replaced through
refactoring of tests, as ES Modules cannot be mocked
(do yourself a favor and don't try to mock them)
- C8 now replaces NYC as a native coverage tool
### Patch Changes
- 98546a97: Migrate away from @openstapps/configuration
- Updated dependencies [98546a97]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [98546a97]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [98546a97]
- Updated dependencies [64caebaf]
- Updated dependencies [0a7e6af1]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [98546a97]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [98546a97]
- @openstapps/core-tools@3.0.0-next.0
- @openstapps/api@3.0.0-next.0
- @openstapps/eslint-config@3.0.0-next.0
- @openstapps/logger@3.0.0-next.0
- @openstapps/core@3.0.0-next.0

View File

@@ -20,6 +20,12 @@ npm run build
node ./lib/cli.js e2e http://localhost:3000
```
Example to clone the full database
```shell
node app.js copy "*" https://mobile.app.uni-frankfurt.de http://localhost:3000 100
```
### Program arguments
```shell

2
backend/e2e-connector/app.js Executable file
View File

@@ -0,0 +1,2 @@
#!/usr/bin/env node
import './lib/cli.js';

View File

@@ -1,7 +1,7 @@
{
"name": "@openstapps/api-cli",
"description": "CLI client for @openstapps/api",
"version": "3.0.0-next.0",
"name": "@openstapps/e2e-connector",
"description": "Connector for running e2e tests",
"version": "3.0.0",
"type": "module",
"license": "GPL-3.0-only",
"repository": "git@gitlab.com:openstapps/api.git",
@@ -15,9 +15,7 @@
"Roman Klopsch",
"Thea Schöbl <dev@theaninova.de>"
],
"bin": {
"openstapps-api": "app.js"
},
"bin": "app.js",
"files": [
"app.js",
"lib",
@@ -27,23 +25,21 @@
],
"scripts": {
"build": "tsup-node",
"deploy": "pnpm --prod --filter=@openstapps/api-cli deploy ../../.deploy/api-cli",
"deploy": "pnpm --prod --filter=@openstapps/e2e-connector deploy ../../.deploy/e2e-connector",
"format": "prettier . -c --ignore-path ../../.gitignore",
"format:fix": "prettier --write . --ignore-path ../../.gitignore",
"lint": "tsc --noEmit && eslint --ext .ts src/",
"lint:fix": "eslint --fix --ext .ts src/",
"test": "c8 mocha --exit"
"test": "c8 --exclude src/cli.ts mocha --exit"
},
"dependencies": {
"@openstapps/api": "workspace:*",
"@openstapps/core": "workspace:*",
"@openstapps/core-tools": "workspace:*",
"@openstapps/eslint-config": "workspace:*",
"@openstapps/logger": "workspace:*",
"@types/cli-progress": "3.11.0",
"@types/cli-progress": "3.11.5",
"@types/express": "4.17.17",
"@types/fs-extra": "9.0.13",
"@types/json-schema": "7.0.11",
"@types/json-schema": "7.0.14",
"@types/junit-report-builder": "3.0.0",
"@types/mocha": "10.0.1",
"@types/node": "18.15.3",
@@ -55,26 +51,26 @@
"wait-on": "6.0.1"
},
"devDependencies": {
"@openstapps/eslint-config": "workspace:*",
"@openstapps/prettier-config": "workspace:*",
"@openstapps/tsconfig": "workspace:*",
"@types/chai": "4.3.5",
"@types/chai-as-promised": "7.1.5",
"@types/chai-spies": "1.0.3",
"@types/chai-spies": "1.0.6",
"c8": "7.14.0",
"chai": "4.3.7",
"chai-as-promised": "7.1.1",
"chai-spies": "1.0.0",
"chai-spies": "1.1.0",
"mocha": "10.2.0",
"mocha-junit-reporter": "2.2.0",
"nock": "13.3.1",
"ts-node": "10.9.1",
"tsup": "6.7.0",
"typescript": "5.1.6"
"tsup": "7.2.0",
"typescript": "5.2.2"
},
"tsup": {
"entry": [
"src/app.ts",
"src/index.ts"
"src/cli.ts"
],
"sourcemap": true,
"clean": true,

View File

@@ -0,0 +1,59 @@
/*
* Copyright (C) 2018-2022 Open StApps
* This program is free software: you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the Free
* Software Foundation, version 3.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
* more details.
*
* You should have received a copy of the GNU General Public License along with
* this program. If not, see <https://www.gnu.org/licenses/>.
*/
import {Logger} from '@openstapps/logger';
import {Command} from 'commander';
import {URL} from 'url';
import waitOn from 'wait-on';
import {HttpClient} from '@openstapps/api';
import {version} from '../package.json';
import {endToEndRun} from './end-to-end.js';
new Command()
.command('<to>')
.argument('<to>', 'The backend to test', url => new URL(url).toString(), 'http://localhost:3000')
.version(version)
.description(
'Run in end to end test mode. Indexing and afterwards retrieving all test files from @openstapp/core to the backend',
)
.option(
'-s --samples path',
'Path to @openstapp/core test files',
'./node_modules/@openstapps/core/test/resources/indexable',
)
.option('-w --waiton [resource]', 'wait-on resource parameter see "www.npmjs.com/wait-on"')
.option('-r --reportPath [reportPath]', 'JUnit Report Path')
// eslint-disable-next-line unicorn/prevent-abbreviations
.action(async (to: string, e2eCommand: {samples: string; waiton?: string; reportPath?: string}) => {
try {
if (typeof e2eCommand.waiton === 'string') {
Logger.info(`Waiting for availibilty of resource: ${e2eCommand.waiton}`);
await waitOn({
resources: [e2eCommand.waiton],
timeout: 300_000,
});
Logger.info(`Resource became available`);
}
await endToEndRun(new HttpClient(), {
to,
samplesLocation: e2eCommand.samples,
reportLocation: e2eCommand.reportPath,
});
Logger.ok('Done');
} catch (error) {
await Logger.error(error);
}
})
.addHelpCommand()
.parse(process.argv);

View File

@@ -0,0 +1,20 @@
import junit from 'junit-report-builder';
import {deepStrictEqual} from 'assert';
import {Logger} from '@openstapps/logger';
import {TestState} from './test-state.js';
/**
* Compares all samples (local and remote) with the same uid and throws if they're not deep equal
*/
export async function compareItems(state: TestState, suite: junit.TestSuite) {
for (const localThing of state.localItems.values()) {
await state.runTest(suite, `Should be the same for ${localThing.type} (${localThing.uid})`, async () => {
if (!state.remoteItems.has(localThing.uid)) {
throw new Error(`Did not retrieve expected SCThing with uid: ${localThing.uid}`);
}
const remoteThing = state.remoteItems.get(localThing.uid);
deepStrictEqual(remoteThing, localThing, `Unexpected difference between original and retrieved sample`);
});
}
Logger.info(`All samples retrieved from the backend have been compared`);
}

View File

@@ -0,0 +1,48 @@
/*
* Copyright (C) 2019-2022 Open StApps
* This program is free software: you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the Free
* Software Foundation, version 3.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
* more details.
*
* You should have received a copy of the GNU General Public License along with
* this program. If not, see <https://www.gnu.org/licenses/>.
*/
import {Logger} from '@openstapps/logger';
import {HttpClientInterface} from '@openstapps/api';
import junit from 'junit-report-builder';
import {indexSamples} from './index-samples.js';
import {retrieveItems} from './retreive-items.js';
import {compareItems} from './compare-items.js';
import {E2EOptions, TestState} from './test-state.js';
/**
* Function that can be used for integration tests.
* Adds all the SCThings that getItemsFromSamples() returns to the backend.
* Afterward, retrieves the items from backend and checks for differences with original ones.
*/
export async function endToEndRun(client: HttpClientInterface, options: E2EOptions): Promise<string[]> {
const state = new TestState(client, options);
const builder = junit.newBuilder();
await indexSamples(state, builder.testSuite().name('e2e index'));
Logger.info(`All samples have been indexed via the backend`);
await retrieveItems(state, builder.testSuite().name('e2e retrieve'));
Logger.info(`All samples have been retrieved from the backend`);
await compareItems(state, builder.testSuite().name('e2e compare'));
if (options.reportLocation) {
builder.writeTo(options.reportLocation);
}
await (state.errors.length > 0
? Logger.error(`\n${state.errors.length} failed test cases`)
: Logger.ok('All tests passed.'));
return state.errors;
}

View File

@@ -0,0 +1,29 @@
import {SCThings} from '@openstapps/core';
import {readdir, readFile} from 'fs/promises';
import path from 'path';
/**
* Get all SCThings from the predefined core test json files
* @param samplesDirectory Filepath to the directory containing to the core test json files
* @returns an Array of all the SCThings specified for test usage
*/
export async function getItemsFromSamples<T extends SCThings>(samplesDirectory: string): Promise<T[]> {
const things: T[] = [];
try {
const fileNames = await readdir(samplesDirectory);
for (const fileName of fileNames) {
const filePath = path.join(samplesDirectory, fileName);
if (filePath.endsWith('.json')) {
const fileContent = await readFile(filePath, {encoding: 'utf8'});
const schemaObject = JSON.parse(fileContent);
if (schemaObject.errorNames.length === 0 && typeof schemaObject.instance.type === 'string') {
things.push(schemaObject.instance);
}
}
}
} catch (error) {
throw error;
}
return things;
}

View File

@@ -0,0 +1,33 @@
import junit from 'junit-report-builder';
import {SCThings, SCThingType} from '@openstapps/core';
import {getItemsFromSamples} from './get-items-from-samples.js';
import {TestState} from './test-state.js';
/**
* Function to add all the SCThings that getItemsFromSamples() returns to the backend
*/
export async function indexSamples(state: TestState, suite: junit.TestSuite): Promise<void> {
const items = await getItemsFromSamples(state.options.samplesLocation);
if (items.length === 0) {
throw new Error('Could not index samples. None were retrieved from the file system.');
}
// sort items by type
const itemMap: Map<SCThingType, SCThings[]> = new Map();
for (const item of items) {
if (!itemMap.has(item.type)) {
itemMap.set(item.type, []);
}
const itemsOfSameType = itemMap.get(item.type) as SCThings[];
itemsOfSameType.push(item);
itemMap.set(item.type, itemsOfSameType);
state.localItems.set(item.uid, item);
}
// add items depending on their type property with one type per bulk
for (const type of itemMap.keys()) {
await state.runTest(suite, `Should index ${type}`, async () =>
state.api.index(itemMap.get(type) as SCThings[], 'stapps-core-sample-data'),
);
}
}

View File

@@ -0,0 +1,30 @@
import junit from 'junit-report-builder';
import {SCSearchRequest} from '@openstapps/core';
import {TestState} from './test-state.js';
/**
* Retrieves all samples previously index using the api
*/
export async function retrieveItems(state: TestState, suite: junit.TestSuite): Promise<void> {
const singleItemSearchRequest: SCSearchRequest = {
filter: {
arguments: {
field: 'uid',
value: 'replace-me',
},
type: 'value',
},
};
for (const {uid, type} of state.localItems.values()) {
await state.runTest(suite, `Should find ${type} (${uid})`, async () => {
singleItemSearchRequest.filter!.arguments.value = uid;
const searchResponse = await state.api.search(singleItemSearchRequest);
if (searchResponse.data.length !== 1) {
throw new Error(
`Search for single SCThing with uid: ${uid} returned ${searchResponse.data.length} results`,
);
}
state.remoteItems.set(uid, searchResponse.data[0]);
});
}
}

View File

@@ -0,0 +1,66 @@
import {SCThings} from '@openstapps/core';
import {ConnectorClient, HttpClientInterface} from '@openstapps/api';
import {Logger} from '@openstapps/logger';
import junit from 'junit-report-builder';
export class TestState {
localItems = new Map<string, SCThings>();
remoteItems = new Map<string, SCThings>();
errors: string[] = [];
api: ConnectorClient;
constructor(client: HttpClientInterface, readonly options: E2EOptions) {
this.api = new ConnectorClient(client, options.to);
}
/**
* Run a test
* @param suite The suite the test belongs to
* @param name The name of the test
* @param scope The scope in which the test is run
*/
async runTest(suite: junit.TestSuite, name: string, scope: () => Promise<void>) {
const testCase = suite.testCase().name(name);
process.stdout.addListener('data', testCase.standardOutput);
process.stderr.addListener('data', testCase.standardError);
const start = performance.now();
try {
await scope();
// eslint-disable-next-line @typescript-eslint/no-explicit-any
} catch (error: any) {
await Logger.error(error);
testCase.failure(error.message);
this.errors.push(error.message);
}
process.stdout.removeListener('data', testCase.standardOutput);
process.stderr.removeListener('data', testCase.standardError);
const end = performance.now();
testCase.time((end - start) / 1000);
}
}
/**
* Options to set up indexing core test files to backend
*/
export interface E2EOptions {
/**
* File path of the directory containing core test files
*/
samplesLocation: string;
/**
* URL of the backend to index to
*/
to: string;
/**
* Location of the report
*/
reportLocation?: string;
}

View File

@@ -30,12 +30,12 @@ import chaiSpies from 'chai-spies';
import {existsSync, mkdirSync, rmdirSync, unlinkSync} from 'fs';
import {createFileSync} from 'fs-extra';
import {HttpClient, HttpClientRequest, HttpClientResponse} from '@openstapps/api';
import {RecursivePartial} from './copy.spec.js';
import {expect} from 'chai';
import path from 'path';
import {fileURLToPath} from 'url';
// eslint-disable-next-line unicorn/prevent-abbreviations
import {e2eRun, getItemsFromSamples} from '../src/e2e.js';
import {endToEndRun} from '../src/end-to-end.js';
import {getItemsFromSamples} from '../src/get-items-from-samples.js';
import {Logger} from '@openstapps/logger';
chai.should();
chai.use(chaiSpies);
@@ -54,6 +54,13 @@ const httpClient = new HttpClient();
const storedThings: Map<string, SCThings> = new Map();
describe('e2e Connector', function () {
beforeEach(function () {
sandbox.on(Logger, 'error', (...parameters) => {
// eslint-disable-next-line no-console
console.log('e2e:', ...parameters);
});
});
afterEach(function () {
sandbox.restore();
});
@@ -75,84 +82,74 @@ describe('e2e Connector', function () {
let failOnCompare = false;
let failOnLookup = false;
sandbox.on(
httpClient,
'request',
async (request: HttpClientRequest): Promise<RecursivePartial<responses>> => {
if (request.url.toString() === `http://localhost${bulkRoute.getUrlPath().toString()}`) {
return {
body: {
state: 'in progress',
uid: 'foo',
},
statusCode: bulkRoute.statusCodeSuccess,
};
}
sandbox.on(httpClient, 'request', async (request: HttpClientRequest): Promise<Partial<responses>> => {
if (request.url.toString() === `http://localhost${bulkRoute.getUrlPath().toString()}`) {
return {
body: {
state: 'in progress',
uid: 'foo',
},
statusCode: bulkRoute.statusCodeSuccess,
};
}
if (
request.url.toString() === `http://localhost${bulkAddRoute.getUrlPath({UID: 'foo'}).toString()}`
) {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
storedThings.set((request.body as any).uid, JSON.parse(JSON.stringify(request.body)));
return {
body: {},
statusCode: bulkAddRoute.statusCodeSuccess,
};
}
if (
request.url.toString() === `http://localhost${bulkDoneRoute.getUrlPath({UID: 'foo'}).toString()}`
) {
return {
body: {},
statusCode: bulkDoneRoute.statusCodeSuccess,
};
}
if (request.url.toString() === `http://localhost${searchRoute.getUrlPath().toString()}`) {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const thing = storedThings.get((request.body as any).filter.arguments.value);
if (failOnCompare) {
thing!.origin!.modified = 'altered';
}
const returnThing = failOnLookup ? [] : [thing];
const returnBody = {
data: returnThing,
facets: [],
pagination: {
count: returnThing.length,
offset: 0,
total: returnThing.length,
},
stats: {
time: 42,
},
};
return {
body: returnBody,
statusCode: searchRoute.statusCodeSuccess,
};
}
if (request.url.toString() === `http://localhost${bulkAddRoute.getUrlPath({UID: 'foo'}).toString()}`) {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
storedThings.set((request.body as any).uid, JSON.parse(JSON.stringify(request.body)));
return {
body: {},
statusCode: bulkAddRoute.statusCodeSuccess,
};
}
if (request.url.toString() === `http://localhost${bulkDoneRoute.getUrlPath({UID: 'foo'}).toString()}`) {
return {
body: {},
statusCode: bulkDoneRoute.statusCodeSuccess,
};
}
if (request.url.toString() === `http://localhost${searchRoute.getUrlPath().toString()}`) {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const thing = storedThings.get((request.body as any).filter.arguments.value);
if (failOnCompare) {
thing!.origin!.modified = 'altered';
}
const returnThing = failOnLookup ? [] : [thing];
const returnBody = {
data: returnThing,
facets: [],
pagination: {
count: returnThing.length,
offset: 0,
total: returnThing.length,
},
stats: {
time: 42,
},
};
return {
body: returnBody,
statusCode: searchRoute.statusCodeSuccess,
};
},
);
}
// tslint:disable-next-line: max-line-length
await e2eRun(httpClient, {
return {
body: {},
statusCode: searchRoute.statusCodeSuccess,
};
});
await endToEndRun(httpClient, {
to: 'http://localhost',
samplesLocation: './node_modules/@openstapps/core/test/resources',
}).should.eventually.have.length(0);
failOnLookup = true;
failOnCompare = false;
// tslint:disable-next-line: max-line-length
await e2eRun(httpClient, {
await endToEndRun(httpClient, {
to: 'http://localhost',
samplesLocation: './node_modules/@openstapps/core/test/resources',
}).should.eventually.include(
@@ -161,8 +158,7 @@ describe('e2e Connector', function () {
failOnLookup = false;
failOnCompare = true;
// tslint:disable-next-line: max-line-length
await e2eRun(httpClient, {
await endToEndRun(httpClient, {
to: 'http://localhost',
samplesLocation: './node_modules/@openstapps/core/test/resources',
}).should.eventually.include('Unexpected difference between original and retrieved sample');
@@ -171,15 +167,14 @@ describe('e2e Connector', function () {
it('should fail to index', async function () {
type responses = HttpClientResponse<SCBulkAddResponse | SCBulkDoneResponse | SCBulkResponse>;
sandbox.on(httpClient, 'request', async (): Promise<RecursivePartial<responses>> => {
sandbox.on(httpClient, 'request', async (): Promise<Partial<responses>> => {
return {
body: {},
statusCode: Number.MAX_SAFE_INTEGER,
};
});
// tslint:disable-next-line: max-line-length
return e2eRun(httpClient, {
return endToEndRun(httpClient, {
to: 'http://localhost',
samplesLocation: './node_modules/@openstapps/core/test/resources',
}).should.eventually.include('');
@@ -190,7 +185,7 @@ describe('e2e Connector', function () {
if (!existsSync(emptyDirectoryPath)) {
mkdirSync(emptyDirectoryPath);
}
await e2eRun(httpClient, {
await endToEndRun(httpClient, {
to: 'http://localhost',
samplesLocation: emptyDirectoryPath,
}).should.be.rejectedWith('Could not index samples. None were retrieved from the file system.');
@@ -207,7 +202,7 @@ describe('e2e Connector', function () {
}
const nonJsonFile = path.join(somewhatFilledDirectoryPath, 'nonjson.txt');
createFileSync(nonJsonFile);
await e2eRun(httpClient, {
await endToEndRun(httpClient, {
to: 'http://localhost',
samplesLocation: somewhatFilledDirectoryPath,
}).should.be.rejectedWith('Could not index samples. None were retrieved from the file system.');

View File

@@ -1,5 +1,109 @@
# @openstapps/proxy
## 3.0.0
### Major Changes
- 64caebaf: Move project to a turbo monorepo & pnpm
Internal dependencies are now defined using `"@openstapps/package": "workspace:*"`
- Removed extraneous files from packages
- `.npmrc`
- `.npmignore`
- `.mailmap`
- `.gitignore`
- `CONTRIBUTING.md`
- `LICENSE` (Project license file is added upon publishing, see [pnpm.io](https://pnpm.io/cli/publish))
- `package-lock.json`
- `.editorconfig`
- `.eslintrc.json` (moved eslint config to `package.json`)
- `.eslintignore`
- `.gitlab-ci.yml` (Most workflows are workspace-level)
- `.gitlab/**` (issue templates etc. are now workspace-level)
- `.dockerignore` (Docker files are determined by which files are deployed with `pnpm deploy`, as per `package.json/files`)
- TSConfig has been moved to its own package (You can now use `"extends": "@openstapps/tsconfig"`)
- Removed ESLint and Prettier peer dependency hell by injecting them through the `.pnpmfile.cjs`
- Added syncpack for keeping dependency versions in sync (and consistent key ordering in `package.json`)
- Replaced conventional changelog with changesets
- Apps with binaries now use a top level `app.js`
```js
#!/usr/bin/env node
import './lib/app.js';
```
- 64caebaf: Migrate to ESM
CommonJS is no longer supported in any capacity. To use the new
version, you will need to migrate your package to ESM.
We recommend using `tsup` and `Node 18`.
```json
{
"type": "module"
}
```
- 64caebaf: Migrate package to Node 18
- Consumers of this package will need to migrate to Node 18 or
higher.
- Packages have been migrated from promisified `readFile` or
`readFileSync` towards `fs/promises`
- Packages use native `flatMap` now
- 64caebaf: Migrate build system to `tsup`
All packages now use an `index.ts` file to expose contents.
You will need to migrate paths from `import foo from '@scope/package/lib/foo` to `import foo from '@scope/package'`
### Minor Changes
- 64caebaf: Migrate tests to C8/Chai/Mocha
- `@testdeck` OOP testing has been removed.
- Tests have been unified
- CommonJS module mocking has been replaced through
refactoring of tests, as ES Modules cannot be mocked
(do yourself a favor and don't try to mock them)
- C8 now replaces NYC as a native coverage tool
### Patch Changes
- 64caebaf: Migrated changelogs to changeset format
```js
import fs from 'fs';
const path = 'packages/logger/CHANGELOG.md';
fs.writeFileSync(path, fs.readFileSync(path, 'utf8').replace(/^#+\s+\[/gm, '## ['));
```
- 98546a97: Migrate away from @openstapps/configuration
- 23481d0d: Update to TypeScript 5.1.6
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [98546a97]
- Updated dependencies [23481d0d]
- Updated dependencies [64caebaf]
- Updated dependencies [98546a97]
- Updated dependencies [64caebaf]
- @openstapps/logger@3.0.0
## 3.0.0-next.4
### Patch Changes
- 23481d0d: Update to TypeScript 5.1.6
- Updated dependencies [23481d0d]
- @openstapps/logger@3.0.0-next.4
## 3.0.0-next.0
### Major Changes

View File

@@ -1,7 +1,7 @@
{
"name": "@openstapps/proxy",
"description": "NGINX proxy that is dynamically configured by a Node.js script",
"version": "3.0.0-next.0",
"version": "3.0.0",
"private": true,
"type": "module",
"license": "AGPL-3.0-only",
@@ -51,10 +51,9 @@
"is-cidr": "4.0.2",
"mustache": "4.2.0",
"semver": "7.3.8",
"typescript": "5.1.6"
"typescript": "5.2.2"
},
"devDependencies": {
"@openstapps/api-cli": "workspace:*",
"@openstapps/eslint-config": "workspace:*",
"@openstapps/prettier-config": "workspace:*",
"@openstapps/tsconfig": "workspace:*",
@@ -76,7 +75,7 @@
"sinon": "15.0.4",
"sinon-chai": "3.7.0",
"ts-node": "10.9.1",
"tsup": "6.7.0"
"tsup": "7.2.0"
},
"tsup": {
"entry": [

View File

@@ -1,5 +1,80 @@
# @openstapps/backend-config
## 3.0.0
### Major Changes
- 64caebaf: Move project to a turbo monorepo & pnpm
Internal dependencies are now defined using `"@openstapps/package": "workspace:*"`
- Removed extraneous files from packages
- `.npmrc`
- `.npmignore`
- `.mailmap`
- `.gitignore`
- `CONTRIBUTING.md`
- `LICENSE` (Project license file is added upon publishing, see [pnpm.io](https://pnpm.io/cli/publish))
- `package-lock.json`
- `.editorconfig`
- `.eslintrc.json` (moved eslint config to `package.json`)
- `.eslintignore`
- `.gitlab-ci.yml` (Most workflows are workspace-level)
- `.gitlab/**` (issue templates etc. are now workspace-level)
- `.dockerignore` (Docker files are determined by which files are deployed with `pnpm deploy`, as per `package.json/files`)
- TSConfig has been moved to its own package (You can now use `"extends": "@openstapps/tsconfig"`)
- Removed ESLint and Prettier peer dependency hell by injecting them through the `.pnpmfile.cjs`
- Added syncpack for keeping dependency versions in sync (and consistent key ordering in `package.json`)
- Replaced conventional changelog with changesets
- Apps with binaries now use a top level `app.js`
```js
#!/usr/bin/env node
import './lib/app.js';
```
- 64caebaf: Migrate to ESM
CommonJS is no longer supported in any capacity. To use the new
version, you will need to migrate your package to ESM.
We recommend using `tsup` and `Node 18`.
```json
{
"type": "module"
}
```
- 64caebaf: Migrate package to Node 18
- Consumers of this package will need to migrate to Node 18 or
higher.
- Packages have been migrated from promisified `readFile` or
`readFileSync` towards `fs/promises`
- Packages use native `flatMap` now
### Minor Changes
- 64caebaf: Migrate tests to C8/Chai/Mocha
- `@testdeck` OOP testing has been removed.
- Tests have been unified
- CommonJS module mocking has been replaced through
refactoring of tests, as ES Modules cannot be mocked
(do yourself a favor and don't try to mock them)
- C8 now replaces NYC as a native coverage tool
### Patch Changes
- 98546a97: Migrate away from @openstapps/configuration
- 23481d0d: Update to TypeScript 5.1.6
## 3.0.0-next.4
### Patch Changes
- 23481d0d: Update to TypeScript 5.1.6
## 3.0.0-next.0
### Major Changes

View File

@@ -1,7 +1,7 @@
{
"name": "@openstapps/backend-config",
"description": "Backend Configuration for OpenStApps",
"version": "3.0.0-next.0",
"version": "3.0.0",
"private": true,
"type": "module",
"license": "GPL-3.0-only",

View File

@@ -5,6 +5,7 @@
<item title="search" title.de="Suche" icon="search" route="/search"/>
<item title="library catalog" title.de="Bibliothekskatalog" icon="local_library" route="/hebis-search"/>
<item title="course catalog" title.de="Vorlesungsverzeichnis" icon="inventory_2" route="/catalog"/>
<item title="job postings" title.de="Jobangebote" icon="work" route="/jobs"/>
</group>
<group title="canteen" title.de="Mensa" icon="local_cafe" route="/canteen"/>
<group title="campus map" title.de="Campus Karte" icon="map" route="/map"/>

View File

@@ -5,6 +5,7 @@
<item title="search" title.de="Suche" icon="search" route="/search"/>
<item title="library catalog" title.de="Bibliothekskatalog" icon="local_library" route="/hebis-search"/>
<item title="course catalog" title.de="Vorlesungsverzeichnis" icon="inventory_2" route="/catalog"/>
<item title="job postings" title.de="Jobangebote" icon="work" route="/jobs"/>
</group>
<group title="canteen" title.de="Mensa" icon="local_cafe" route="/canteen"/>
<group title="campus map" title.de="Campus Karte" icon="map" route="/map"/>

View File

@@ -1,5 +1,97 @@
# @openstapps/eslint-config
## 3.0.0
### Major Changes
- 64caebaf: Move project to a turbo monorepo & pnpm
Internal dependencies are now defined using `"@openstapps/package": "workspace:*"`
- Removed extraneous files from packages
- `.npmrc`
- `.npmignore`
- `.mailmap`
- `.gitignore`
- `CONTRIBUTING.md`
- `LICENSE` (Project license file is added upon publishing, see [pnpm.io](https://pnpm.io/cli/publish))
- `package-lock.json`
- `.editorconfig`
- `.eslintrc.json` (moved eslint config to `package.json`)
- `.eslintignore`
- `.gitlab-ci.yml` (Most workflows are workspace-level)
- `.gitlab/**` (issue templates etc. are now workspace-level)
- `.dockerignore` (Docker files are determined by which files are deployed with `pnpm deploy`, as per `package.json/files`)
- TSConfig has been moved to its own package (You can now use `"extends": "@openstapps/tsconfig"`)
- Removed ESLint and Prettier peer dependency hell by injecting them through the `.pnpmfile.cjs`
- Added syncpack for keeping dependency versions in sync (and consistent key ordering in `package.json`)
- Replaced conventional changelog with changesets
- Apps with binaries now use a top level `app.js`
```js
#!/usr/bin/env node
import './lib/app.js';
```
- 64caebaf: Migrate to ESM
CommonJS is no longer supported in any capacity. To use the new
version, you will need to migrate your package to ESM.
We recommend using `tsup` and `Node 18`.
```json
{
"type": "module"
}
```
- 64caebaf: Migrate package to Node 18
- Consumers of this package will need to migrate to Node 18 or
higher.
- Packages have been migrated from promisified `readFile` or
`readFileSync` towards `fs/promises`
- Packages use native `flatMap` now
### Minor Changes
- 64caebaf: Migrate tests to C8/Chai/Mocha
- `@testdeck` OOP testing has been removed.
- Tests have been unified
- CommonJS module mocking has been replaced through
refactoring of tests, as ES Modules cannot be mocked
(do yourself a favor and don't try to mock them)
- C8 now replaces NYC as a native coverage tool
- 64caebaf: Update rules
- `unicorn/no-array-reduce` is now off
- `unicorn/no-non-null-assertion` is now off
- Removed prettier rules (use prettier formatting instead)
### Patch Changes
- 64caebaf: Migrated changelogs to changeset format
```js
import fs from 'fs';
const path = 'packages/logger/CHANGELOG.md';
fs.writeFileSync(path, fs.readFileSync(path, 'utf8').replace(/^#+\s+\[/gm, '## ['));
```
- 98546a97: Migrate away from @openstapps/configuration
- 23481d0d: Update to TypeScript 5.1.6
- 64caebaf: Add license header enforcement rule (unfinished)
## 3.0.0-next.4
### Patch Changes
- 23481d0d: Update to TypeScript 5.1.6
## 3.0.0-next.0
### Major Changes

View File

@@ -1,7 +1,7 @@
{
"name": "@openstapps/eslint-config",
"description": "A collection of configuration base files for StApps projects. Just an (unused) experiment for now.",
"version": "3.0.0-next.0",
"version": "3.0.0",
"type": "commonjs",
"license": "GPL-3.0-only",
"repository": "git@gitlab.com:openstapps/eslint-config.git",
@@ -18,16 +18,16 @@
"devDependencies": {
"@openstapps/tsconfig": "workspace:*",
"@types/node": "18.15.3",
"eslint": "8.43.0",
"typescript": "5.1.6"
"eslint": "8.53.0",
"typescript": "5.2.2"
},
"peerDependencies": {
"@typescript-eslint/eslint-plugin": "5.60.1",
"@typescript-eslint/parser": "5.60.1",
"eslint": "8.43.0",
"eslint-config-prettier": "8.8.0",
"eslint-plugin-jsdoc": "46.4.2",
"eslint-plugin-prettier": "4.2.1",
"eslint-plugin-unicorn": "47.0.0"
"@typescript-eslint/eslint-plugin": "6.10.0",
"@typescript-eslint/parser": "6.10.0",
"eslint": "8.53.0",
"eslint-config-prettier": "9.0.0",
"eslint-plugin-jsdoc": "46.8.2",
"eslint-plugin-prettier": "5.0.1",
"eslint-plugin-unicorn": "49.0.0"
}
}

View File

@@ -1,5 +1,90 @@
# @openstapps/prettier-config
## 3.0.0
### Major Changes
- 64caebaf: Move project to a turbo monorepo & pnpm
Internal dependencies are now defined using `"@openstapps/package": "workspace:*"`
- Removed extraneous files from packages
- `.npmrc`
- `.npmignore`
- `.mailmap`
- `.gitignore`
- `CONTRIBUTING.md`
- `LICENSE` (Project license file is added upon publishing, see [pnpm.io](https://pnpm.io/cli/publish))
- `package-lock.json`
- `.editorconfig`
- `.eslintrc.json` (moved eslint config to `package.json`)
- `.eslintignore`
- `.gitlab-ci.yml` (Most workflows are workspace-level)
- `.gitlab/**` (issue templates etc. are now workspace-level)
- `.dockerignore` (Docker files are determined by which files are deployed with `pnpm deploy`, as per `package.json/files`)
- TSConfig has been moved to its own package (You can now use `"extends": "@openstapps/tsconfig"`)
- Removed ESLint and Prettier peer dependency hell by injecting them through the `.pnpmfile.cjs`
- Added syncpack for keeping dependency versions in sync (and consistent key ordering in `package.json`)
- Replaced conventional changelog with changesets
- Apps with binaries now use a top level `app.js`
```js
#!/usr/bin/env node
import './lib/app.js';
```
- 64caebaf: Migrate to ESM
CommonJS is no longer supported in any capacity. To use the new
version, you will need to migrate your package to ESM.
We recommend using `tsup` and `Node 18`.
```json
{
"type": "module"
}
```
- 64caebaf: Migrate package to Node 18
- Consumers of this package will need to migrate to Node 18 or
higher.
- Packages have been migrated from promisified `readFile` or
`readFileSync` towards `fs/promises`
- Packages use native `flatMap` now
### Minor Changes
- 64caebaf: Migrate tests to C8/Chai/Mocha
- `@testdeck` OOP testing has been removed.
- Tests have been unified
- CommonJS module mocking has been replaced through
refactoring of tests, as ES Modules cannot be mocked
(do yourself a favor and don't try to mock them)
- C8 now replaces NYC as a native coverage tool
### Patch Changes
- 64caebaf: Migrated changelogs to changeset format
```js
import fs from 'fs';
const path = 'packages/logger/CHANGELOG.md';
fs.writeFileSync(path, fs.readFileSync(path, 'utf8').replace(/^#+\s+\[/gm, '## ['));
```
- 98546a97: Migrate away from @openstapps/configuration
- 23481d0d: Update to TypeScript 5.1.6
## 3.0.0-next.4
### Patch Changes
- 23481d0d: Update to TypeScript 5.1.6
## 3.0.0-next.0
### Major Changes

View File

@@ -1,7 +1,7 @@
{
"name": "@openstapps/prettier-config",
"description": "StApps Prettier Config",
"version": "3.0.0-next.0",
"version": "3.0.0",
"type": "module",
"license": "GPL-3.0-only",
"repository": "git@gitlab.com:openstapps/prettier-config.git",
@@ -19,6 +19,6 @@
"test": "npx prettier --config index.json --check \"test/*.js\""
},
"peerDependencies": {
"prettier": "2.8.6"
"prettier": "3.1.0"
}
}

View File

@@ -1,5 +1,143 @@
# @openstapps/projectmanagement
## 3.0.0
### Major Changes
- 64caebaf: Move project to a turbo monorepo & pnpm
Internal dependencies are now defined using `"@openstapps/package": "workspace:*"`
- Removed extraneous files from packages
- `.npmrc`
- `.npmignore`
- `.mailmap`
- `.gitignore`
- `CONTRIBUTING.md`
- `LICENSE` (Project license file is added upon publishing, see [pnpm.io](https://pnpm.io/cli/publish))
- `package-lock.json`
- `.editorconfig`
- `.eslintrc.json` (moved eslint config to `package.json`)
- `.eslintignore`
- `.gitlab-ci.yml` (Most workflows are workspace-level)
- `.gitlab/**` (issue templates etc. are now workspace-level)
- `.dockerignore` (Docker files are determined by which files are deployed with `pnpm deploy`, as per `package.json/files`)
- TSConfig has been moved to its own package (You can now use `"extends": "@openstapps/tsconfig"`)
- Removed ESLint and Prettier peer dependency hell by injecting them through the `.pnpmfile.cjs`
- Added syncpack for keeping dependency versions in sync (and consistent key ordering in `package.json`)
- Replaced conventional changelog with changesets
- Apps with binaries now use a top level `app.js`
```js
#!/usr/bin/env node
import './lib/app.js';
```
- 64caebaf: Migrate to ESM
CommonJS is no longer supported in any capacity. To use the new
version, you will need to migrate your package to ESM.
We recommend using `tsup` and `Node 18`.
```json
{
"type": "module"
}
```
- 64caebaf: Migrate package to Node 18
- Consumers of this package will need to migrate to Node 18 or
higher.
- Packages have been migrated from promisified `readFile` or
`readFileSync` towards `fs/promises`
- Packages use native `flatMap` now
- 11c9d742: Move images to separate packages
Removed builder image due to migration to Kaniko
- 64caebaf: Migrate build system to `tsup`
All packages now use an `index.ts` file to expose contents.
You will need to migrate paths from `import foo from '@scope/package/lib/foo` to `import foo from '@scope/package'`
### Minor Changes
- 64caebaf: Migrate tests to C8/Chai/Mocha
- `@testdeck` OOP testing has been removed.
- Tests have been unified
- CommonJS module mocking has been replaced through
refactoring of tests, as ES Modules cannot be mocked
(do yourself a favor and don't try to mock them)
- C8 now replaces NYC as a native coverage tool
- 64caebaf: Migrate away from `@krlwlfrt/async-pool`
```ts
import {mapAsyncLimit} from '@openstapps/collection-utils';
await mapAsyncLimit(
[1, 2, 3],
async it => {
await someNetworkRequest(it);
},
5,
);
```
- 64caebaf: Migrate moment.js to date-fns
### Patch Changes
- 64caebaf: Migrated changelogs to changeset format
```js
import fs from 'fs';
const path = 'packages/logger/CHANGELOG.md';
fs.writeFileSync(path, fs.readFileSync(path, 'utf8').replace(/^#+\s+\[/gm, '## ['));
```
- 64caebaf: Moved project-docs to the [GitLab Wiki](https://gitlab.com/openstapps/openstapps/-/wikis/home) repo.
- 98546a97: Migrate away from @openstapps/configuration
- 4e4c7b5c: Update release configs
- 23481d0d: Update to TypeScript 5.1.6
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [64caebaf]
- Updated dependencies [98546a97]
- Updated dependencies [64caebaf]
- Updated dependencies [98546a97]
- Updated dependencies [23481d0d]
- Updated dependencies [64caebaf]
- Updated dependencies [98546a97]
- Updated dependencies [64caebaf]
- @openstapps/collection-utils@3.0.0
- @openstapps/gitlab-api@3.0.0
- @openstapps/logger@3.0.0
## 3.0.0-next.4
### Major Changes
- 11c9d742: Move images to separate packages
Removed builder image due to migration to Kaniko
### Patch Changes
- 4e4c7b5c: Update release configs
- 23481d0d: Update to TypeScript 5.1.6
- Updated dependencies [23481d0d]
- @openstapps/collection-utils@3.0.0-next.4
- @openstapps/gitlab-api@3.0.0-next.4
- @openstapps/logger@3.0.0-next.4
## 3.0.0-next.3
### Patch Changes

Some files were not shown because too many files have changed in this diff Show More