deichman merge requestshttp://gitlab.deichman.no/digibib/deichman/-/merge_requests2022-02-21T19:58:28Zhttp://gitlab.deichman.no/digibib/deichman/-/merge_requests/531Forlagsserier - #830 mapping2022-02-21T19:58:28ZTom AdamForlagsserier - #830 mappingTom AdamTom Adamhttp://gitlab.deichman.no/digibib/deichman/-/merge_requests/530DEICH-6150 populate options in year range filter dynamically2022-03-04T10:03:45ZPetter Goksøyr ÅsenDEICH-6150 populate options in year range filter dynamicallyCloses DEICH-6150Closes DEICH-6150http://gitlab.deichman.no/digibib/deichman/-/merge_requests/529DEICH-6102 - Suggestion to improve mapping code2022-02-17T07:47:38ZTom AdamDEICH-6102 - Suggestion to improve mapping codeThis structure avoids that the mapping functions are calling each other, thus it becomes possible to override a base mapping function in Marc21 or Normarc implementation classes. This stops the spreading of no-op mapping functions if one...This structure avoids that the mapping functions are calling each other, thus it becomes possible to override a base mapping function in Marc21 or Normarc implementation classes. This stops the spreading of no-op mapping functions if one would like to move out a mapping function to a stand-alone function.
Closes DEICH-6102Tom AdamTom Adamhttp://gitlab.deichman.no/digibib/deichman/-/merge_requests/528DEICH-6102 - ind2HasOneOf simplifications2022-02-16T11:31:50ZTom AdamDEICH-6102 - ind2HasOneOf simplificationsThe function was returning an Optional and its value was never used. Rewritten code to replace return type with boolean. Also eliminated some hard to read ifPresentOrElse constructs.The function was returning an Optional and its value was never used. Rewritten code to replace return type with boolean. Also eliminated some hard to read ifPresentOrElse constructs.Tom AdamTom Adamhttp://gitlab.deichman.no/digibib/deichman/-/merge_requests/527DEICH-0000 - Added config to enforce node and npm version.2022-02-25T10:45:54ZTom AdamDEICH-0000 - Added config to enforce node and npm version.Versions were picked from the docker container.
Closes DEICH-0000Versions were picked from the docker container.
Closes DEICH-0000Tom AdamTom Adamhttp://gitlab.deichman.no/digibib/deichman/-/merge_requests/526DEICH-6173 - removed message2022-02-15T08:39:41ZTom AdamDEICH-6173 - removed messageCloses DEICH-6173Closes DEICH-6173Tom AdamTom Adamhttp://gitlab.deichman.no/digibib/deichman/-/merge_requests/525DEICH-6102 - In preparation to mapping improvements2022-02-15T11:55:02ZTom AdamDEICH-6102 - In preparation to mapping improvementsAdded tag to MarcRecord.DataField. This required introducing a builder pattern thanks to the sequential processing the way XML parsing happens.
Added many todos for the next week :)
Closes DEICH-6102Added tag to MarcRecord.DataField. This required introducing a builder pattern thanks to the sequential processing the way XML parsing happens.
Added many todos for the next week :)
Closes DEICH-6102http://gitlab.deichman.no/digibib/deichman/-/merge_requests/524Resolve DEICH-6167 "isomorfic bug"2022-02-10T12:55:47ZTom AdamResolve DEICH-6167 "isomorfic bug"Closes DEICH-6167Closes DEICH-6167Tom AdamTom Adamhttp://gitlab.deichman.no/digibib/deichman/-/merge_requests/523DRAFT DEICH-6169 add tracing to redia-api, sibyl and euler2022-02-11T11:53:19ZTorsteinDRAFT DEICH-6169 add tracing to redia-api, sibyl and eulerCloses DEICH-6169Closes DEICH-6169http://gitlab.deichman.no/digibib/deichman/-/merge_requests/522DEICH-6106 - BIBBI Marc 21 - still disabled in prod + search source feature t...2022-02-15T15:18:34ZTom AdamDEICH-6106 - BIBBI Marc 21 - still disabled in prod + search source feature toggleAdded BIBBI 21 endpoint
As of now the endpoint will be disabled using feature toggle. One might invoke the source, but it will be filtered out, and no result is returned at all from this source.
As of now instead of BIBI21 the old, BIB...Added BIBBI 21 endpoint
As of now the endpoint will be disabled using feature toggle. One might invoke the source, but it will be filtered out, and no result is returned at all from this source.
As of now instead of BIBI21 the old, BIBI endpoint serves the clients with the old NORMARC format.
Note that a bug was found when moving over to MARC 21, which needs further investigation (https://jira.oslo.kommune.no/browse/DEICH-6167).
As the last commit message reads:
_-For the BIBI21 endpoint one of the tests is somewhat fragile. Model isomorphism check fails for still not totally discovered reasons. However, BIBI21 is still disabled din prod, so the code might be merged in to master, and further work can be done to discover the reason of the problem._
_Observations so far:_
- seems like if the blank node ID reaches 1000, the model isomorphism check fails - which means that if the test case ("searchType": "TEXT","requests": [{"searchKind": "ALL","terms": ["M-065-11040-0"]) is moved further down in the searchTestSources.json file, it fails._
- if DeichModel.newIRI is run prior to code execution, the isomorphism check fails again_
_The problem showed up after that the MARC21 format was introduced for the BIBI21 source.
Apparently ALMA - which is using MARC21 as well - does not show these symptoms._
_My current guess is that a combination of the data returned by BIBBI21 and how DeichModel is being built up causes the issue. Further investigation is needed._Tom AdamTom Adamhttp://gitlab.deichman.no/digibib/deichman/-/merge_requests/521DEICH-6157 tjenestekatalog: change info text on reporting2022-02-17T10:10:51ZBenjaminDEICH-6157 tjenestekatalog: change info text on reportingCloses DEICH-6157Closes DEICH-6157http://gitlab.deichman.no/digibib/deichman/-/merge_requests/520DEICH-6114 Nytt API+refaktorert kodebase2022-02-07T07:52:23ZTom AdamDEICH-6114 Nytt API+refaktorert kodebaseThe enum contains al possible formats, and in addition the parser, model and json converter supliers (since a new instance needs to be provided for each parsing/conversion).
The format is defined as a property, and will be autowired by S...The enum contains al possible formats, and in addition the parser, model and json converter supliers (since a new instance needs to be provided for each parsing/conversion).
The format is defined as a property, and will be autowired by Spring.
More concise parsing and conversion handling.
Note, Search tests are broken - even still they run green :) Needs to be fixed as new test architecture comes to place.
Closes DEICH-6114Tom AdamTom Adamhttp://gitlab.deichman.no/digibib/deichman/-/merge_requests/519DEICH-6518 don't try to validate empty email2022-02-02T12:36:04ZPetter Goksøyr ÅsenDEICH-6518 don't try to validate empty emailCloses DEICH-6158Closes DEICH-6158http://gitlab.deichman.no/digibib/deichman/-/merge_requests/518DEICH-6145 add reserves bound to checked-out items cleanup query2022-02-17T06:12:45ZPetter Goksøyr ÅsenDEICH-6145 add reserves bound to checked-out items cleanup queryNB må ikke merges før https://gitlab.deichman.no/digibib/deichman/-/merge_requests/508
Closes DEICH-6145NB må ikke merges før https://gitlab.deichman.no/digibib/deichman/-/merge_requests/508
Closes DEICH-6145http://gitlab.deichman.no/digibib/deichman/-/merge_requests/517DEICH-6139 - Moved static field to make AutoExpandingJsonWriter happy2022-02-07T09:18:43ZTom AdamDEICH-6139 - Moved static field to make AutoExpandingJsonWriter happyCloses DEICH-6139Closes DEICH-6139Tom AdamTom Adamhttp://gitlab.deichman.no/digibib/deichman/-/merge_requests/516DEICH-6154 update ES to avoid log4j RCE2022-02-03T09:45:42ZTorsteinDEICH-6154 update ES to avoid log4j RCEThere are breaking changes, but none seem to hit Deichman's usage.
The changes in the config is to avoid warnings.
The discovery-type is a security measure.
The code was tested locally and a full reindexing was done with good results.
...There are breaking changes, but none seem to hit Deichman's usage.
The changes in the config is to avoid warnings.
The discovery-type is a security measure.
The code was tested locally and a full reindexing was done with good results.
- Search and autocomplete was done in prod and locally on ~same data and gave the ~same results (prod index is likely slightly out of sync)
- There are slight differences in ordering on automcomplete (can be because of difference in underlying data which affects ranking)
As a part of the release, me and Petter have discussed doing a "clean slate" reindexing to try to get the production index in sync.
This means that we can safely deploy this code, but we should do reindexing the same evening.
NOTE: this procedure should be done in the test environment first!
- Notify about potential issues with search to operations and on Workplace
- Stop Sibyl and Elasticsearch
- `TRUNCATE sibyl.index_docs` and `TRUNCATE sibyl.koha_cache` in Sibyl's DB
- Optional: consider deleting the elasticsearch volume (`docker volume rm deichman_elasticsearch_data`)
- Bring up ES check logs, if ok, bring up Sibyl
- Wait for Koha-cache to be updated in Sibyl (takes a few minutes)
- Start full reindexing
- Wait... :)
Closes DEICH-6154http://gitlab.deichman.no/digibib/deichman/-/merge_requests/515Resolve DEICH-6154 "Log4j og logback"2022-02-01T06:03:51ZTom AdamResolve DEICH-6154 "Log4j og logback"Closes DEICH-6154Closes DEICH-6154http://gitlab.deichman.no/digibib/deichman/-/merge_requests/514DEICH-6119 Mimir - rewrite2022-01-28T22:21:41ZTom AdamDEICH-6119 Mimir - rewriteSkal legge til en tes til SearchController etter merge til blackbox test branchen. Ellers klar for gjennomgang.Skal legge til en tes til SearchController etter merge til blackbox test branchen. Ellers klar for gjennomgang.Tom AdamTom Adamhttp://gitlab.deichman.no/digibib/deichman/-/merge_requests/513DEICH-6142 record branch and time of when an item is bound to reserve2022-02-08T08:32:12ZPetter Goksøyr ÅsenDEICH-6142 record branch and time of when an item is bound to reserveThe reserves table needs 2 additional columns:
ALTER TABLE reserves ADD COLUMN `bound_at` datetime;
ALTER TABLE reserves ADD COLUMN `bound_branch` VARCHAR(10);
Closes DEICH-6144The reserves table needs 2 additional columns:
ALTER TABLE reserves ADD COLUMN `bound_at` datetime;
ALTER TABLE reserves ADD COLUMN `bound_branch` VARCHAR(10);
Closes DEICH-6144http://gitlab.deichman.no/digibib/deichman/-/merge_requests/512DEICH-6097 DEICH-6149 make events cancellable2022-01-27T12:30:33ZPetter Goksøyr ÅsenDEICH-6097 DEICH-6149 make events cancellableMigration SPARQLs for existing events:
Set cancelled prop to true/false:
```
PREFIX : <https://tjenester.deichman.no/ontology/>
WITH <https://tjenester.deichman.no>
INSERT {
?id :cancelled ?cancelled .
} WHERE {
?id a :LibraryEvent...Migration SPARQLs for existing events:
Set cancelled prop to true/false:
```
PREFIX : <https://tjenester.deichman.no/ontology/>
WITH <https://tjenester.deichman.no>
INSERT {
?id :cancelled ?cancelled .
} WHERE {
?id a :LibraryEventInstance ; :title ?title .
FILTER NOT EXISTS { ?id :cancelled ?cancelled }
BIND(IF(regex(?title, "avlyst", "i"), true, false) AS ?cancelled)
}
```
Remove "AVLYST" variants from beginning title:
```
PREFIX : <https://tjenester.deichman.no/ontology/>
WITH <https://tjenester.deichman.no>
DELETE { ?id :title ?old_title }
INSERT { ?id :title ?new_title }
WHERE {
?id a :LibraryEventInstance ; :title ?old_title .
BIND(REPLACE(?old_title, "^\\s?avlyst[!:.]*(\\s*)?-?\\s?", "", "i") AS ?new_title)
FILTER(REGEX(?old_title, "avlyst", "i"))
}
```
Closes DEICH-6097
Closes DEICH-6149