![JSON-Schema
JSON File
[{
"album": "A Tribute to Jack Johnson",
"artist": "Miles Davis",
"genre": [
"Jazz"
],
"mime": "audio/mp3",
"performers": [
"Miles Davis"
],
"tracks": [
"Part 1",
"Part 2"
],
"year": 1970
}, ...]
Schema File
{"title": "Music Album",
"type": "object",
"properties": {
"album": {
"type": "string",
"minLength": 1
},
"mime": {
"type": "string",
"minLength": 7,
"pattern": "^audio/",
"enum": [
"audio/flac",
"audio/mp2",
"audio/mp3",
"audio/mp4",
"audio/vorbis"
]
},
...},
"required": ["album", "year"],
"additionalProperties": false}
Library Network of Western Switzerland 13 Lausanne, 2017/03/10](https://image.slidesharecdn.com/esmeetupmaj2017-170511115655/85/Elasticsearch-a-key-element-of-Invenio-3-13-320.jpg)
![ES Mapping
JSON File
[{
"album": "A Tribute to Jack Johnson",
"artist": "Miles Davis",
"genre": [
"Jazz"
],
"mime": "audio/mp3",
"performers": [
"Miles Davis"
],
"tracks": [
"Part 1",
"Part 2"
],
"year": 1970
}, ...]
Mapping File
{"mappings": {
"record-v1.0.0": {
"date_detection": false,
"numeric_detection" : false,
"properties": {
"album": {
"type": "string",
"analyzer": "english",
"copy_to": "sort_album"
},
"sort_album": {
"type": "string",
"index": "not_analyzed"
},
"artist": {
"type": "string",
"analyzer": "standard",
"copy_to": "facet_artist"
},
"facet_artist": {
"type": "string",
"index": "not_analyzed"
},
...
} } } }
Library Network of Western Switzerland 14 Lausanne, 2017/03/10](https://image.slidesharecdn.com/esmeetupmaj2017-170511115655/85/Elasticsearch-a-key-element-of-Invenio-3-14-320.jpg)
![Configuration
Sorting Configuration
RECORDS_REST_SORT_OPTIONS = dict(
records=dict(
bestmatch=dict(
fields=[’-_score’],
title=’Best match’,
default_order=’asc’,
order=1,
),
mostrecent=dict(
fields=[’-_created’],
title=’Most recent’,
default_order=’asc’,
order=2,
)
)
)
RECORDS_REST_DEFAULT_SORT = dict(
records=dict(query=’bestmatch’, noquery=’mostrecent’),
)
Library Network of Western Switzerland 17 Lausanne, 2017/03/10](https://image.slidesharecdn.com/esmeetupmaj2017-170511115655/85/Elasticsearch-a-key-element-of-Invenio-3-17-320.jpg)
Elasticsearch: a key element of Invenio 3
- 1. Elasticsearch: a key element of Invenio 3 Elasticsearch Meetup Johnny Mariéthoz Lausanne, 2017/03/10
- 2. About Me 12 years as computer scientist in machine learning 7 years as Invenio developer and instance maintainer bass and double bass player newbie as analog camera photographer Library Network of Western Switzerland 2 Lausanne, 2017/03/10
- 3. Library Network of Western Switzerland Library Network of Western Switzerland 3 Lausanne, 2017/03/10
- 4. RERO: Library Network of Western Switzerland 220 libraries academic libraries, heritage libraries, public libraries, school libraries or specialized libraries 50’000 students 5 cantons: FR, GE, JU, NE, VS 280’000 registered patrons 3 academic universities Geneva, Fribourg, Neuchâtel 1 University of Applied Sciences central office in Martigny 19 employees Library Network of Western Switzerland 4 Lausanne, 2017/03/10
- 5. Typical Data Centered Web Application Data Web Server Data Schema Persistant Storage Search Engine PID Store External Files REST API HTML WEB Pages GUI Search Engines: Google, etc. External Services Access Rights Files Download / Preview for users Other Formats Browser apps Desktop apps Library Network of Western Switzerland 5 Lausanne, 2017/03/10
- 6. Common Needed Features data management with versioning, validation and PID (Persistent Identifiers) search engine rights management (ACL, oauth) web page management with templates (search results and others, such as news, front-page, etc.) url management (routing) REST API generation format conversion/migration CLI utilities data acquisition (html forms based editor) Library Network of Western Switzerland 6 Lausanne, 2017/03/10
- 7. Development modular software architecture easy new module creation webassets management for less, sass, nodejs, etc. asynchronous task management unit testing and logging i18n (translations) web front-end and back-end and many more... Library Network of Western Switzerland 7 Lausanne, 2017/03/10
- 8. Library Network of Western Switzerland 8 Lausanne, 2017/03/10
- 9. History digital library and document repository software created by CERN mature platform: first public release v0.0.9 in 2002 open source project originated in high-energy physics institutional repository: CERN Document Server integrated library system: CERN Document Server disciplinary repository: INSPIRE open research data server: ZENODO self-contained python, mysql web application until 1.x transition with v2.x complete new rewritten v3 Library Network of Western Switzerland 9 Lausanne, 2017/03/10
- 10. Used Technologies set of python modules include module interaction mechanisms delivered as a framework around state-of-the-art technologies steep learning curve Library Network of Western Switzerland 10 Lausanne, 2017/03/10
- 11. + Library Network of Western Switzerland 11 Lausanne, 2017/03/10
- 12. Elasticsearch Integration SQL only for persistent data, no more SQL query during the HTTP request data model using JSON, JSON-Schema and ES Mapping sorting, facets and query configuration by type of object use official elasticsearch python package: elasticsearch-dsl CLI to create indexes, push mappings and index the data Library Network of Western Switzerland 12 Lausanne, 2017/03/10
- 13. JSON-Schema JSON File [{ "album": "A Tribute to Jack Johnson", "artist": "Miles Davis", "genre": [ "Jazz" ], "mime": "audio/mp3", "performers": [ "Miles Davis" ], "tracks": [ "Part 1", "Part 2" ], "year": 1970 }, ...] Schema File {"title": "Music Album", "type": "object", "properties": { "album": { "type": "string", "minLength": 1 }, "mime": { "type": "string", "minLength": 7, "pattern": "^audio/", "enum": [ "audio/flac", "audio/mp2", "audio/mp3", "audio/mp4", "audio/vorbis" ] }, ...}, "required": ["album", "year"], "additionalProperties": false} Library Network of Western Switzerland 13 Lausanne, 2017/03/10
- 14. ES Mapping JSON File [{ "album": "A Tribute to Jack Johnson", "artist": "Miles Davis", "genre": [ "Jazz" ], "mime": "audio/mp3", "performers": [ "Miles Davis" ], "tracks": [ "Part 1", "Part 2" ], "year": 1970 }, ...] Mapping File {"mappings": { "record-v1.0.0": { "date_detection": false, "numeric_detection" : false, "properties": { "album": { "type": "string", "analyzer": "english", "copy_to": "sort_album" }, "sort_album": { "type": "string", "index": "not_analyzed" }, "artist": { "type": "string", "analyzer": "standard", "copy_to": "facet_artist" }, "facet_artist": { "type": "string", "index": "not_analyzed" }, ... } } } } Library Network of Western Switzerland 14 Lausanne, 2017/03/10
- 15. Schema and Mapping JSON-SCHEMA: schemas/records/record-v1.0.0.json should be included in the data ($schema) used for data validation is the documentation for humans name is important: i.e. index_name: records-record-v1.0.0, document_type: record-v1.0.0 for record indexing Mapping mappings/records/record-v1.0.0.json can be set using a CLI name is important: index_name: records-record-v1.0.0 with alias=records, document_type: record-v1.0.0 during the index creation Library Network of Western Switzerland 15 Lausanne, 2017/03/10
- 16. Configuration Facets Configuration RECORDS_REST_FACETS = dict( records=dict( aggs={ ’genre’: dict(terms=dict(field=’genre’, size=10)), ’years’: dict(date_histogram=dict( field=’year’, interval=’year’, format=’yyyy’) ) }, filters=dict( genre=terms_filter(’genre’) ), post_filters=dict( years=range_filter( ’year’, format=’yyyy’, end_date_math=’/y’), ) ) ) Library Network of Western Switzerland 16 Lausanne, 2017/03/10
- 17. Configuration Sorting Configuration RECORDS_REST_SORT_OPTIONS = dict( records=dict( bestmatch=dict( fields=[’-_score’], title=’Best match’, default_order=’asc’, order=1, ), mostrecent=dict( fields=[’-_created’], title=’Most recent’, default_order=’asc’, order=2, ) ) ) RECORDS_REST_DEFAULT_SORT = dict( records=dict(query=’bestmatch’, noquery=’mostrecent’), ) Library Network of Western Switzerland 17 Lausanne, 2017/03/10
- 18. Configuration REST Configuration RECORDS_REST_ENDPOINTS = dict( recid=dict( search_class=RecordsSearch, search_index=’records’, search_type=None, record_serializers={ ’application/json’: (’invenio_records_rest.serializers’ ’:json_v1_response’), }, search_serializers={ ’application/json’: (’invenio_records_rest.serializers’ ’:json_v1_search’), }, search_factory_imp=es_search_factory, list_route=’/records/’, item_route=’/records/<pid(recid):pid_value>’ ) ) Library Network of Western Switzerland 18 Lausanne, 2017/03/10
- 19. Demo Library Network of Western Switzerland 19 Lausanne, 2017/03/10
- 20. Conclusion very generic and flexible tool great open source community (many thanks to the CERN) easy to prototype and develop new applications and features demands time to master (learning curve) at the center of RERO’s future developments swiss open access research publications repository (SONAR) new Integrated Library System (ILS) for public libraries (3 years project) and many more projects... Library Network of Western Switzerland 20 Lausanne, 2017/03/10
- 21. References RERO http://www.rero.ch Invenio http://invenio-software.org/ Invenio Documentation http://invenio.readthedocs.io Elasticsearch https://www.elastic.co CERN http://home.cern JSON-LD http://json-ld.org/ JSON Schema http://json-schema.org/ ZENODO https://zenodo.org/ Library Network of Western Switzerland 21 Lausanne, 2017/03/10