Natural Language Understanding of Systems Engineering Artifacts
0 likes292 views
The document discusses advancements in natural language understanding (NLU) within the context of systems engineering artifacts, highlighting the development of model-self descriptions (MSDs) as a bridge between NLU algorithms and domain-specific models. The study evaluates the effectiveness of traditional and modern search techniques, specifically comparing TF-IDF and BERT models in semantic searches related to systems engineering. Preliminary findings suggest that combining simplistic and advanced approaches could enhance the comprehension of domain-specific semantics, while future work is needed to refine these techniques for improved search quality.
Natural Language Understanding of Systems Engineering Artifacts
- 1. Natural Language Understanding of Systems Engineering Artifacts INCOSE International Symposium 2022 Detroit 2022 June 25-30 Authors: Géza Kulcsár (IncQuery Labs), Gaëtan Pruvost (Thales Global Services), Olivier Constant (TGS), István Ráth (IQL), Máté Füzesi (IQL), Dénes Harmath (IQL) Presenter: Ákos Horváth (IQL) akos.horvath@incquerylabs.com
- 2. The Y-Model of Digital Systems Engineering
- 3. Systems Model Understanding: Beware, Experts Only Systems Engineering Artifacts Querying Query Result Set (Relevant Patterns) Processing Information Reports MODELING EXPERTISE DOMAIN EXPERTISE Analysis
- 4. Vision: Accessible System Model Analysis Multi-Domain Knowledge Graph Systems Engineering Artifacts Querying Query Result Set (Relevant Patterns) MODELING EXPERTISE DOMAIN EXPERTISE Traces Traces
- 5. Contribution 1: Model-Self Descriptions Model Self- Descriptions (MSD) Systems Engineering Artifacts Querying Query Result Set (Relevant Patterns) MODELING EXPERTISE Language Model Representation Model to Text DOMAIN EXPERTISE Traces Traces are available
- 6. What is Natural Language Understanding? Source: “A Comprehensive Guide to Natural Language Generation.” Medium, Sciforce. 2019
- 7. What is Natural Language Understanding? • The ongoing revolution in Natural Language Understanding (NLU) results in better solutions for automatically interpreting “general” texts • That is, if the nature of our input text corresponds to “typical” documents (as found, e.g., on the internet), then we have good chances to have quality results for NLU tasks • However, sometimes, there is a specific meaning to a text, (meant to be) understood by experts only • Model-self descriptions are an intermediate step between NLU algorithms and domain-specific artifacts – here, system models https://www.kdnuggets.com/2019/03/deconstructing-bert-part-2-visualizing-inner-workings-attention.html
- 8. Model-Self Descriptions: Models to text • Current approach: a query- and template-based information extractor and text generator • (Potential for future work: utilizing neural networks for text production) • Sample queries • VQL (Viatra Query Language) “look for SysML blocks in a given package” • Gremlin (a tool-independent graph query language for the Apache Tinkerpop framework) “get Capella elements with Logical Component or Physical Component as domain type”
- 9. Example MSD Sentences • Example for this presentation: The Hybrid SUV SysML model provided by OMG • When turning the results of such queries into English sentences based on pre- defined templates (only presupposing knowledge of SysML but not of the source model or the domain it comes from), we get sentences like: • The Environment block does not have any value properties. • The Environment block has a weather part property with the type of Weather. • The Weather block is in the HSUVModel package. • The Transmission block contains 2 part properties, which are the fuelReturn property with the type of Fuel and the fuelSupply property with the type of Fuel.
- 10. Contribution 2: Natural Search in Models Model Self- Descriptions (MSD) Systems Engineering Artifacts Querying Query Result Set (Relevant Patterns) MODELING EXPERTISE Language Model Representation NLP Natural search Model to Text DOMAIN EXPERTISE Traces Traces are available
- 11. Language Models • Generally, a language model is a representation of natural-language texts in numerical form, allowing for machine interpretations • However, their complexity ranges from simple, fixed word-to-vector mappings to cutting-edge neural networks • Without getting too much into technical details, we did experiments using two established language models: • TF-IDF (term frequency – inverse document frequency) • A rather simple approach, with vector representations of words only slightly process: a term gets higher weights if it is frequent in the input document we study, but rare in the overall language corpus. Intuitively, this model could support domain-specific search scenarios such as ours. • BERT by Google (a deep learning model with context awareness features) • Pre-trained on Wikipedia and other generic, large-volume English-language sources, BERT can be downloaded by anyone and used for NLP/NLU scenarios. It promises advanced, human-like understanding in certain contexts by its context-sensitive vector encodings.
- 12. Experiment: Semantic Search in the HSUV Model • In this experiment, we use an intuitive set of five search terms, representing various semantic facets of the input model (and its domain), as well as divers search strategies from single-word searches to actual questions: • environment • environmental awareness • fuel • What properties do we store about fuel? • subsystem of HybridSUV
- 13. Results: TF-IDF
- 14. Results: BERT
- 15. Qualitative discussion • The “environment” search term reveals that in a model-based setting, using deeper natural-language semantics does not obviously bring benefits: result with TF-IDF are much cleaner, while BERT results also include nonsense • With a slightly more complicated term, “environmental awareness”, TF-IDF seems to produce erroneous behavior, while the results of BERT look smart • Results for “fuel”, again, show that simplicity also has its merits: BERT’s findings feel totally irrelevant • However, interestingly, the power of BERT is somewhat increased if we look for the same thing in a natural-language questioning scenario: “What properties do we store about fuel?”
- 16. Conclusion and Future Work • We have provided a conceptual framework for bridging state-of-the-art NLU techniques and systems model comprehension, along with an experimental prototype • First, preliminary results show that domain-specific semantic comprehension most likely requires a combination of simplistic search mechanisms and advanced semantic approaches • Obviously, there is need for further evidence with more search terms, diverse corpora and more engineering domains • There is a yet unexploited potential in fine-tuning (training) neural networks like BERT on actual system models (MSDs) to increase search quality
- 17. akos.horvath@incquerylabs.com incquery.io +36 30 5511164 Thank you! @IncQueryLabs