Gen AI with LLM for construction technology
0 likes611 views
The document by Taewook Kang presents a comprehensive overview of generative AI and AI applications in construction technology and management. It discusses the progression of technology integration in the AEC industry, including the use of digital twins, IoT, and machine learning, while providing various use cases of generative AI, licensing for open-source tools, and technical specifications for smart devices. Additionally, it emphasizes the importance of fun in technology, highlighting developments and funding trends in the smart construction field.
![Jackpot & hallucination
output.append((transformer.generate(t
exts = ['bridge chair','computer table',
'chair table'] , temperature = 1) ))
output.append((transformer.generate(texts =
['sofa','bed', 'computer screen', 'bench', 'chair', 'table' ] ,
temperature = 0.0) ))
Not
practical.
MeshGPT](https://image.slidesharecdn.com/genaillm10-240722153326-1e90b9a5/85/Gen-AI-with-LLM-for-construction-technology-78-320.jpg)
![Gen AI + LLM
class ChatPDF:
vector_store = None
retriever = None
chain = None
def __init__(self):
# OLLAMA의 mistral 모델 이용
self.model = ChatOllama(model="mistral")
# PDF 텍스트 분할
self.text_splitter = RecursiveCharacterTextSplitter(chunk_size=1024, chunk_overlap=100)
self.prompt = PromptTemplate.from_template(
"""
<s> [INST] You are an assistant for question-answering tasks.
Use the following pieces of retrieved context
to answer the question. If you don't know the answer,
just say that you don't know. Use three sentences
maximum and keep the answer concise. [/INST] </s>
[INST] Question: {question}
Context: {context}
Answer: [/INST]
"""
)](https://image.slidesharecdn.com/genaillm10-240722153326-1e90b9a5/85/Gen-AI-with-LLM-for-construction-technology-87-320.jpg)
![Gen AI + LLM
ENA Model ID Batch
size
Epochs Layer architecture Normaliza
tion &
Activation
Learning
rate
Pre-trained model
embedding transformers
M1.1. MLP 32 150 [14]-[128,64,32]-[10] ReLU,
batch
normal,
dropout
0.001 - -
M1.2. MLP 32 150 [14]-[64,128,64]-[10] 0.001 - -
M1.3. MLP 32 150 [14]-[64,128,64,32]-[10] 0.001 - -
M1.4. MLP 32 150 [14]-[32,64,32]-[10] 0.001 - -
M2.1. LSTM 32 150 [14]-LSTM[128]-[10] dropout 0.001 - -
M2.2. LSTM 32 150 [14]-LSTM[128]-[64,32]-[10] 0.001 - -
M2.3. LSTM 32 150 [14]-LSTM[256]-[128,64]-[10] 0.001 - -
M3.1. Transformers
32 300 [320,512]-ENC[512]-[64]-[10]
multi-head
attention,
layer
normal,
dropout
1.00E-05
BERT-base-
uncased
-
M3.2.Transformers
64 300 [320,512]-ENC[512]-[64]-[10] 1.00E-05
BERT-base-
uncased
-
M3.3.Transformers
128 300 [320,512]-ENC[512]-[64]-[10] 1.00E-05
BERT-base-
uncased
-
M4.1. LLM
32 150 [293]-EMB-ENC-[10] 1.00E-05
BERT-base-
uncased
BERT
M4.2 LLM
32 300 [293]-EMB-ENC-[10] 1.00E-05
BERT-base-
uncased
BERT
1](https://image.slidesharecdn.com/genaillm10-240722153326-1e90b9a5/85/Gen-AI-with-LLM-for-construction-technology-97-320.jpg)
Gen AI with LLM for construction technology
- 1. 강태욱 연구위원 공학박사 Ph.D Taewook, Kang laputa99999@gmail.com https://daddynkidsmakers.blogspot.com 건설AI 히치하이커를 위한 가이드 GEN AI + LLM에 초점을 맞춘
- 2. KICT Intro Advanture. usecase from AI to Gen AI My lab tools. open source Jackpot and hallucination Deep dive. Gen AI + LLM Journey. Breadcrumbs The Hitchhiker’s Guide to the Galaxy
- 3. KICT Intro
- 4. Maker, Ph.D. 12 books author BIM principle and Digital transformation: BIM principle & DX 사이트 소개, Profile (dxbim.blogspot.com)
- 5. Study history P4 Life cycle “If it isn’t fun, you’re doing the wrong technology.” - ivan sutherland
- 6. Study history P4 Roadmap Computer graphics, CAD BIM-GIS + ISO (2012) BIM + IoT + Scan (2014) Scan to BIM + Robotics + AI (2017) Digital Twin + AI (2021) BIM VDCO Mining IoT Scan Virtual Design Construction Operations 1D, 2D 3D, nD Semantic Knowledge Sensors Robotics
- 8. Generative AI identified by Deloitte • Marketing content assistant • Code assistant for developers • Customer support on demand • Product design assistant • Research-based report generation • Synthetic data generation • Enterprise-wide data search and access • Game content development • Language translation • Simulation of urban planning scenarios • Hyper-personalised education • Onboarding assistant Usecase
- 9. Generative AI in AECO • Multi Agent • Anomaly detection in contract documents • Reasoning in IoT dataset for O&M • Risk management • Optimization in Construction Management • Concept design generation • Report summarization and generation • Training data generation • Query using LLM • Language translation • Simulation of urban planning scenarios • Personalised education and guidance system Usecase
- 10. Scenario CPS & DT Digital Twin Structural health monitoring Track and trace Remote diagnosis Remote services Remote control Condition monitoring Systems health monitoring BIM as i-DB IoT… AI Sensor device ICBM Simulation Robotics Scan-Vision Smart contract based on Blockchain Gen AI Multi Agent
- 11. Scenario IoT Big data management AI + Simulation using LLM Cloud platform Machine control Field monitoring IoT sensor Usecase for safety, accuracy, productivity sensing Data analysis & prediction GIS IoT based monitoring Field control Infra IoT service connection Plant control system (SCADA) Field monitoring system LoRA, BLE, WiFi… Layer 8 | IISL (Infra IoT Service Layer) Worker Agency <device_definition id=‘dd#1’> <device id=‘T#1’name=‘temp’type=‘temperature’> <maker name=‘CH korea’ email=‘laputa99999_9@gmail.com’ tel=‘82-0330-0802-1013’ location=‘…’/> <specification> <op_range name=‘voltage’ unit=‘V’type=‘real’value=‘3.3’/> <op_range name=‘temperature’ unit=‘degree’ type=‘real’begin=‘-10.0’end=’60.0’/> <op_range name=‘humidity’unit=‘%R.H’type=‘real’begin=‘0.0’end=’50.0’/> <op_range name=‘GPS’unit=‘WGS84’type=‘vector2D’begin=‘(0,0)’ end=‘(127, 32)’/> <op_range name=‘characteristic_curve’unit1=‘temperature’ unit2=‘voltage’ type=‘vector2D’> (0,0), (1.2, 2.4), (3.5, 6.2), (4.1, 7.2) </op_range> <op_range name=‘period’unit=‘year’value=‘2’/> </specification> </device> </device_definition> Intelligent IoT sensor •Self diagnose •IISL protocol •Security •Availability 1 1 2 3 4 5 6 7 8 Plant sensing
- 12. Overview AI ConTech $3B funding from 2021 (Daniel Laboe, 2023.10) 2024년 상반기 스마트 건설과 BIM 기술 동향 (buildingSMART ) Machine learning Deep learning Gen AI Multi- agent AGI
- 13. Usecase AI 기반 텍스트 렌더링 Revit 애드인 (VERAS, EnvolveLAB)
- 15. Usecase
- 16. Usecase Technology to create 2D drawings with perfect dimensions from 3D models with one click (Autodesk Fusion, November 2023)
- 17. Usecase CreationAI-based design system AiCorb (Obayashi Corporation)
- 18. Usecase AI-based structural frame, automatic creation of member cross sections SYMPREST (Shimizu Construction)
- 19. Usecase
- 22. Usecase Spot-R
- 25. Usecase Rebar Tying 로봇(Florida DoT, 2020)
- 26. Usecase Energy savings through machine learning AMR DNA, Energy savings through machine learning, http://www.energyassets.co.uk/service/amr-dna
- 27. Usecase Siemens Process Simulate (left) connects to NVIDIA Omniverse (right) to enable a full-design-fidelity, photorealistic, real-time digital twin. https://www.robotics247.com/article/siemens_xcelerator_nvidia_omniverse_accel erate_digital_twins_manufacturing NVIDIA Founder and CEO Jensen Huang (left) and Siemens CEO Roland Busch
- 28. My lab tools open source
- 29. Open source Alberto Rizzoli, 2022, 27+ Most Popular Computer Vision Applications and Use Cases in 2022, V7
- 31. Open source Generative Adversarial Networks Fake Obama created using AI video tool - BBC News. Jul 19, 2017 https://www.vegaitglobal.com/media-center/knowledge- base/what-is-stable-diffusion-and-how-does-it-work
- 32. Open source CARLA - Open Urban Driving Simulator https://github.com/carla-simulator/carla YOUTUBE
- 33. Open source Alberto Rizzoli, 2022, 27+ Most Popular Computer Vision Applications and Use Cases in 2022, V7 YOLACT (You Only Look At CoefficienTs) Detectron
- 34. Open source Alberto Rizzoli, 2022, 27+ Most Popular Computer Vision Applications and Use Cases in 2022, V7 Tesseract-OCR Use Google Cloud Vision API to process invoices and receipts
- 35. Open source Open Neural Network eXchange
- 36. Open source
- 38. Open source
- 40. Open source - github
- 41. Open source
- 42. Open source http://guswnsxodlf.github.io/software-license GNU General Public License(GPL) 2.0 – 의무 엄격. SW 수정 및 링크 경우 소스코드 제공 의무 GNU Lesser GPL(LGPL) 2.1 – 저작권 표시. LPGL 명시. 수정한 라이브러리 소스코드 공개 Berkeley Software Distribution(BSD) License – 소스코드 공개의무 없음. 상용 SW 무제한 사용 가능 Apache License – BSD와 유사. 소스코드 공개의무 없음. Mozilla Public License(MPL) – 소스코드 공개의무 없음. 수정 코드는 MPL에 의해 배포. 이외 결합 프로그램 코드는 공개 필요 없음 MIT License – 라이선스 / 저작권만 명시 조건
- 43. Gym for AI Research
- 44. Gym for AI Research Install pythonDownload Python | Python.org Install anaconda Free Download | Anaconda Install vscode Download Visual Studio Code - Mac, Linux, Windows Install sublime Download -Sublime Text cmd*관리자권한으로실행해야함 cd mkdirprojects cd projects mkdirtest pip install virtualenv pip install jupyterlab pip install ipywidgets
- 45. Gym for AI Research 1. pip install tensorflow pip로 TensorFlow 설치 2. pip install keras keras · PyPI 3. pip install torch torchvision torchaudio Start Locally | PyTorch
- 46. Gym for AI Research Kaggle
- 47. Gym for AI Research
- 48. Gym for AI Research pip install virtualenv pip3 install virtualenv .venvScriptsactivate source ./venv/bin/activate
- 49. Gym for AI Research
- 50. Gym for AI Research Install Docker Get Started | Docker Creating a Simple Web Server with Docker: A Step-by-Step Guide to Running Your Web Server as a Container | by Srija Anaparthy | AWS Tip Run cmd docker run -d -p 8080:80 nginxdemos/hello nginxdemos/hello - Docker Image | Docker Hub
- 51. Gym for AI Research
- 52. Gym for AI Research 1. Install Node.JS Download | Node.js (nodejs.org) 2. Run cmd 3. cd c:projects 4. mkdir test 5. cd test 6. npm install -g --unsafe-perm node-red 7. node-red
- 53. Gym for AI Research
- 54. Gym for AI Research https://colab.research.google.com/
- 58. example Linear(4, out = 6) ReLU() BatchNorm1d(in = 6) Linear(in = 6, out = 16) ReLU() BatchNorm1d(in = 16) Linear(in = 16, out = 9) ReLU() … ReLU() BatchNorm1d(in = 3) Linear(in = 3, out = 1)) epochs = 5000 learning_rate = 0.001 batch_size = 64 Loss=0.002. Train MAPE & Acc = (0.025, 95.30%). Test MAPE & Acc = (0.292, 70.76%)
- 64. Gen AI core multi-modal Variational autoencoder
- 65. Gen AI core 잠재공간에 맵핑된(인코딩된) 데이터(Alexej Klushyn, 2019.12, Learning Hierarchical Priors in VAEs)
- 66. Gen AI core multi-modal CLIP(Contrastive Language-Image Pre- Training. Open AI. 2021 https://daddynkidsmakers.blogspot.com/2024/02/clip.html
- 68. Gen AI core
- 70. Gen AI core
- 71. Gen AI core
- 72. Gen AI core
- 73. Gen AI core 스테이블 디퓨전 기술 개발 주역 Computer Vision & Learning Group (ommer-lab.com) Prof. Dr. Björn Ommer University of Munich
- 78. Jackpot & hallucination output.append((transformer.generate(t exts = ['bridge chair','computer table', 'chair table'] , temperature = 1) )) output.append((transformer.generate(texts = ['sofa','bed', 'computer screen', 'bench', 'chair', 'table' ] , temperature = 0.0) )) Not practical. MeshGPT
- 79. Jackpot & hallucination Large Language Models as General Pattern Machines
- 80. Jackpot & hallucination Large Language Models as General Pattern Machines
- 81. Deep Dive Gen AI + LLM
- 82. Gen AI + LLM table data query
- 83. Gen AI + LLM LLAMA & Ollama & Langchain A100 (80GB) GPU = 21 day (traning) ollama run llava "describe this iamge: ./cat.jpg" "테이블 다리로 보이는 것 옆에 똑바로 앉아 있는 얼룩무늬 고양이 의 이미지입니다. 고양이는 뚜렷한 어두운 줄무늬가 있는 밝은 주황 색 털을 가지고 있는데, 이는 흔히 볼 수 있는 패턴입니다. 얼룩 고 양이는 눈을 크게 뜨고 카메라를 정면으로 차분한 태도로 바라보고 있는 모습입니다. 배경에는 바닥에 다음과 같은 패턴의 러그가 깔려 있습니다. 베이지 색과 기타 중성색이 포함됩니다. 전경에 테이블 다리가 있기 때문에 다이닝 룸과 같은 생활 공간처럼 보입니다."
- 84. Gen AI + LLM LLAMA & Ollama & Langchain Langchain CEO. 2022. 30M$ in 2023. https://daddynkidsmakers.blogsp ot.com/2024/04/blog- post_21.html
- 85. Gen AI + LLM Chroma CEO. Jeff Huber, Anton Troynikov 18M$. 2023
- 86. Gen AI + LLM
- 87. Gen AI + LLM class ChatPDF: vector_store = None retriever = None chain = None def __init__(self): # OLLAMA의 mistral 모델 이용 self.model = ChatOllama(model="mistral") # PDF 텍스트 분할 self.text_splitter = RecursiveCharacterTextSplitter(chunk_size=1024, chunk_overlap=100) self.prompt = PromptTemplate.from_template( """ <s> [INST] You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise. [/INST] </s> [INST] Question: {question} Context: {context} Answer: [/INST] """ )
- 88. Gen AI + LLM class ChatPDF: def ingest(self, pdf_file_path: str): docs = PyPDFLoader(file_path=pdf_file_path).load() # 랭체인의 PDF 모듈 이용해 문서 로딩 chunks = self.text_splitter.split_documents(docs) # 문서를 청크로 분할 chunks = filter_complex_metadata(chunks) # 임메딩 벡터 저장소 생성 및 청크 설정 vector_store = Chroma.from_documents(documents=chunks, embedding=FastEmbedEmbeddings()) self.retriever = vector_store.as_retriever(search_type="similarity_score_threshold", search_kwargs={ "k": 3, "score_threshold": 0.5, }, ) # 유사도 스코어 기반 벡터 검색 설정 self.chain = ({"context": self.retriever, "question": RunnablePassthrough()} | self.prompt | self.model | StrOutputParser()) # 프롬프트 입력에 대한 모델 실행, 출력 파서 방법 설정 def ask(self, query: str): # 질문 프롬프트 입력 시 호출 if not self.chain: return "Please, add a PDF document first." return self.chain.invoke(query)
- 89. Gen AI + LLM https://daddynkidsmakers.blogspot.com/2024/ 02/github-copilot-ai.html
- 90. Gen AI + LLM GeoLLM
- 91. Gen AI + LLM GeoLLM
- 92. Gen AI + LLM earthwork quantity takeoff H1 H1 H2 H1 H2 H3 H1 H2 H3 H4 H1 H2 H3 H4 L={H1, H2, H3, H4} F1=L1 F2=L2 Step 1. Step 2. Step 3. Step 4. Step 5. Step 6.
- 93. Gen AI + LLM
- 94. Gen AI + LLM
- 95. Gen AI + LLM
- 96. Gen AI + LLM
- 97. Gen AI + LLM ENA Model ID Batch size Epochs Layer architecture Normaliza tion & Activation Learning rate Pre-trained model embedding transformers M1.1. MLP 32 150 [14]-[128,64,32]-[10] ReLU, batch normal, dropout 0.001 - - M1.2. MLP 32 150 [14]-[64,128,64]-[10] 0.001 - - M1.3. MLP 32 150 [14]-[64,128,64,32]-[10] 0.001 - - M1.4. MLP 32 150 [14]-[32,64,32]-[10] 0.001 - - M2.1. LSTM 32 150 [14]-LSTM[128]-[10] dropout 0.001 - - M2.2. LSTM 32 150 [14]-LSTM[128]-[64,32]-[10] 0.001 - - M2.3. LSTM 32 150 [14]-LSTM[256]-[128,64]-[10] 0.001 - - M3.1. Transformers 32 300 [320,512]-ENC[512]-[64]-[10] multi-head attention, layer normal, dropout 1.00E-05 BERT-base- uncased - M3.2.Transformers 64 300 [320,512]-ENC[512]-[64]-[10] 1.00E-05 BERT-base- uncased - M3.3.Transformers 128 300 [320,512]-ENC[512]-[64]-[10] 1.00E-05 BERT-base- uncased - M4.1. LLM 32 150 [293]-EMB-ENC-[10] 1.00E-05 BERT-base- uncased BERT M4.2 LLM 32 300 [293]-EMB-ENC-[10] 1.00E-05 BERT-base- uncased BERT 1
- 98. Gen AI + LLM ENA Model ID Train No Loss Accuracy Model size (kb) Time performance (minutes) M1.1. MLP 1650 0.0870 0.9494 61 0:02:34 M1.2. MLP 1714 0.0852 0.9519 84 0:02:36 M1.3. MLP 1716 0.0882 0.9544 93 0:02:42 M1.4. MLP 1718 0.1322 0.9507 30 0:02:25 M2.1. LSTM 1730 0.0889 0.9408 812 0:02:13 M2.2. LSTM 1732 0.0851 0.9420 850 0:02:17 M2.3. LSTM 1734 0.0886 0.9408 3,312 0:02:00 M3.1. Transformers 2003 0.3533 0.7744 74,557 0:11:06 M3.2.Transformers 2014 0.3551 0.7719 74,557 0:07:48 M3.3.Transformers 2021 0.3596 0.7423 74,557 0:06:25 M4.1. LLM 0103 0.0587 0.9507 427,783 3:12:05 M4.2. LLM 2334 0.0534 0.9531 427,783 7:59:00
- 99. Gen AI + LLM
- 100. Gen AI + LLM BIM RAG
- 101. Gen AI + LLM BIM RAG
- 102. Gen AI + LLM
- 103. Gen AI + LLM
- 104. Gen AI + LLM
- 105. Gen AI + LLM BIM RAG Accuracy = 41% (7 / 17) Domain specific RAG challenge, direction • Limited input context for LLM • No vocaboary • Long token distance • Hallucination
- 106. Gen AI + LLM how to make LLM dataset
- 107. Gen AI + LLM how to make LLM dataset
- 108. Gen AI + LLM how to make LLM dataset
- 109. Gen AI + LLM how to make LLM dataset
- 110. Gen AI + LLM how to make LLM dataset
- 111. Gen AI + LLM domain knowledge fine tuning. ex) BIM-GIS No fine tuning
- 112. Gen AI + LLM domain knowledge fine tuning. ex) BIM-GIS
- 113. Gen AI + LLM
- 114. Gen AI + LLM fine tuning. Epoch = 3 (90 min)
- 115. Gen AI + LLM fine tuning. Epoch = 10 (over 5 hours)
- 116. Journey breadcrumbs
- 117. guide 생성AI 시대 BIM 기술 동향과 해외 스마트 건설 사례 – 이강, 정숭용 생성AI LLM과 스테이블 디퓨전 최신 기술 및 활용 동향 - 최돈현, 김태영
- 121. Conclusion
- 122. conclusion • Google, Meta, MS 등 선진국 테크기업 중심 생성AI 잠재력 테스트 중 • 파운데이션 모델 개발은 이미 선진국 테크기업 및 중국 중심 • AI 개발 시 빅테크기업 공개 플랫폼&도구, Huggingface, W&B 등 사용 은 필수. 아키텍처 모델 직접 개발은 극소수. • LLM 은 멀티모달, 다중 에이전트 플랫폼으로 발전 • GPU 리소스 한계로 RAG, finetuning 등 다양한 기술 발전 • 온라인 문제로 생성AI가 스마트폰, 노트북 같은 모바일 장치에 내장되 는 추세 • Langchain과 같은 RAG, 에이전트 기술 테크 기업 출현 • 일부 선진국 중심으로 생성AI 기반 건설 테크 기업 출현 • 생성AI가 제대로 동작될려면 챌린지와 연구가 많이 남아 있음
- 123. conclusion
- 124. conclusion Hitchhiker's Guide - Earth Destroyed and Guide Join Gen AI world