TMU_GDSC_20240509.pdfTMU_GDSC_20240509.pdf
- 2. Slide Code
- 3. Taipei Medical University 張庭維 ● 臺灣大學資工所AI碩一 張庭維 ● 專長：自然語言處理、 3D列印 ● 經歷 ○ 臺北醫學大學醫學系 103級 ○ 北醫附醫一般醫學科住院醫師 ○ 經濟部3D列印積層製造工程師 ○ TMU 3D printing teaching assistant ○ Medical Dialog Analysis Competition (17/368)
- 4. Outline 1. Introduction of NLP 2. Generative AI in NLP a. Basic introduction b. Applications 3. LLM agent a. Introduction of Multimodality b. VLM challenge 4. Prompt engineering 5. Hands-on Lab: a. gemini pro b. Lab: gemini pro vision c. Lab: challenge
- 15. Generative AI in NLP
- 16. What is Generative AI 李宏毅INTRODUCTION TO GENERATIVE AI 2024 SPRING https://speech.ee.ntu.edu.tw/~hylee/genai/2024-spring.php
- 17. What is not Generative AI 李宏毅INTRODUCTION TO GENERATIVE AI 2024 SPRING https://speech.ee.ntu.edu.tw/~hylee/genai/2024-spring.php
- 30. LLM agent
- 31. From LLM to LLM agent https://twitter.com/sundyme/status/1664847838793306113/photo/1
- 32. AI agent ● An agent is an entity that perceives and acts. ● Arational agent selects actions that maximize its (expected) utility. ● Characteristics of the percepts, environment, and action space dictate techniques for selecting rational actions
- 33. Types of AI agents ● Simple Reflex Agents ● Model-based reflex Agents ● Goal-based Agents ● Utility-based Agents ● Learning Agents https://www.javatpoint.com/types-of-ai-agents https://jimmystore.medium.com/%E4%BA%BA%E5%B7%A5%E6%99%BA%E6%85%A7%E7%AD%86%E8%A8%9 8-%E4%BA%8C-agent-2f6908b899c6
- 34. arXiv:2309.07864
- 39. Text-Image pairs
- 40. Issue of VLM ● To be specific, the captions in the LAION dataset, used to train vision language models, come from alt HTML tags (Alttext) and often describes only a narrow aspect of the image, neglecting significant visual details. ● For example, an image of a person can have as Alttext the name of the person and the name of the photographer, but not a description of their appearance, their clothes, their position, or the background. arXiv:2310.16656
- 43. Write clear and specific instructions https://learn.deeplearning.ai/courses/chatgpt-prompt-eng/lesson/1/introduction 李宏毅INTRODUCTION TO GENERATIVE AI 2024 SPRING https://speech.ee.ntu.edu.tw/~hylee/genai/2024-spring.php
- 50. 0-shot Chain of Thought arXiv:2201.11903 arXiv:2205.11916v4
- 51. Take a deep breath arXiv:2309.03409
- 54. Hands-on lab
- 55. Google Gemini Pro API https://ai.google.dev/
- 64. Challenge
- 66. 自由分享
- 67. Useful resource ● ChatGPT: https://chatgpt.com/ ● Claude: https://claude.ai/chats ● Gemini: https://gemini.google.com/ ● Google AI studio: https://aistudio.google.com/ ● 李宏毅課程: https://www.youtube.com/@HungyiLeeNTU ● Deeplearning.ai short course: https://www.deeplearning.ai/short-courses/ ● Prompt engineering: https://www.promptingguide.ai/
- 68. 宣傳