![Marco De Nittis
marco.denittis [a] gmail.com
LLM-ize your apps 101](https://image.slidesharecdn.com/aiheroes-2023-llm101-slideshare-231204171741-a5526d6e/85/Integrate-LLM-in-your-applications-101-1-320.jpg)
![thank you!
Marco De Nittis
marco.denittis [a] gmail.com
https://speakerscore.it/AIHEROES-101
https://github.com/mdnmdn/aiheroes-2023](https://image.slidesharecdn.com/aiheroes-2023-llm101-slideshare-231204171741-a5526d6e/85/Integrate-LLM-in-your-applications-101-28-320.jpg)
Integrate LLM in your applications 101
- 1. Marco De Nittis marco.denittis [a] gmail.com LLM-ize your apps 101
- 2. whoami • Independent Cloud Architect • Trainer • ❤ cloud, serverless, devops, AI, wasm • Curious and tinkerer
- 3. objectives • Explore an LLM app with no framework • Bash scripts / curl (almost) • Highlight fundamentals behaviours and building blocks
- 5. in the beginning… Everyone uses ChatGPT! We should do either to improve our business! We could ask to some professionals Too expensive! Alex the waiter could do it! He has a playstation and plays Fortnite every day
- 6. in the beginning… Ok, we need Python I hate snakes, no pythons here And also several cloud servers with GPUs Ok, use this, that’s the same
- 7. …then
- 8. fi rst try • Open AI API • Fair price • Rest HTTPS calls with auth token
- 9. do it!
- 10. fi rst try • Very easy to implement • No contextual responses
- 11. second try - context • Prompt engineering • Adding context with background informations • Mixing user text with “prepared contextual text”
- 12. prompt engineering • Several established techniques • Role: “you are an helpful restaurant assistant.. • Shots: providing examples • Ask for steps
- 13. do it!
- 14. second try - context • Tones and background are ok • No integrations with company informations
- 15. what is a LLM • Probabilistic engine: most probable text response from input (according to the corpus) • Black box with input and output pipe • No dynamic/short term memory • Context
- 16. context • Max input + output size • Measured in tokens • Group of letters • English 1.2 token/word • Typical size 4k - 32k, and beyond…
- 17. memory with context • Adding all the informations in each request • How "the chat" ChatGPT works • Limited to context size
- 18. RAG + external memory • Semantic memory • => retrieve the “most relevant” informations • Vector database + RAG: • Retrieval Augmented Generation • Include in context only the relevants • Easy to edit the memory
- 19. fi ne tuning • Enhancing the “cultural baggage” of the LLM • Train again the model with custom speci fi c data • Expensive (not so much) • Dif fi cult to “edit” the data in memory
- 20. embeddings • Place a text in a multidimensional “space of concepts" • Digest the informations • text => vector (of fl oats) • High dimensional vector (300+) • Relevancy => mathematical distance • A model “calculates” the embeds • Vector database to store & search
- 21. vector database • A database capable to handle vector of fl oats • Indexing • Search • Find distances (several algorithms) • Ad hoc db: Qdrant, Pinecone, Chroma, Milvus, … • Almost any common db: postgres, redis, sqlite, …
- 22. phases • Storing and indexing company info in a vector DB • Embeddings contents • Una tantum • Embeddings of user prompt • Search of relevant infos • Embed infos in the prompt
- 23. phases
- 24. do it!
- 25. improvements • Chunking input for large document • Check for prompt injection • feedback (or self feedback) for the response
- 26. improvements • Integrate LLM framework • Assistant base approach • An LLM coordinating the work of other LLMs • Needs a full- fl edged framework
- 27. point of attentions • Control costs • Input control • Predictability • Sustainability
- 28. thank you! Marco De Nittis marco.denittis [a] gmail.com https://speakerscore.it/AIHEROES-101 https://github.com/mdnmdn/aiheroes-2023