2024-OCT-23 NYC Meetup - Unstructured Data Meetup - Unstructured Halloween
1 like203 views
The document outlines details about an unstructured data meetup hosted by Zilliz, featuring talks on topics like vector search and metadata lakes. It also promotes the Milvus open-source vector database and emphasizes the importance of a respectful community and active participation. Additionally, it invites participants to connect on Discord, share their experiences, and highlights job opportunities at Zilliz.
2024-OCT-23 NYC Meetup - Unstructured Data Meetup - Unstructured Halloween
- 1. 1 | © Copyright 2024 Zilliz 1 Presented by: October 2024 New York Unstructured Data Meetup
- 2. 2 | © Copyright 2024 Zilliz 2 Slides https://www.slideshare.net/bunkertor/pr esentations
- 3. 3 | © Copyright 2024 Zilliz 3 3 | © Copyright 10/22/23 Zilliz 3 | © Copyright 2024 Zilliz Tim Spann Principal Developer Advocate, Zilliz tim.spann@zilliz.com https://www.linkedin.com/in/timothyspann/ https://x.com/PaaSDev Unstructured Data Meetup | Host
- 4. 4 | © Copyright 2024 Zilliz 4 Code of Conduct Be respectful and kind When communicating with all event participants, speakers, and hosts. Be considerate All ideas are welcome Be present and participate actively in discussions. Ask questions and reach out for help when needed. Report inappropriate behavior Any inappropriate behavior is not tolerated at this event. Inform a Zilliz team member immediately if you see any behavior deemed inappropriate
- 5. 5 | © Copyright 2024 Zilliz 5 5 | © Copyright 10/22/23 Zilliz 5 | © Copyright 2024 Zilliz Milvus Open Source Self-Managed Zilliz Cloud SaaS Fully-Managed github.com/milvus-io/milvus Getting Started with Vector Databases zilliz.com/cloud
- 6. 6 | © Copyright 2024 Zilliz 6 Zilliz is Hiring! Join our Team Zilliz.com/careers • Developer Advocate • Senior Software Engineer • Staff Software Engineer • Solutions Architect
- 7. 7 | © Copyright 2024 Zilliz 7 Join the Milvus Discord!
- 8. 8 | © Copyright 2024 Zilliz 8 Become a Speaker! Interesting in speaking at and/or sponsoring a Zilliz Unstructured Data Meetup? Fill out this form! 🎤🎤🎤
- 9. 9 | © Copyright 2024 Zilliz 9 Have you built something cool using Milvus or Zilliz? We want to hear all about it. Share Your Story
- 10. 10 | © Copyright 2024 Zilliz 10 Star Milvus for a chance to win a prize tonight!
- 11. 11 | © Copyright 2024 Zilliz 11 Share your photos! #ZillizUnstructuredData @zilliz_universe, @milvusio Zilliz, Milvus
- 12. 12 | © Copyright 2024 Zilliz 12 12 | © Copyright 10/22/23 Zilliz 12 | © Copyright 2024 Zilliz Welcome Speakers Introduction to Vector Search Metadata Lakes for Next-Gen AI/ML Unstructured Document Data Extraction at Scale with LLMs: Challenges and Solutions TECH TALK 1 TECH TALK 2 TECH TALK 3 Uri Goren CEO, Argmx Lisa N. Cao Product Manager, Datastrato Naren H Co-founder/COO, Unstract
- 13. 13 | © Copyright 2024 Zilliz 13 Join us at our next meetup! lu.ma/unstructured-data-meetup
- 14. 14 | © Copyright 2024 Zilliz 14 14 | © Copyright 10/22/23 Zilliz 14 | © Copyright 2024 Zilliz Quick Intro to Halloween Unstructured
- 15. 15 | © Copyright 2024 Zilliz 15 15 | © Copyright 10/22/23 Zilliz 15 | © Copyright 2024 Zilliz
- 16. 16 Vote on Dates for Future Meetups January February March April May June July August September October November December 10 11 12 13 14 15 16 17 18 19 20 21 11 12 13 14 15 16 17 18 19 20 21 22
- 17. Upcoming Women in Tech Hackathon
- 18. 18 | © Copyright Zilliz 18 01 Introduction
- 19. 19 | © Copyright Zilliz 19 About Milvus Milvus is an open-source vector database for GenAI projects. pip install on your laptop, plug into popular AI dev tools, and push to production with a single line of code. 29K GitHub Stars 25M Downloads 250 Contributors 2,600 Forks Easy Setup Pip-install to start coding in a notebook within seconds Integration Plug into OpenAI, Langchain, LlmaIndex, and many more Reusable Code Write once, and deploy with one line of code into the production environment Feature-rich Dense & sparse embeddings, filtering, reranking and beyond
- 20. 20 | © Copyright Zilliz 20 02 Hybrid Search
- 21. 21 | © Copyright Zilliz 21 Hybrid Search ● https://zilliz.com/blog/metadata-filtering-hybrid-search-or-agent-in-rag-applications
- 22. 22 | © Copyright Zilliz 22 Hybrid Search Support the fusion of vector search and full-text search Support the fusion of multimodal vectors from various unstructured data types such as images, videos, audio, and text files Utilize various types of vector embeddings. This includes dense embeddings from models like BERT and Transformers and sparse embeddings from algorithms like BM25, BGEM3, and SPLADE.
- 23. 23 | © Copyright Zilliz 23 Hybrid Search ● Milvus supports the creation of up to 10 vector fields for the same dataset within a single collection. Based on this support, hybrid search allows users to search across multiple vector columns simultaneously. This capability allows for combining multimodal search, hybrid sparse and dense search, and hybrid dense and full-text search, offering versatile and flexible search functionality. ● These vectors in different columns represent diverse facets of data, originating from different embedding models or undergoing distinct processing methods. The results of hybrid searches are integrated using various re-ranking strategies.
- 24. 24 | © Copyright Zilliz 24 Hybrid Search This feature enables different columns to: ● Represent multiple perspectives of information. For instance, in e-commerce, product images include front, side, and top views. Different views can be represented with different types or dimensions of vectors. ● Utilize various types of vector embeddings. This includes dense embeddings from models like BERT and Transformers and sparse embeddings from algorithms like BM25, BGE-M3, and SPLADE. ● Support the fusion of multimodal vectors from various unstructured data types such as images, videos, audio, and text files. For example, in criminal investigations, suspects can be represented through biometric modalities such as fingerprints, voiceprints, and facial recognition, aiding in identifying individuals across different modalities. ● Support the fusion of vector search and full-text search. https://milvus.io/docs/multi-vector-search.md
- 25. 25 | © Copyright Zilliz 25 When is Hybrid Search Recommended? Hybrid search is ideal for complex situations demanding high accuracy, especially when an entity can be represented by multiple, diverse vectors. This applies to cases where the same data, such as a sentence, is processed through different embedding models or when multimodal information (like images, fingerprints, and voiceprints of an individual) is converted into various vector formats. By assigning weights to these vectors, their combined influence can significantly enrich recall and improve the effectiveness of search results.
- 26. 26 | © Copyright Zilliz 26
- 27. 27 | © Copyright Zilliz 27 https://bit.ly/4eFdMlK https://bit.ly/3BLeLCx
- 28. 28 | © Copyright Zilliz 28 https://bit.ly/3zXW8dX https://bit.ly/3NuK5ru
- 29. 29 | © Copyright Zilliz 29 https://bit.ly/4gZ4Lpn Metadata Filtering Hybrid Search Agents https://bitz.ly/3UbqUqx
- 30. 30 | © Copyright Zilliz 30 https://bit.ly/3YpKd1K Smart Chunking Embedding Model Choice Improving Information Retrieval and RAG with Hypothetical Document Embeddings HyDE https://bit.ly/4f8Ckne
- 31. 31 | © Copyright Zilliz 31 Best Practices in RAG Apps https://zilliz.com/blog/best-practice-in-implementing-rag-apps
- 32. 32 | © Copyright Zilliz 32 | © Copyright Zilliz 32 RESOURCES
- 33. 33 | © Copyright 2024 Zilliz 33 33 This week in Milvus, Towhee, Attu, GPT Cache, Gen AI, LLM, Apache NiFi, Apache Flink, Apache Kafka, ML, AI, Apache Spark, Apache Iceberg, Python, Java, Vector DB and Open Source friends. https://bit.ly/32dAJft https://github.com/milvus-io/milvus AIM Weekly by Tim Spann
- 34. 34 | © Copyright 2024 Zilliz 34 milvus.io github.com/milvus-io/ @milvusio @paasDev /in/timothyspann Connect with me! Thank you!
- 35. 35 | © Copyright 2024 Zilliz 35 Join us at our next meetup! meetup.com/unstructured-data-meetup- new-york/
- 36. 36 | © Copyright Zilliz 36 T H A N K Y O U