![What LLMs cannot do well yet?
Travel planning
[J. Xie et al, TravelPlanner: A Benchmark for Real-World Planning with Language Agents, ICML’24 (Spotlight)]](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-4-320.jpg)
![What LLMs cannot do well yet?
[J. Xie et al, TravelPlanner: A Benchmark for Real-World Planning with Language Agents, ICML’24 (Spotlight)]](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-5-320.jpg)
![Using SoTA LLMs for Travel Planning (not
great)
First tool use,
Then plan the travel
Ground-truth tool use,
Then plan the travel
Even SoTA LLMs struggle for such hard planning problems
[J. Xie et al, TravelPlanner: A Benchmark for Real-World Planning with Language Agents, ICML’24 (Spotlight)]](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-6-320.jpg)
![GPT-4-turbo %
[J. Xie et al, TravelPlanner: A Benchmark for Real-World Planning with Language Agents, ICML’24 (Spotlight)]
How about o1?](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-7-320.jpg)
![LLM planning is still a hard problem
Number of Cities
Number of People
Trip planning
Meeting planning
[H. S. Zheng et al, NATURAL PLAN: Benchmarking LLMs on Natural Language Planning, arXiv’24]](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-8-320.jpg)
![Option One: The Scaling Law
More data
More compute
Larger models
Does that work for
reasoning/planning?
Very expensive
[J. Hoffmann*, S. Borgeaud*, A. Mensch* et al, Training Compute-Optimal Large Language Models]](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-11-320.jpg)
![Multi-round Dialogs to Collect Information
User has hidden constraints,
how to figure out?
🡪 Proactively ask!
[Jiang et al, Towards Full Delegation: Designing Ideal Agentic Behaviors for Travel Planning]
(b) APEC-Travel Agent](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-16-320.jpg)
![Searchformer: A* Search as a Token
Prediction Task
0 1 2
2
1
0
Start
Goal
Plan step
Frontier state
Closed state
[L. Lehnert, et al, Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrapping, COLM’24]
Wall](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-18-320.jpg)
![DualFormer (Searchformer
v2)
[D. Su et al, Dualformer: Controllable Fast and Slow Thinking by Learning with Randomized Reasoning Traces, arXiv’24]](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-31-320.jpg)
![Solve the Combinatorial Problem in the Latent
Space
Original Space Latent Space
Nonlinear optimization with
combinatorial constraints Surrogate optimization
combinatorial
constraints
solved by existing combinatorial solvers
[A. Ferber et al, SurCo: Learning Linear Surrogates For Combinatorial Nonlinear Optimization Problems, ICML’23 and outstanding paper in SODS workshop]](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-41-320.jpg)
![SurCo: Surrogate combinatorial opt
•
[A. Ferber et al, SurCo: Learning Linear Surrogates For Combinatorial Nonlinear Optimization Problems, ICML’23 and outstanding paper in SODS workshop]](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-43-320.jpg)
![[A. Zharmagambetov et al, Landscape Surrogate: Learning Decision Losses for Mathematical Optimization Under Partial Information, NeurIPS’23]
Limitation of SurCo
[A. Ferber et al, GenCO: Generating Diverse Solutions to Design Problems with Combinatorial Nature, ICML’24]](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-50-320.jpg)
![Concrete Example: Modular Addition
[T. Zhou et al, Pre-trained Large Language Models Use Fourier Features to Compute Addition]
Does neural network have an implicit table to do retrieval?](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-54-320.jpg)
![Concrete Example: Modular Addition
Learned representation = Fourier basis 🤯
Why? 🤔
[T. Zhou et al, Pre-trained Large Language Models Use Fourier Features to Compute Addition]
Does neural network have an implicit table to do retrieval?](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-55-320.jpg)
![Problem Setup
One-hot(a)
Bottom layer
Top layer
[Y. Tian, Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets, arXiv’24]](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-56-320.jpg)
![What a Gradient Descent Solution look
like?
Frequency
Hidden node index
[Y. Tian, Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets, arXiv’24]](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-58-320.jpg)
![Symmetry due to
Hermitian condition
Order-6
solutions
What a Gradient Descent Solution look
like?
[Y. Tian, Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets, arXiv’24]](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-59-320.jpg)
![Order-6
Order-4
What a Gradient Descent Solution look
like?
[Y. Tian, Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets, arXiv’24]](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-60-320.jpg)
![Order-4 and order-6
solutions really happen!
More Statistics on Gradient Descent
Solutions
[Y. Tian, Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets, arXiv’24]](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-61-320.jpg)
![Stronger
weight decay
Effect of Weight
Decay
[Y. Tian, Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets, arXiv’24]](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-62-320.jpg)
![Structure of Loss Functions
[Y. Tian, Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets, arXiv’24]](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-64-320.jpg)
![Structure of Loss Functions
Sufficient conditions of Global Optimizers:
[Y. Tian, Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets, arXiv’24]](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-65-320.jpg)
![Composing Global Optimizers from Partial
Ones
[Y. Tian, Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets, arXiv’24]](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-74-320.jpg)
![Composing Global Optimizers from Partial
Ones
[Y. Tian, Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets, arXiv’24]](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-75-320.jpg)
![Composing Global Optimizers from Partial
Ones
[Y. Tian, Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets, arXiv’24]](https://image.slidesharecdn.com/8-250108072535-72b186c4/85/Toward-unified-framework-and-symbolic-decision-making-Berkeley-LLM-AI-Agents-MOOC-76-320.jpg)
Toward unified framework and symbolic decision making - Berkeley LLM AI Agents MOOC
- 1. Towards a unified framework of Neural and Symbolic Decision Making Yuandong Tian Research Scientist Director Meta AI (FAIR)
- 2. Large Language Models (LLMs) Conversational AI Content Generation AI Agents Reasoning Planning
- 3. Large Language Models (LLMs) Conversational AI Content Generation AI Agents Reasoning Planning
- 4. What LLMs cannot do well yet? Travel planning [J. Xie et al, TravelPlanner: A Benchmark for Real-World Planning with Language Agents, ICML’24 (Spotlight)]
- 5. What LLMs cannot do well yet? [J. Xie et al, TravelPlanner: A Benchmark for Real-World Planning with Language Agents, ICML’24 (Spotlight)]
- 6. Using SoTA LLMs for Travel Planning (not great) First tool use, Then plan the travel Ground-truth tool use, Then plan the travel Even SoTA LLMs struggle for such hard planning problems [J. Xie et al, TravelPlanner: A Benchmark for Real-World Planning with Language Agents, ICML’24 (Spotlight)]
- 7. GPT-4-turbo % [J. Xie et al, TravelPlanner: A Benchmark for Real-World Planning with Language Agents, ICML’24 (Spotlight)] How about o1?
- 8. LLM planning is still a hard problem Number of Cities Number of People Trip planning Meeting planning [H. S. Zheng et al, NATURAL PLAN: Benchmarking LLMs on Natural Language Planning, arXiv’24]
- 9. What are the Solutions?
- 10. What are the Solutions? Option One: Scaling Law Option Two: Hybrid System Deep Models Solver End2end Deep Models Solver Provide data Deep Models Solver Call deep models (policy, values) Option Three: Emerging Symbolic Structure from Neural network
- 11. Option One: The Scaling Law More data More compute Larger models Does that work for reasoning/planning? Very expensive [J. Hoffmann*, S. Borgeaud*, A. Mensch* et al, Training Compute-Optimal Large Language Models]
- 12. Option Two: Hybrid Systems Deep Models Solver End2end Deep Models Solver Provide data Deep Models Solver Tool use
- 13. Option Two: Hybrid Systems Deep Models Solver End2end Deep Models Solver Provide data Deep Models Solver Tool use
- 14. Language-Driven Guaranteed Travel Planning LLMs can not handle too many constraints? -> Combinatorial Solvers can! • Realistic dataset: collect from the real world • User instruction translator: Fine-tuned LLM to convert user request into symbolic description, augmented by flight/hotel information from database. • Impose constraints and formalize the travel planning as Mixed Integer Linear Programming (MILP). • Build a combinatorial solver to give optimal solution. Ju et al, To the Globe (TTG): Towards Language-Driven Guaranteed Travel Planning (EMNLP’24 Demo)
- 15. Experiments (End-to-end Human Evaluation) Net Prompter Scores (NPS) and its breakdown in three dimensions: satisfaction, value and efficiency. Ju et al, To the Globe (TTG): Towards Language-Driven Guaranteed Travel Planning (EMNLP’24 Demo)
- 16. Multi-round Dialogs to Collect Information User has hidden constraints, how to figure out? 🡪 Proactively ask! [Jiang et al, Towards Full Delegation: Designing Ideal Agentic Behaviors for Travel Planning] (b) APEC-Travel Agent
- 17. Option Two: Hybrid Systems Deep Models Solver End2end Deep Models Solver Provide data Deep Models Solver Tool uses
- 18. Searchformer: A* Search as a Token Prediction Task 0 1 2 2 1 0 Start Goal Plan step Frontier state Closed state [L. Lehnert, et al, Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrapping, COLM’24] Wall
- 19. Searchformer: A* Search as a Token Prediction Task <trace><plan> bos create 0 2 c0 c3 close 0 2 c0 c3 create 0 1 c1 c2 close 0 1 c1 c2 create 0 0 c2 c1 create 1 1 c2 c1 close 0 0 c2 c1 create 1 0 c3 c0 close 1 0 c3 c0 plan 0 2 plan 0 1 plan 0 0 plan 1 0 eos 0 1 2 2 1 0 Start Goal Plan step Frontier state Closed state Wall <prompt> bos start 0 2 goal 1 0 wall 1 2 wall 2 0 eos
- 20. Train a Transformer to predict the next token via teacher forcing. Training Method Encoder <prompt> <trace><plan> Decoder Encoder <prompt> <plan> Decoder Solution-Only Model Search-Augmented Model Model (100-400 tokens) (100-6500 tokens)
- 22. Search-Augmented vs. Solution-Only Models 30x30 Maze Navigation
- 23. Search-Augmented vs. Solution-Only Models 30x30 Maze Navigation
- 24. Search-Augmented vs. Solution-Only Models 30x30 Maze Navigation Search-augmented is much more parameter & data efficient!
- 25. Search-Augmented vs. Solution-Only Models Search-augmented is much more parameter & data efficient! Sokoban
- 26. How to go beyond? Imitation Learning Fine-tuning Using solver’s trace to train the Transformer with teacher forcing Fine-tune the model to achieve shorter trace but still leads to optimal plan! (Reinforcement Learning task) Search-augmented Models Searchformer
- 27. Beyond A*: Improving search dynamics via bootstrapping
- 28. Repeated bootstrapping increases the Improved Length Ratio (ILR) Improving search dynamics via bootstrapping
- 29. Fine-tuning improves performance initially. Improving search dynamics via bootstrapping
- 30. Searchformer outperforms largest solution-only model. Improving search dynamics via bootstrapping
- 31. DualFormer (Searchformer v2) [D. Su et al, Dualformer: Controllable Fast and Slow Thinking by Learning with Randomized Reasoning Traces, arXiv’24]
- 32. DualFormer (Searchformer v2) Dualformer automatically switches between fast mode (System 1) and slow mode (System 2) and works better for dedicated models on either modes.
- 33. Fast mode performance Slow mode performance
- 35. Math Problems
- 37. Option Two: Hybrid Systems Deep Models Solver End2end Deep Models Solver Provide data Deep Models Solver Tool uses
- 38. Nonlinear objective with combinatorial constraints •Real-world domains: • Computer system planning • Designing photonic devices • Throughput optimization • Antenna design • Energy grid Combinatorial feasible region
- 39. Example: Embedding Table Placement •
- 40. Example: Embedding Table Placement • Formulation
- 41. Solve the Combinatorial Problem in the Latent Space Original Space Latent Space Nonlinear optimization with combinatorial constraints Surrogate optimization combinatorial constraints solved by existing combinatorial solvers [A. Ferber et al, SurCo: Learning Linear Surrogates For Combinatorial Nonlinear Optimization Problems, ICML’23 and outstanding paper in SODS workshop]
- 42. Solve the Combinatorial Problem in the Latent Space Original Space Latent Space Nonlinear optimization with combinatorial constraints Surrogate optimization combinatorial constraints solved by existing combinatorial solvers Proposal: gradient-based optimization
- 43. SurCo: Surrogate combinatorial opt • [A. Ferber et al, SurCo: Learning Linear Surrogates For Combinatorial Nonlinear Optimization Problems, ICML’23 and outstanding paper in SODS workshop]
- 44. Gradient-based Optimization • Assumed differentiable Recent work on differentiable optimization Differentation of blackbox optimizers CVXPYLayers MIPaaL Etc. Assumed differentiable
- 45. Embedding Table Sharding •Public Deep Learning Recommendation Model (DLRM dataset) placing between 10 to 60 tables on 4 GPUs •Baseline: Greedy •SoTA: RL approach Dreamshard1 •SurCo: Surrogate NN model learned via CVXPYLayers (differentiable LP Solver) 1 Zha et al. NeurIPS 2022 Dataset: https://github.com/facebookresearch/dlrm_datasets
- 46. Results – Table Sharding
- 48. Inverse Photonic Design •Dataset: Ceviche Challenges1 •Most baselines don’t work here due to combinatorial constraints •SoTA: Brush-based algorithm 1 •SurCo: Surrogate learned via blackbox differentiation2 of brush solver 1 Schubert et al. ACS Photonics 2022 2 Vlastelica et al. ICLR 2019 Dataset: https://github.com/google/ceviche-challenges Wavelength division multiplexe Mode converter Beam splitter Waveguide bend
- 49. Inverse photonics Convergence comparison + Solution example Takeaways: - SurCo-Zero finds loss-0 solutions quickly - SurCo-Hybrid uses offline training data to get a head start Wavelength division multiplexer
- 50. [A. Zharmagambetov et al, Landscape Surrogate: Learning Decision Losses for Mathematical Optimization Under Partial Information, NeurIPS’23] Limitation of SurCo [A. Ferber et al, GenCO: Generating Diverse Solutions to Design Problems with Combinatorial Nature, ICML’24]
- 51. Option Three: Does Deep Model Actually Converge to Anything Symbolic? Deep Models Emerging Symbolic Structure
- 52. https://medium.com/@fenjiro/large-language-models-llms-emergent-abilities-chatgpt-talks-moroccan-dialect-as-an-example-c945f93aa63a LLM shows emergent behaviors!! Debate: Is LLM doing retrieval or true reasoning?
- 53. Debate: Is LLM doing retrieval or true reasoning? LLM is just doing retrievals!!
- 54. Concrete Example: Modular Addition [T. Zhou et al, Pre-trained Large Language Models Use Fourier Features to Compute Addition] Does neural network have an implicit table to do retrieval?
- 55. Concrete Example: Modular Addition Learned representation = Fourier basis 🤯 Why? 🤔 [T. Zhou et al, Pre-trained Large Language Models Use Fourier Features to Compute Addition] Does neural network have an implicit table to do retrieval?
- 56. Problem Setup One-hot(a) Bottom layer Top layer [Y. Tian, Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets, arXiv’24]
- 57. (Scaled) Fourier Transform Hermitian condition holds
- 58. What a Gradient Descent Solution look like? Frequency Hidden node index [Y. Tian, Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets, arXiv’24]
- 59. Symmetry due to Hermitian condition Order-6 solutions What a Gradient Descent Solution look like? [Y. Tian, Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets, arXiv’24]
- 60. Order-6 Order-4 What a Gradient Descent Solution look like? [Y. Tian, Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets, arXiv’24]
- 61. Order-4 and order-6 solutions really happen! More Statistics on Gradient Descent Solutions [Y. Tian, Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets, arXiv’24]
- 62. Stronger weight decay Effect of Weight Decay [Y. Tian, Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets, arXiv’24]
- 63. Why? 🤔
- 64. Structure of Loss Functions [Y. Tian, Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets, arXiv’24]
- 65. Structure of Loss Functions Sufficient conditions of Global Optimizers: [Y. Tian, Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets, arXiv’24]
- 66. How to Optimize? The objective is highly nonlinear !! However, nice algebraic structures exist!
- 67. How to Optimize? The objective is highly nonlinear !! However, nice algebraic structures exist!
- 68. How to Optimize? The objective is highly nonlinear !! However, nice algebraic structures exist!
- 74. Composing Global Optimizers from Partial Ones [Y. Tian, Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets, arXiv’24]
- 75. Composing Global Optimizers from Partial Ones [Y. Tian, Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets, arXiv’24]
- 76. Composing Global Optimizers from Partial Ones [Y. Tian, Composing Global Optimizers to Reasoning Tasks via Algebraic Objects in Neural Nets, arXiv’24]
- 77. Exemplar constructed global optimizers Order-4 (2*2, mixed with order-6) Perfect memorization (order-d per frequency)
- 78. Exemplar constructed global optimizers Perfect memorization (order-d per frequency)
- 79. Exemplar constructed global optimizers Perfect memorization (order-d per frequency)
- 80. Gradient Descent solutions matches with construction
- 81. Gradient Descent solutions matches with construction 100% of the per-freq solutions are order-4/6
- 82. Gradient Descent solutions matches with construction 95% of the solutions are factorizable into “2*3” or “2*2”
- 83. Gradient Descent solutions matches with construction Factorization error is very small
- 84. Gradient Descent solutions matches with construction 98% of the solutions can be factorizable into the constructed forms
- 85. Gradient Descent solutions matches with construction Distribution of the parameters in the solutions
- 86. Possible Implications Do neural networks end up learning more efficient symbolic representations that we don’t know? Does gradient descent lead to a solution that can be reached by advanced algebraic operations? Will gradient descent become obsolete, eventually?
- 87. Thanks! 87