Uday Presentation prompt - A sample from LLMs generated content
2. Tell us about the most complex and difficult project you ran
An Ad optimization project in media.net with high stakes
- Improve display ads P&L by increasing quality
- plus , optimize spends
Why complex?
- People + product + stakeholders and lots at stake.
- Early in the company, need to take not so popular calls
Action(s)
- Changed the direction of spam system,
- Took ad optimizations systems live
- xFN, xPA collaboration.
Result:
10-12% ROI growth in 6-8 months , sustained !
Cultural Shift - ROI based research and generic approach.
3. Tell us when you struggled to manage an individual or team
Not struggling but challenging. Took time
A Director .. stakeholder @ Amazon
- Goals and incentives did not align. Baggage from historical org conflicts.
- Were not vocal about reasons for non-alignment
- Key to the success of my org and users of Alexa/Amazon.
Action(s)
- Tried to understand the reasons. Create a win-win and walk the talk.
- Aligned rest of the org first and showed outcomes instead talking.
- Offered help and made them realise why earlier leads took some calls.
- Made efforts to promise long term wins
- Looked as an ally rather than a competitor
Result:
Alignment happened slowly and Alexa payments launched with press and SVP Dave limp;s
appreciations. The teams still collaborate well now.
4. How would you go about implementing culture that management proposed ?
Understand the culture first.. Is it feeding into the outcomes and goals?
Philosophy and Actions.
- Culture is not a 0 to 1. It evolves. It;s a journey.
- Two great companies with totally different DNAs can be successful. (Amzon vs Google)
- Evolve the culture. Be open for feedback and opportunities.
- Be a role model. Show how it helps in right outcomes.
- Share the though process. Be an evangelist of culture that you believe in.
- Hire - Bring in culture ads, not just culture fits.
- Nurture - It takes time and it;s bi directional
- Act on culture misfits ! Dont wait too long.
5. How would you partner with demanding customers to build a non-existent product?
Understand the customers. What are the pain points?
Be a problem solver ! Without writing a line of code!! w/o building product first.
- Earn the trust
- Make customers part of the process. Don't solve independently.
- Think about the larger potential base not just the active base.
- Explain the constraints.
- Show value … iteratively
- Deliver the best outcomes and repeat.
An example which also applies to next topic of discussion
6. Tell us about a time when you demonstrated resilience.
7. Universal SQL System Design, Plan and POC
AMA (Ask Me Anything) from EMA
Detailed Plan
8. 30,000 feet view
1. Authentication and Authorization: User authentication and role based access to data
1. Configuration interface - AD/Oauth configs, Role based access rules for data
2. NL Queries Management: Users input the queries, choose the format of results and gets appropriate
responses.
1. [P0] Real-Time Queries
1. Query Request Journey - Query editor / templated/previously ran query
2. Query Response Journey - simple text or tabular or infographic response
2. [P1] Scheduled Queries - daily, weekly, monthly, quarterly scheduled natural queries.
3. [P2] Advanced Query capabilities
1. Collaborative queries / Frequently asked queries on similar data sets
2. Generative Queries - Pro Tips, Try this query etc.
3. AMA Engine Core: NL Queries to Results Core system requirements
1. AMA - Configuration & Data preparation
1. [P0] Admin interface to setup connectors, test and update.
2. AMA - Uber Orchestrator: Define NL query work flows and execute
3. AMA - NL Query understanding
1. Natural language to SQl query conversion
4. AMA - SQL Result Orchestration: Choose underlying LLM, Call and filter result set.
5. AMA - Resultset Augmentation (RAG) & Response
10. MVP Scope & Milestones
1. [Prototype - POC] build an end to end AMA core flow with hardcoded data definition fields from one
organization. Support a tabular and CSV format for response to start with.
2. Finalize design.
3. Identify and plan for bottlenecks and risks.
4. Prepare task list, estimate and move on.
5. [Iteratively]
a. Build basic content security and augmentation.
b. Make the design multi tenant and end to end
c. At this point we are ready for multiple customers' feedback as early adopters.
d. Build interfaces and make this iterative.
e. Build all user facing features and error cases.
f. Think of on-cloud and on-device based on customer types.
g. Launch / deployment readiness
h. Launch in beta
6. ML Quality improvements (To meet Launch bar)
11. Success Metrics
Topline metrics:
1. Daily active users, monthly active users, L28 etc.
2. queries per day,
3. Active tenants per day.
4. Monthly revenue/billing (total, per tenant etc.)
5. CSAT for feature
System metrics (quality):
6. Query response quality:
a. NL query success rate (precision/recall)
b. NL query to SQL success
c. SQL resolution success rate
d. SQL execution error rate
7. Overall Quality:
a. Human eval feedback (80%+ positive rating)
b. User inline feedback/reports
System metrics (performance):
1. Low level system logging
a. Each API call, with params
b. Candidates obfuscated
c. Ranking logs
d. Feedback logs
2. System health/perf:
a. AMA API throughput
b. LLM API throughput
c. AMA API Latency (P99, P90
etc)
d. queries per second
e. Uptime / availability etc.
12. Project Execution Plan (Detailed)
Milestone/Workstream SWEs Workstream
start time
Duration (weeks)
Prototype 2 T 3 weeks
AMA Engine (Queries) 3 T + 3 weeks 6 weeks
AMA Engine (Responses) 3 T + 3 weeks 4 weeks
AMA Engine (Quality/data
pipelines)
3 T + 9 weeks
AMA APIs/Integrations 1 T + 6 weeks 4 weeks
Deployments/Rollup
(SDE/SRE)
1 T + 12 weeks N/A
Admin
interfaces/Configurations
2 T + 4 weeks
Goal: Launch with above MVP features for 4 early adopter clients (2 cloud and 2 on premise).
On premise is a stretch goal. We will end up launching for the cloud as priority but design for on-prem as a test bed.
Resourcing Plan:
Total MVP story points: ~90
each at approx 1 week estimated.
Total: ~ 100 SWE weeks.
Total : 8-9 members.
Work across streams as needed.
Start with 3, grow to 8 and sustain.