Speero

Most conferences are about who’s on stage. Experimentation Live 2025 (Oct 21-22, Austin, TX) is about who’s in the room. - No endless keynotes.  - No sales pitches.  - No passive listening. Just real conversations with people actually solving the same problems you are. Here’s how it works: - Short, no-BS keynotes to kick off discussions. - Small-group roundtables (5-10 people) where you actually get to talk. - Plenty of time to connect between sessions. - Networking that doesn’t stop at 5 PM – dinner, drinks, and conversations that keep going. If you hate typical conferences, you’ll probably love this one. Early bird 25% off → Code: EARLYBIRD25 (valid until first 50 sold). Consultants bringing a client? Email ben@speero.com for an even bigger discount. See you in Austin EXL 2025: By Speero and Eppo by Datadog.

Does your testing or ops program have a strategy document? If not, this will help. The most important sections are the purpose and mission, and this document outlines the vision for democratizing an experimentation culture. It's designed to answer key questions like: "How might we concisely communicate and align on the goal, function, and direction of the experimentation program?" "What parts of the program are we working on to improve the speed of testing, who owns these initiatives, and what are they measured by?" The goal is to embrace a learning and data-driven approach, and experimentation as the key to your success. Yes, experimentation is an umbrella term that includes things like causal inference modeling and MVP bets, but this document first addresses the need for a culture where we A/B test more often to get quicker feedback. You have an opportunity to help other teams understand the value of testing and embrace it to benefit in a similar way. To do this, you need to foster a culture that views testing and failure as a necessity, not a hindrance. Key question in your document: "We can set up various mechanisms to enable teams to test more often, but we likely need a 'forcing function' to incentivize every team to engage in testing. What incentive can we put in place for teams to adopt and embrace experimentation?"

Every event seems the same: you show up, listen to some keynotes you could find on YouTube, and leave without making any meaningful connections. The truth is, all of us attend conferences with specific challenges and goals, but random encounters rarely lead to high-value connections or actionable solutions. The greatest return on investment with events comes from building relationships, not from watching recycled content. That's why we've designed Experiment Live 2025 as a deliberate counterpoint to this. Our main focus is to help you make lasting connections. We "matchmake" you with peers who share the same challenges and opportunities through our attendee surveys. You'll then sit at small tables—no more than 8 people—to discuss those issues, learn, and exchange ideas. This transforms networking from a time-consuming chore into a highly productive and targeted activity. The agenda revolves around these table discussions, not keynotes. Keynotes are there to start conversations, not to steal the limelight. You’ll leave the event with a year's worth of networking and actionable solutions for things troubling you now. EXL 2025 - by Speero and Eppo by Datadog

The best kind of experimentation doesn't start with a test. It starts with a problem. And if you do it right, it could be worth a million dollars. My background is in electrical engineering and RFIC semiconductor design, where I started a company to build simulation tools. After the company was acquired, I worked at Intel, using the software I had helped develop. But it was at National Semiconductor that I got a new perspective.I got exposed to system-level design and started to see how our chips were being used, which helped us drive innovation upstream. We were always doing things the traditional way, but the people using our chips weren't thinking big about the possibilities that new technology could open up.I was looking at wireless power charging and thinking, "What else can we do in that space?". A business development person came to me with a problem: hearing aid users have a hard time changing batteries every week, and the batteries are expensive. A wireless, power-charged battery was a perfect fit. We rapidly tested this idea by prototyping a solution in the lab with an inductive coil and a battery. The business person helped us set up a meeting with one of the top hearing aid manufacturers, Starkey. We demoed the charging capability live. We walked out of the meeting with a $1 million check to build the proof of concept and make a business line out of it. This is a different kind of experimentation. It wasn't an A/B test. It was about finding a problem statement and demonstrating a solution. PS: Read the whole blog post from the link in the comments!

"The way experimentation is practiced today makes almost no sense. We have teams of people diligently experimenting on the front-end user experience, worrying about p-values and guardrails and such. Meanwhile, almost all other business decisions - especially the biggest ones - are made based on opinion only, even if that opinion is dressed up as a rational decision with a deck and some charts. Why are we so carefully testing only certain things? It's because experimentation has become synonymous with A/B testing, and apparently, nothing can exist in business unless there's a piece of software for it. Experimentation isn’t a piece of software; it’s simply a way of making better decisions using evidence. The process and method are more important than any specific tool or technique." Original post and image by Jonny Longden. In this week's Speero's best take of the week: where we celebrate the best takes and opinions from our fellow experimentation experts.

Internal feedback showed that the number of promotional elements on the PLP tiles is an issue. Our hypo: If we remove the 'recently sold' promotional elements, there will be no negative impact on the user experience.   The results showed something different. Bounce rate increased by 1.75% at 80% CTBC.   The test had a negative effect on click-through rate, cart additions, and transactions, resulting in a decrease in the low CTBC in all of them.   It's possible that removing the sense of urgency and/or indication of “best sellers or best pick” from PLP can cause a lower need to take action and boredom among users.   Our next steps were to iterate upon this test, like removing some of the messages (leave only the “100+ sold recently” message), and leave the others. Another opportunity is to use an icon that would refer to a “popular item”.

Product Ops perspectives are all the rage right now. It's the "missing link" that most organizations are lacking. We have plenty of tools for task management like Jira and Monday, and we have wikis and docs. But we're missing the tool that allows a team to do resource and portfolio management connected to cascaded goals and strategic initiatives. At its simplest, this can look like a program dashboard, showing a portfolio of all the bets a team is making. Do you have a picture like this for your program or team? A perspective across the portfolio of bets that the team is doing? This kind of dashboard can include a summary of test results, showing wins, flats, and losses. It can also display the monthly relative potential in dollars for discovered wins and saves. You can see recommended actions like abandon, implement, and iterate. It's a way to see all your work and how it connects to your goals. This isn't about a sheet of A/B tests being worked on. It’s a tactical, tangible mechanism that aligns your activities and outcomes with your broader strategy. It’s what helps make a team accountable but also not stuck in an optimization rut.

Testing isn’t just testing. Experimentation relies heavily on data, tools, and platforms —its infrastructure. This is why 🎪EXL 2025🎪 is 50% focused on platforms: the foundation of good experimentation. For EXL 2025, we’re bringing 40+ moderator peers who facilitate networking and mini roundtable discussions. This is what makes ExperimentLive stick out: We’ll ask you about your issues, goals, and priorities. We’ll find people who have them as well. You all get together and discuss them (in 5-10 small groups) You find solutions and mingle. 🎪 Who’s coming ? Expect folks from: – Netflix – Disney Streaming – ESPN – Coinbase – Ancestry – GoDaddy – Lowe’s – Amex – WHOOP – BigCommerce – Clay – Atlassian – Ebay – Comcast – General Motors – Microsoft – Reddit 🎈🎪 This isn’t your typical conference; it’s three days of no-fluff sessions, honest case studies, and shared war stories from folks doing the hard work. Want in? Get 25% off with code EARLYBIRD25 (good until first 50 sold). Consultants: Bring a client, get a bigger discount. Hit up ben@speero.com for details. Experimentation Live 2025, Austin, October (20) 21-22, by Speero and Eppo by Datadog

Thanks for joining our exclusive live broadcast. Feel free to share your questions and interact with other participants in the chat.

It’s a common secret in our industry: experimentation often feels disconnected from the company’s biggest growth goals. Your team runs great tests and gets good results, but your program still struggles for C-suite buy-in. Experimentation is seen as a tactical function, not a strategic driver. The reason no one says anything is because it feels like a problem with no easy solution. It's difficult to connect the daily work of running A/B tests to the big-picture objectives that leaders care about. It’s hard to get a seat at the table when your work isn't framed in terms of sales, revenue, and market share. If we stay silent, experimentation will remain an under-resourced, siloed function. It will never get the funding or influence it needs to make a company-wide impact. We’ll continue to run tests for incremental wins while leaving massive growth opportunities on the table. But what if we re-framed the entire conversation? What if we acknowledged that experimentation is a language of growth? We’re talking with Timi Olotu, ex-Facebook, Chief Revenue Officer at CharlieHR, about how his role, overseeing sales, product, and marketing, allows him to connect experimentation directly to overarching growth objectives. This cross-functional perspective is key to making experimentation a strategic imperative. Join us on August 20th at 12 pm EST with Ben Labay and Jonny Longden to learn how to change this conversation and unlock your program's full potential.