Effective Use of the Twitter Search API

Editor's Notes

• #4: i will talk about: - start by giving some of our thinking about why we have a search api and what differentiates it from the other api’s twitter offers - i’ll get into some technical implications of these differences with respect to polling on search versus tracking keywords on the streaming api - next, i’ll talk briefly about how the search api has changed over time - and then we’ll dig into the most recent change where we began ranking the top results beyond recency order. i’ll show you how i’ve modified one of our own search api clients to take advantage of that change
• #5: simple definition: user provides a query by engaging with an api application, we provide the best stuff (currently tweets and trends) for that query Obviously the “best” stuff for twitter has a lot to do with how recent it is, so our primary focus is on the “here and now”
• #6: Just to give you an idea of the parameters search operates under: - as ev told you yesterday we are doing more than 600M queries per day, seen up to 750M on a day recently - while realtime is our main focus, our index does contain hundreds of millions of tweets and we’ve roughly doubled its size in the last six months. - of course, the amount of tweets has grown even faster than we’ve increased that index size, so this only covers about a week of them right now, but that is something we’re currently working on expanding
• #7: So obviously we’re operating a large scale, but what’s really interesting to me about the search API is the variety of applications you as developers have found for it. I’ve listed just a few here to illustrate what people are currently doing with the API.
• #8: So that’s what people are doing with the search api, but the streaming api also supports tracking keywords and some location and language filtering. So, if you’re developing a new app, how do you decide which to use?
• #9: The biggest difference between the search API and the track API is how you get new results matching your standing query. On the streaming API the push model makes this obvious: new results are sent to you as they come in. Since the focus of the search API is on apps that let the user manipulate the query (whether explicitly or implicitly), registering a standing query for every request makes less sense. Instead, the search API uses a polling model with a cursor. --- make sure you explain this diagram by pointing at it (or at least describing it). It took me a minute to get the visual presentation
• #10: One question that comes up frequently is why we encourage apps to use this cursor to poll and how that helps us to support refreshes more efficiently, so here’s a diagram of what happens under the covers. A lot like the streaming API, when you make any query to search we actually do register that as a standing query, but only in one of our caching layers we call the timeline cache.
• #11: Next I’d like to take a step back and talk briefly about the history of the search API and how our thinking about it has developed. twitter search and the API have been around for about two years now, and we made a lot of changes early on like supporting location search, but after that we had to shift our focus to scaling the system to support the growth in tweets and queries. It’s really just in the last six months that we’ve made enough progress with scaling and grown the search team enough to be able to focus more on relevance and figuring out what that means for twitter search.
• #12: Our mission: ---- Under “many factors” you should note that it’s not always the popular users that show up here -- that seems to be an early misconception. Our algorithm looks to find things that are interesting from any user - things that “resonate,” to use a word that Dick talked about yesterday (good to tie it in to other things being said at Chirp). Rather than “not final” (which seems to imply there is a “final” step when we won’t be improving this) I’d say something like “First step of a long road of relevance improvements” (implying that we’ve got lots of ideas and we’ll be delivering cool stuff for a long way.
• #13: right now at the top
• #18: explain that this uses since_id
• #23: we want to hear from you