Deep Data At Macys v1.0

Editor's Notes

• #4: 1 minute [Greetings, introductions] Peter’s bio: I have a background in the electrical engineering and physics of semiconductors, but since 2007 works in the software development for the various companies first as a software developer, then for many years as the system analyst and project manager. Since 2013 I’m working with Macy’s.com on the various of the projects, and since the beginning of this year I work with the Macys team developing the search engine and broadly the platform for the navigation and search as the technical solution manager. In this role I help in integrating the different software systems providing the search and browse functionality to the macys.com customers. Eugene’s bio: I switched from academic research in the field of numerical analysis to IT industry in early 2000s. Joining stealth startup in the field of business intelligence, I learned foundations of information retrieval and became fascinated with scalable systems in general. Later, In 2006, I helped to found Grid Dynamics, a consulting company with the focus in scalable platforms and applications. Since then, I played a many engineering management and technical roles in variety of programs ranging from cloud PaaS solutions to financial services systems. In 2011, I joined Macy's program aimed to re-platform customer facing keyword search and guided navigation capabilities to Solr-based solution, which launched in 2013. As a solution architect, I am helping to evolve this platform ever since.
• #5: First, let me briefly talk about the Macy’s itself, its history and the key dates. Currently Macy’s is operating by about 800 departments stores in the United States, and the business was started by Rowland Hussey Macy in 1858. The current name – Macy’s – company has been given at 19-02. So, it is a business with the century and half history, there are several cultural dates, associated with it, like Macy’s Thanksgiving Parade and July 4th fireworks. Some key dates which you can see. ============================== Q: Do we know how much we sold the first day we opened? A: $7.52! (about $200 in today’s money) Let’s talk about Omnichannel. Omnichannel integration is helping Macy’s and Bloomingdale’s to develop deeper relationships with loyal customers who appreciate the convenience and flexibility that we bring to the shopping experience. We believe that the key to success is the integration across channels – blurring the line between our stores, the internet and mobile technology to the point that we surround our customers and can respond to their needs no matter which way they prefer to shop … and we can access inventory no matter where it exists across the company. Macy's, Inc., with corporate offices in Cincinnati and New York, is one of the nation's premier retailers, with fiscal 2011 sales of $26.4 billion. The Macy’s brand includes about 800 Macy's department stores and furniture galleries in 45 states, the District of Columbia, Guam and Puerto Rico, as well as macys.com. The Bloomingdale’s brand includes 37 department stores and home stores in 11 states, bloomingdales.com and (as of January 28, 2012) seven Bloomingdale's Outlet stores in five states. Bloomingdale's also operates in Dubai under a lease agreement with Al Tayer Group LLC. And now, you will become a part of our history. At Macy’s, our greatest strength lies in the skill, judgment and talent of our people. Every day a production of enormous magnitude takes place on our selling floors, online, and behind the scenes. This is where our people bring the company’s strategic goals to life. Our priority of attracting, retaining and growing the most talented people in the retail industry has been and will continue to be our greatest advantage. Now that we know more about our history with Macys.Inc, let’s learn about our humble beginnings as a dotcom organization.
• #6: Macy’s run e-commerce business at 19-98, it was one of the dot-com pioneers, and now it is one of the biggest tech employers in San Francisco and we are hiring. ===== Here are some of the highlights of MCOM history 1997 “MCOM” operates out of San Francisco, California and Brooklyn, New York Since all of Macy’s is under Federated, the Brooklyn office is specifically a subsidiary of Federated Direct,and is referred to as Macy’s By Mail. Our Brooklyn office is specifically a catalogue business. We operate on the 6th floor of what was formerly Abraham & Strauss (currently Macy’s) Bloomingdale’s By Mail, which is also under Federated Direct, operates on the 7th floor. The San Francisco office operates as an unofficial dotcom out of the Union Square store. Dawn Robertson and Jeff Sherman are Co-chairs of MCOM. May 18, 1998 MCOM is officially launched and Kent Anderson is the first employee and he is appointed President. The San Francisco office is using images from our catalogues in order to build our website. We expand from offering fashion items to also offering home goods. Helaine Suval is EVP and is instrumental in building the NY office. 2000 We invest in digital photography (Vertis) and begin using one fulfillment center, in Cheshire, CT. Before this, we mainly shipped from stores. Feb 2001 New York offices moves from Brooklyn to the 6th floor of 1440 Broadway in Manhattan. We are such a small and intimate group (of about 70 – 100 employees) that we celebrate birthdays each month: “We would bake sheet cakes and have the entire NY office gather in the 6th floor boardroom. We would sing happy birthday and make the individuals wear silly birthday hats.” – Stacey Maggi In the NY office, we had 13 people in all of planning department. “We didn’t have allocators. They came on board in 2007.” Sept 2001 Macy’s By Mail catalogue business shuts down following business declines after 9/11. New York office functions solely as a dotcom. Catalogue business shuts down making Macys.com the sole provider of merchandise by mail. Buying and Planning operate out of NY. Creative and Marketing operate out of SF. 2002 Creative Team is transferred from SF to NY in order to be closer to buyers. 2003 Merchant world in NY transfers from Galvin cataloguing system to Mainframe (Green Screens) to manage financials. 2004 Federated Direct goes away. Tim McAdams is appointed chairman of MCOM. 2007 Marketing department is broken out between NY and SF.
• #7: One more date deserved to mention, especially at this conference, is 2013 when we run the solr-based keyword search.  We started the Solr-replatforming in Dec-2011, and go-live in prod in Apr-2013, so, it took about 1.5 years. Migration also included a development of the tool for merchandising managers, what is it and why that component is needed we’ll discuss some later. In 2014 we’ve also migrated the browse to the search engine, and it also becomes available in Merchandizing Management Tool, unifying by this the category browse and keyword search capabilities and the ways we manage them. One more feature to mention is the dynamic grouping-ungrouping of the product collection, was rolled out in 20-15.
• #8: Here are some KPIs related to the solr-powered search. ===== Autocomplete: 2.19% Order Conversion with manual search phrases, 2.37% (+8%) with autocompleted. 16% Lift in incremental sales with new MML logic BCOM: 39% Lift in Search Order Conversion over LY
• #9: So, what we wanted to present today how we search, why the search in retail could be difficult, what are the features we’ve implemented to resolve these difficulties, and we’ll start with where we see the department stores in the retail. Let’s talk about it. Let’s first understand the types of retail and where the Macy’s as the department store is in. First, we can divide all the retailers to two types by the source of the data and the associated level of the data curation – mall and specialty. Malls are focusing mostly on the brand management while specialties are on the inventory management. We also can have a look on the different models of retail based on the size of the inventory. The two edge categories here are boutiques and big boxes. While the main focus of Boutique is on presentation and customer service, Big-Boxes are focusing on discovery and self-service. ========== Boutique: focus on presentation, customer service Big-Box: focus on discovery, self-service Mall: focus on brand management Specialty: focus on inventory management Department: focus on merchandizing, curation, cultural impression (TBD???)
• #10: So, we can represent these two dimensions as the two axis as it is shown – where one axis shows the inventory volume, and the other one is the level of curation. For the specialty the source is the retailer’s catalog, which is managed by retailers itself. The difficulty here is careful managing of the product attributes. Instead, malls are usually the houses for the several retailers, and the source of the data here are the small retailers housing within the mall. Mall model does not suppose the deep understanding of the attributes of the products and SKUs every retailer is selling, but it supposes the retailers management, providing general for all capabilities to sell. And the boutiques and big boxes are as I said present the very different volume of data to deal with. And department stores seem to be somewhere in the middle - because they have to know the product attribution quite well, quiet carefully curate the data but also they have the pretty big catalog, and this case absorbs all the difficulties of all four cases. =====
• #11: And the key to manage that complexity is merchandising, which is, as you can see the definition - the activity of promoting the sales of goods, especially by their presentation in retail outlet. There is a pretty comprehensive set of instruments and KPIs to present and promote the products, which will be reviewed later, such as: Promotions Discoverability - i.e. ability for customers to find what they need through the browsing of the product catalog, or through the search. Recommendations Side-by-side comparison Curation, Bias ====== Promotions Discoverability Recommendations Side-by-Side Comparison Curation Bias Zones & Presentation Conversion Average Order Value A/B Testing Comprehensive Inventory SEO & SEM Upselling & Cross-Selling Categorization
• #12: So, let’s now have a look on why do we need the merchandising, from the search perspectives. You know there are two major characteristics of the data retrieval systems: Precision and Recall. Just to remind, assume we have some set of documents  through which user is searching in order to close the information need. The ratio of the relevant documents within the returned search result is the precision, while the ratio of the returned relevant document within the entire set of the relevant documents is the recall. By some mean, these two characteristics are concurrent, and we cannot achieve the ideal precision, and at the same time have the ideal recall. But in retail we have the definition of information need a bit blurred between two types of users - Customers and Merchandisers, which needs are different.  Moreover, customers usually behave in the different modes - tracker, hunter, explorer, which also affect what the information need is. So, the relevancy in retail is the moving target. A trade off between the needs of the customers and merchandisers should be found. ====== Precision is what % of the search result is expected Recall is what % of expected results were returned What’s “expected”? Expected by who? Traditionally defined by Information Need Two perspectives: Customer and Merchandiser Customer expects result that “feels right” Shopping modes: tracker, hunter, explorer Merchandiser expects result that “entices to purchase” Merchandising KPIs: conversion, AOV, sales
• #13: I really like this picture, especially this guy, and this guy and that lady. And it’s really one size does not fit all. We also put here the onion as the pungent metaphor of the dependencies of the definitions we deal with. Like, Relevance is derived from the Precision and recall, which in turn, derived from the information need, which in turn derived from the customers needs, and merchandising needs, which also depend on the customer shopping mode and the KPIs a merch wants to achieve, such as conversion ratio, average order value, sales, etc. Often that problem of the precision and recall balance is solved by the relevance tuning, but it’s hard to do because once you tune the relevance for one category, one family of business - the other once could get wrong. It’s also hard because once you tune one of the KPI – another one could get wrong. Instead of relevance tuning we provide the well-understood building blocks, which can be combined and let merchandisers basically do their work and tune the system not by the hardcoded set of formulas, but by the tuning of these blocks. So, let’s now review which building blocks we built in for that. ======= Image source: http://castabigger.net/uncategorized/marketing-strategy-one-size-fits-all-think-again/ Relevance is defined by the balance of Precision and Recall Precision and Recall are derivative of Information Need Information Need is defined by the balance of Customer and Merchandiser Expectations Expectations are derivative of Customer shopping modes and Merchandiser KPI targets “Relevancy Tuning” – don’t do it! Yet configurability is key to success Focus on well-understood building blocks instead of an all-encompassing formula Let Merchandisers do the rest
• #14: Start first with the customer’s capabilities: quality which is the precision of the search. As I said we don’t tune the relevancy, so, the products in the result set are provided based on the natural relevancy. Refinement - is basically the way to customers to improve the precision of the search by themselves. Usability - pretty straightforward. And targeting is the way to improve the recall by accounting the customer’s demographic, geography, by personalized and collaborative search, in attempt to improve the recall.  === Quality – how precisely does the set of products presented to the Customer match her explicitly stated expectations? Refinement – how easy it is for the Customer to slice and dice her results to find exactly what she is looking for, or to be inspired to look for something she did not think about before? Usability – how easy, convenient and exciting is it for the Customer to interact with the system? Targeting – how precisely does the set of products presented to the Customer match who she is (implicitly stated expectations)?
• #15: EUGENE In eCommerce, the main sellable item is SKU (stockkeeping unit). Obviously, you can only sell an item of particular size, weight, color and material. However, in reality, many SKUs are very similar and differ only in some combination of size, color and therefore price, while sharing brand, material, style and many other characteristics. If we would show SKUs in our search results, we would food our screens with identically looking images of different flavors of the same products. Obviously, nobody wants to waste precious screen real estate, so majority of site merchandize Products, not SKUs. Further, many sites organize products into Collections, which have different purpose and grouping logic. We should be able to search and retrieve items from two logical levels – products and collections. Those business requirements emphasize structured, hierarchical nature of eCommerce catalog which should be respected when designing Search solution for eCommerce. Ignoring this structure leads to customer experience issues which we will talk about next.
• #16: Lets consider simple example. Here is a sample catalog where we have 2 Products, each having 2 SKUs of different colors and sizes. Customer is searching for small red samsonite suitcase, so we should be able to match on the attributes belonging to 2 different levels. Brand Samsonite and Type suitcase belongs to Product, while Color and Size belong to SKU. As our search result should consist of products, not SKUs, it is very tempting to model our catalog as having only Product-level documents. To ensure matching on SKU level attribute, we can just rollup all attributes of SKU to the corresponding products as multi-valued attributes, which should ensure matching. It also delivers excellent performance, as number of products typically is order of magnitude less than number of SKU. However, when doing that we are ignoring the structure of catalog, e.g. loosing the information from which SKU particular size and color came. This leads to dreaded false positive matches. In our example, due to attribute rollup, both P1 and P2 will match the query, and P1 will be false positive as there is no both Red and Small SKU within this product. Customer will be very disappointed to discover this fact on Product Details or even worse, Checkout pages. With Collections, this problem is getting on another level of magnitude, when we start mixing attributes of completely unrelated products. Some eCommerce sites seems to accept this negative implications. For many others, including Macy’s, this is not acceptable customer experience.
• #17: Okay, we learned that attribute rollup causes functional issues. Maybe, instead of rolling up attributes, it is better to roll them down? In this case, we will propagate all product level attributes down to SKU and we will search on the SKU level. Once we done searching, we will have to group results back to output Products. This approach provides precise filtration, yet leads to significant performance issues, especially when we are dealing with multi-level hierarchies. Some of the reasons for poor performance in this case: On the SKU level, posting lists of inherited fields, such as Brand or Type will be cluttered with identical data. E.g. All SKUs of Product 1 will be Samsonite Suitcases. If we will be searching for Samsonite suitcases, all BRAND and TYPE scorers will match on every SKU of P1 which will lead to useless collecting and scoring those matches. For grouping results, we will have to retrieve and evaluate productID field of every matched SKU. White Uninverted index and docValues provide decent performance in accessing those fields, many people reporting result grouping be a bottleneck in their applications. All of the above applies when calculating facets If you have shallow hierarchy, do not plan to extend it and your performance is acceptable, probably this approach will work well. However, if you have to support deep hierarchies we should look for a compromise between attribute rollup and rolldown approaches.
• #18: In our implementation, after considering alternative approaches and experimenting, we decided to go with BlockJoin approach in modeling our catalog. In this approach, we are performing “Index Time Join” by embedding our parent-child relationship inside the Lucene index structure. We index our hierarchical data as a blocks of document, where each child is preceding his parent. Thus, after matching on each child we can retrieve corresponding parent, very fast without explicit result grouping step. This approach allows us to avoid propagation and duplication of attributes data. It is obviously less performing than attribute rollup approach, yet in case of our system it delivers good performance for some very complex queries even for 3-4 levels deep hierarchies. We also use use exact model of relationships in the index for some other advanced merchandizing use cases, such as dynamic grouping. We will talk about it later. As with many designs aimed at solving specific problem, there are trade offs with this approach. I can name some of them. Block Join Index is not friendly to document updates. If you need frequently update your documents you’ll have to re-index whole blocks containing those documents. Note that if you need just to update particular fields, there are solutions which allow you to workaround this limitation BJQ in general leads to more complex queries, they are somewhat harder to develop. We implemented our own BJQ faceting component – there is no high performance BJQ faceting component in available in Solr yet
• #20: Another important aspect of search quality is Concept Search. eCommerce Search engine should be able to work beyond phrase tokenization and standard text matching. Standard text matching overlooks multi-word concepts, such as brands “material girl”, “mickael kors” or product type “tea pot”. Another issue is polysemy. “Pink Rose handbag” query phrase doesn’t mean that customer is looking for something pink – she is looking for brand named ‘pink rose”. In other cases, there is unresolvable polysemy. For example “silver jeans” may mean both brand and combination of color and product type. In the example here we see that correctly recognizing concepts in the user query will lead to precise, high quality results, while just matching on the elements of the query phrase may lead to wildly irrelevant matches. Relevancy tuning will not help much here as irrelevant matches, even if they will be buried, will contribute to facet counts and contaminate facet values. However, high precision concept search alone is not enough. Sometimes, precision is just too high and we can’t find in the catalog anything which matches customer query exactly. In this case, we should start relaxing our query constraints, gradually increasing recall until we will find satisfactory results. This allows to strike a balance between precision and recall.
• #22: And on the other side is Merchandising capabilities: curation, bias, comprehensiveness, and referring. Let’s just go briefly through all of them. Curation - is the way to easy promote the fixed products and product sets, overriding automated features. To support that we built in the tool which I mentioned above - Saturn - where the site managers can easily configure business rules in order to manage the presentation of the results to customers. For example, for the given keyword or category ID we can trigger the action of showing the product pool at the top positions of the search result. Or if the result set contains the products of some dominant style - we can show the specific banner. Or we can show or hide the certain type of facets. Or boost the products having the special value of the certain attribute. Each rule is the set of triggers which could be joined logically, and an action. They can tie Bias - While curation deals with the some sort of actions and some products within the result set, bias is applied to the result set as a whole. Basically, if we text search - we have the product sorted by relevancy to the search phrase. But in retail it is unlike that we want to show the products to customers in that order. The second one is that completely different products could have the same relevance score. Instead, we want to sort products in the order like most popular products, i.e. based on the number of purchases of this item. Or based on the dollar sales. Or we want to show the newest products first. Or we even can apply the formula for that kind of metrics, for example get the newness and dollar sales with the 50% weights, sum it up and sort the products by that calculated value. Comprehensiveness shows how large is the discoverable inventory and how accurately does it reflect the product price and availablity. It’s also related to the omnichanel data. Omnichannel integration is helping Macy’s and Bloomingdale’s to develop deeper relationships with loyal customers who appreciate the convenience and flexibility that we bring to the shopping experience. We believe that the key to success is the integration across channels – blurring the line between our stores, the internet and mobile technology to the point that we surround our customers and can respond to their needs no matter which way they prefer to shop … and we can access inventory no matter where it exists across the company. And the last is Referring - if you buy a suit - how we can refer you a dress shoes? Or, if you are looking for a jacket - how we can provide you with more choices? That problem is laying a bit outside of the search, but it helps to close the gap between precision and recall, basically by improving the last. ===== Curation – how easy it is to promote fixed products and product sets, overriding automated features? Bias – how easy it is to improve Merchandising Relevancy by promoting some products over others based on the likelihood of selling or other business drivers? Comprehensiveness – how large is our discoverable inventory? How accurately does it reflect product price and availability? Referring – how easy it is to refer the Customer to other relevant products?
• #23: One of the big feature in Macy’s way of merchandizing is a collection of products. Collections is not just a group of products, they have their own attributes are used in variety of use cases. Product sets, such as bedding and dinnerware, group products which have common brand and design, and people tend to purchase this set as a whole, but still have an opportunity to purchase individual items Outfits, where merchandizers group products of the same style together as a mechanism to upsell Gift this purchase as a mechanism to promote particular products grouping of products by attributes not present in UPCs (“rugs” use case) (need to clarify with DS) Regular search algorithms are able to find product collections, along with all their member products, and return the result to the customer. Even though all results will be relevant from the Customer point of view, Merchandiser may want to unclutter the search result by combining similar-looking products, or eliminating matched product collections if the query was very specific. These rules vary from query to query, and must be configured per product collection Lets consider an example of dinnerware collection. When a customer is searching for just dinnerware, it would be counterproductive to show all the members of this collection – it will clutter the screen and confuse the customer. Instead, we should recognize that customer is looking for collection as a whole and show only dinnerware collection. Now, assume that customer is searching for “white dinnerware”. We could output all white items inside dinnerware collections, but in this case looks like that all the collection is mostly white, so it still make sense to group it. However, if customer is looking for white cup, we see that collection is not a collection of cups, and thus irrelevant as a whole, so we should ungroup the collection and show only matched members
• #24: To illustrate dynamic grouping and ungrouping business logic, we can try to image what would it take to implement something like this in SQL. Turns out, that even expressing power of SQL is not enough to support those requirement. We need something like “conditional grouping” feature, which we express with the made-up clause WHEN. WHEN clause evaluates a predicate for a group of items collected by GROUP BY statement and has a power to cancel grouping if its predicate evaluates to false. Lets see how we implemented this feature in Solr. . Key challenge here is to retrieve precise data about way how exactly query matched in the collection data. We need to know what part of the query matched on Product level, and that part – on Collection level. As you remember from previous slides, our index is structured, which provides a great help in providing such data. So, we have a “Match Spotting” routine which on every match evaluates a matched path in the scorers tree and provides information for custom grouping collector about precise nature of the match. Grouping collector has access to whole matching block information and can make decisions what to put into result set – collection, or its matching members, or both.
• #25: Question of relevancy scoring is one of the most important questions in finding the good compromise between customer and merchandizer intentions. When we implemented concept search, we naturally discarded traditional TF-IDF relevancy formulae. Underlying TF-IDF assumptions does not fit well ecommerce reality. Instead, we created a coarse relevancy score which is tightly coupled with particular query pattern customer is hitting. In this example query “Silver jeans” has polysemy and can be recognized as Brand=“silver jeans” or combination of color and product type. We consider all products matching same pattern as having same score and proceed to next stage of scoring. On next stage, we employ sorting by merchandizing rules which take into account such signals as newness, promotions, popularity, ratings, etc… Finally, ontop of this automated tiered ranking, merchandisers may apply special rules to boost or bury particular products. We will talk about those rules next. Example for poor TF-IDF (In case of question) Product 1 DESCRIPTION: black dress Product 2 DESCRIPTION: shiny black leather belt, perfect for black dresses and black items TF will rank P2 higher than P1 Another example: Leather (1000 items) Red (50 items) “Red leather shoes” query IDF -> will give preference to black items wrt leather items, which is not necessary correct.
• #26: Sometimes, we have issues with product attribution which is relatively hard to fix upstream. In this case, we employ merchandizing rules which allow to manipulate search results by adding, removing or reordering items. Examples would be that merchandisers don’t want to attribute some products as “Peanuts” (popular comic character), yet they want for them to show up in Search results when somebody typing “Peanuts”. Usually, sooner or later such things become official attribution and corresponding rules are removed MMs can also remove some products from result sets if they are hitting some edge cases of match algorithm or have issues with attribution. And finally, MMs can promote some of the products based on their attributes. For example they may bury on-sale jeans (which is typical for the stores when cheap stuff sits on lower shelves and more expensive items are in front of your eyes)
• #27: We started by talking about the history of Macys and its place in the overall retail industry. We recognized that Department Stores business model is unique, and we believe it has a great future because of its versatility. At the same time the same versatility creates challenges for product discovery in the context of e-commerce. We realized that attacking the issue of producing a relevant result by playing with the scoring formula alone does not, and cannot satisfy all stakeholders. We identified the discipline of Merchandising as a key differentiator in the department store business model. We have carefully reviewed common e-commerce use cases, and grouped them in Customer-facing and Merchandiser-facing capabilities. We suggest that the right path to relevant result leads through the combination of multiple configurable, well-defined, purposeful features rather that looking for a silver bullet. We gave examples of several highly innovative features from the Customer-facing and from the Merchandiser-facing arsenals we have implemented at Macys. Recall Concept Search, Precision Reduction, Tiered Natural Scoring, and Rule-Driven Query Rewriting. We gave special attention to the Deep Data (structured catalog) features demanded from both Customer (Precise Filtration) and Merchandiser (Dynamic Grouping) points of view. This is an example where the right choice of technology (index-time join) produced synergistic benefit for both sets of stakeholders. As for many retail giants, future strategy for Macys is framed by the Omnichannel integration. From Solr technology point of view this presents all flavors of scalability challenges: from the request arrival rate to the amount of data and data updates we must handle. Similarly to what was already achieved, finding the right technical answer to the scalability challenges will have a synergistic benefit by allowing Merchandisers to present comprehensive Omnichannel catalog data in a way uniquely tailored for each Customer. If you like tough technical challenges which do not have an easy answer, and sometimes require difficult compromises, please join us at Macys in San Francisco. We are hiring.