Aggregation and standardization of financial transactions from multiple marketplaces
Download as PPTX, PDF0 likes5,535 views
This document discusses MapSherpa's process for aggregating and standardizing financial transactions from multiple marketplaces into a single database. MapSherpa sells print maps through various channels, including their own platform, Amazon, and MapTrove. They needed a way to collect sales data from these disparate sources and calculate additional costs like printing, shipping, and royalties in a standardized format. The solution involves using FME to extract and transform data from CouchDB, CSV files, Amazon MWS API, and Google Sheets into normalized tables in PostgreSQL. Reports are then generated from these tables in Google Data Studio for accounting and business decisions.
Aggregation and standardization of financial transactions from multiple marketplaces
- 1. Aggregation and standardization of financial transactions from multiple marketplaces Christy Nieman
- 2. The MapSherpa platform connects map sellers with content from publishers around the world It’s a catalog of digital versions of print maps
- 3. Today’s treatment 1. What is MapSherpa 2. How we sell maps 3. What we need 4. Standardizing our sales data 5. The workflow 6. Output reports Data Detox
- 4. Who are we and what do we do? The MapSherpa system is a digital repository of maps intended for print ● Custom maps: users can search by place name and select the page size, orientation and map extent ● Print on Demand maps: users can search for static maps by keyword
- 5. “Digital” Sales MapSherpa Kiosk is our front-end tool for brick-and-mortar map stores ● Our retail partners order maps for their clients ● MapSherpa provides them with download links to PDFs for printing the maps ● Our retail partners can also integrate the MapSherpa API into their web stores
- 7. Print Sales We sell print copies of maps directly to consumers too ● Amazon Canada, USA and UK ● MapTrove.ca and MapTrove.com Sales from these five sites have additional costs we need to calculate/store ● Printing costs ● Shipping costs ● Inventory costs ● ...
- 8. All Sales-related Data in One Place As our sales volume has increased, it has become increasingly necessary to have an automated process to collect and standardize all related data This solution is needed to drive ● Retailer partner invoicing ● Supplier billing (map publishers, printers) ● Corporate accounting and decision-making
- 9. What We Need 1. Collect transactions from the three sources a. MapSherpa b. Amazon c. MapTrove 2. Standardize sales/returns from all three sources in the same format 3. Calculate printing costs and data royalties 4. Collect all sales related fees (transaction fees, marketplace fees, etc) 5. Display this data in useful ways
- 11. MapSherpa Records - CouchDB 1. Sales/returns for digital sales (our retail partners) 2. Partial records for merchant-fulfilled Amazon orders 3. Partial records for maps sent into Fulfilled by Amazon 4. Partial records for MapTrove orders
- 12. MapSherpa Records - The Easy Part
- 13. MapTrove Sales - CSV CSV export of sales data from Magento database 1. E-commerce sales/returns for MapTrove 2. The additional part of MapTrove records from CouchDB PayPal API to get PayPal fees for each transaction
- 14. MapTrove Records - The Harder One
- 15. Amazon Sales - CA, US, UK - MWS API Retrieve data from the Amazon Marketplace Web Service (MWS) API 1. Finances - get all individual financial transactions 2. Orders - financial transactions don’t contain all info we need (e.g. shipping province for tax purposes) 3. Reports - orders don’t contain all the additional info we need (e.g. warehouse location for Fulfilled by Amazon orders)
- 16. Amazon Records - The Hardest One
- 17. Google Sheets - Pricing and Printing Costs Two additional data sources 1. One Google Sheets document per currency (CAD, USD, GPB) with print retail prices 2. One Google Sheets document with printing cost calculation rules
- 18. Google Sheets - The Extra Data
- 19. Output to PostgreSQL Tables Four tables contain all of the data we currently need 1. Transactions - records for sales, returns, marketplace fees 2. Printers - records for printing costs for each sale transaction record 3. Licensors - records for data royalties for each sale transaction record 4. Shipping - records for “Amazon Buy Shipping” costs These tables are combined into various views for report creation
- 20. PostgreSQL Output - The Satisfying Resolution
- 22. MWSCaller Custom Transformer ● Currently tested with three Amazon MWS API paths: ○ Finances ○ Orders ○ Reports ● Outputs XML or plain text (when possible - i.e. for some reports) ● No one place has all of the information we need. We make calls to the three API paths and combine the data together for each sale
- 24. Google Data Studio Reports can be created, accessed and modified for accounting and business decision making
- 25. Google Data Studio Several database views have been created from the four base tables to satisfy report requirements Allows us to optimise for each report without having multiple places to update the base data
- 26. In combining all our sales and production data in one place we can now more efficiently do billing, invoicing and planning
Editor's Notes
- #3: We have relationships with publisher from around the world and are licensed to provide their maps to our retailer network.
- #5: Custom maps: UK Ordnance Survey 1:25,000 and 1:50,000 topo maps, Canada/USA streets made from TomTom data, Lucidmap city/province maps Print on Demand maps: National Geographic historical back catalog, 10th Atlas of the World, Reference series; Lucidmap Canadian cities/provinces, Backroad MapBooks Topo sheets, Twisted Spirit Mythical Monsters Maps
- #6: Our retailers all have accounts in MapSherpa where they can search for custom and print on demand maps for their customers. They are able to order and download PDFs to be printed. These orders have data royalties owed to the map publisher calculated when they are submitted to the system. MapSherpa OnDemand API allows customizing and ordering of all of the custom maps and print on demand maps available to the retailer doing the integration. There are methods for listing available custom/POD maps, bringing up the customize dialog, setting page size, orientation, title, etc.
- #8: We have agreements in place with many of our publishing partners to sell their maps on Amazon and on our own e-commerce site. The royalty model is different from the one calculated automatically on digital sales, so this needs to be handled when we process the financial information each month. We also have to calculate the production costs based on printing rules established with our printing partners.
- #9: When sales volume was low, it was feasible to accumulate and calculate all required information manually, though this was prone to error. Now we are at a point where it is no longer practical, and for the last year we have been working on refining a database solution containing all relevant data and a front end solution of reports.
- #11: First Click: MapSherpa records Second Click: Amazon records Third Click: MapTrove/PayPal records Fourth click: writing to postgresql
- #12: MapSherpa’s back-end database isCouchDB. We store all sales and returns of digital (retail partner) sales in the MapSherpa database, along with orders created to retrieve the PDFs for print sales (amazon, maptrove). We also send maps into Amazon’s warehouses, and we need to store and calculate appropriate data for these maps as inventory.
- #13: Make sure the view containing the appropriate couchdb documents exists in the database, request all records for the desired time frame, and do a little clean up. Inventory orders in the MapSherpa database need to have the “print retail price” set, which is different from the content access fee that MapSherpa stores by default for digital sales.
- #14: MapTrove integrates the MapSherpa API, so when a map is ordered, the appropriate sale is recorded in the MapSherpa database. Each month we use a CSV sales report to get the sales revenue, shipping revenue, taxes, etc. and join them to the MapSherpa records. MapTrove uses PayPal as its payment processor, so we also need to retrieve the paypal fees for each transaction and store them along with the sales information.
- #15: Top: get appropriate data from the PayPal API, format it and filter by sales and returns Bottom: read in the sales report CSV, format the records, join to paypal records, join to MapSherpa records, set attributes accordingly
- #16: We use three parts of the MWS API to retrieve the information we need for each sale/refund transaction.
- #17: Basic workflow: Get financial transactions (XML), filter them so we know what kind of transaction they are and format the attributes Get orders for all sales and returns in the current time period being processed and join them to the financial transactions More attribute formatting, filtering seller and amazon fulfilled transactions Join seller fulfilled transactions with mapsherpa records Get all fees and charges for sales and returns Get amazon fulfilled orders report so we can join to amazon fulfilled transactions so we know what warehouse the maps came from
- #19: We store the current retail price of all maps sold in Amazon and MapTrove in a google sheet as it is quick to update prices based on production cost changes in a spreadsheet. Once the sheets are updated, changes are applied to the marketplaces We have one sheet per region with printing rules by finish, page size, etc. that is used to calculate printing costs in FME Printing cost rules are also stored in a Google Sheets document and are retrieved for calculating printing costs
- #21: All records end up here to have data licensing fees and printing costs calculated as appropriate. Finally everything is saved to the database, and the materialized views build from main tables are updated.
- #22: The custom transformer itself is pretty basic with just a handful of transformers, but the main part is a python caller that makes the appropriate requests to the MWS API
- #25: The current endpoint of all of this is a collection of Google Data Studio reports. This allows us to quickly put together the reports we need and make changes as necessary, which occasionally requires changes to the views in the database.