Developing an SDK for Personalization at the Edge

Editor's Notes

• #2: Thanks, Brian As Brian said, my name is Chris Reynolds and I’m going to be talking about Developing a software development kit for personalization at the edge
• #3: So a quick intro to me I work at Pantheon and I am the technical lead on the Edge Integrations project Prior to Pantheon I worked at Human Made and spent much of my last year there working on their Altis Digital Experience Platform Before that I worked at WebDevStudios, so I’ve been working in agency environments on large enterprise-level projects for a while When I’m not working, I’m probably planning the next session in an ongoing D&D campaign I’ve been running for the past couple years You can find me online, I’m jazzsequence most places – sometimes with a 3 and sometimes without depending on the platform
• #4: What is “the Edge”? Short version: The edge refers to the layer between the application server and the user consuming a web site, e.g. a CDN. Generally speaking, the edge layer is located geographically close to the web user to minimize download times. Sometimes, CDNs – like Fastly, which is what we use at Pantheon – are able to run logical operations or surface data about the user from the GET request that was sent that can be acted upon. This is what edge computing is.
• #5: What is “Personalization”? You’re probably familiar with the concept of content personalization. Basically, because I am interested in and run an ongoing Dungeons & Dragons campaign, I am frequently consuming information about Dungeons & Dragons. Instagram and other vendors might see my behavior data and what things I click on or spend more time on, and put me in a bucket of D&D-interested-users which advertisers might subscribe to. Because I’m flagged as a D&D-interested-user, I will see ads specifically targeting people interested in Dungeons and Dragons and roleplaying which means that I get a lot of stuff about roleplaying Kickstarter campaigns, D&D loot crates, dice making companies – it’s really on point, tbh This is all powered by a personalized profile derived from my behavior that is able to make decisions and assumptions based on what I interact with
• #6: So far, not much of this is likely to be new information. What is “Personalization at the Edge”?
• #7: Personalization is the hot new buzzword. Everyone wants it, not many people know how to execute it.
• #8: Personalization is hard. Website owners interested in pxn have to develop a strategy about what they want to do with the personalized information, how to deliver personalized content and how to measure that the personalized content is having a measurable impact. Marketers tasked with creating personalized content are faced with the daunting task of having to create exponentially more content for each audience group they are trying to reach. Engineers need to figure out how to implement personalization in a sort of wild west, with very little regulation, few working use cases to base their work on, and potentially very little direction.
• #9: Because pxn is hard – but is often in demand – services exist to offload your pxn work to a third party. All you need to do is add a bit of Javascript code and all your pxn is handled. There are a number of problems with this model: Most of these services require you to build and add your personalized content in the service itself, rather than in your CMS. This can create a “shadow CMS” – where some content lives in the pxn service and other content lives in the CMS, and it may not always be clear which is which. Taken to a far enough extent, this could end up in a situation where none of the content served on your site actually lives inside WordPress anymore and, instead, the marketing team only uses the third-party service to manage content. If you ever want to disentangle yourself from the third-party, your content would need to be migrated away, and it’s likely that this process would need to be manual. Because you’re entering all your content into a third-party, you no longer own your own content, you don’t have access to it, and you can’t move it elsewhere at will. In addition to this, frequently you will end up with a “flash of unpersonalized content”. It takes time for the browser to load all the content and scripts on a page, and there is probably going to be at least some amount of time where the page will render before the third-party script is executed, resulting in generic or unpersonalized content rendering in the initial page load and then quickly swapped out by the personalization javascript. Finally, one more script is one more script and anyone who’s worked with multiple third party scripts knows that adding in more external resources is not often a recipe for success.
• #10: What if we could serve personalized content at the edge? What does that look like?
• #11: I am vegan and I live in the united states. I visit a food magazine website and I am immediately presented with a hero at the top of the page that recognizes where I am visiting from. If I’m visiting from the US, maybe it shows me a burger recipe If i’m visiting from Canada, maybe it shows me a poutine recipe
• #12: I don’t necessarily notice this, but I continue on, navigating to the recipes site. As a vegan, I’m not interested in the featured recipe I’m immediately presented with, something meat-based.
• #13: But, there are some interesting vegan recipes further below. As I click multiple vegan recipes, a profile starts to build and an interest is stored for me so that, eventually, the featured recipe that’s shown to me is a vegan recipe.
• #14: This is an actual working use case we have on a Proof-of-Concept site at p13n.me
• #15: What’s actually going on?
• #16: Using Varnish Configuration Language (VCL) combined with data-handling variables provided by our edge CDN, we’re able to identify where geographically a user is visiting the site from. We can process and send this data back as response headers to the browser. Meanwhile, if we get multiple requests that match the same key generated by their location, we’re able to cache a variant of the page. This means that when I visit from the US, I might be presented with a cached version of the US homepage showing a burger, whereas if I am visiting from Canada, I might be presented with the cached version of the Canadian homepage showing poutine.
• #17: As I click through different recipes, my specific interest in vegan food flags a “vegan” taxonomy term as an interest. This is stored in the browser as a cookie. The cookie is sent to the server with the request. The CDN is able to read the cookie and it checks to see if a cached version of a page exists that is relevant to my interest in vegan food. If a cached version does exist, I get the cached vegan recipes page. If a cached version doesn’t exist, a new cached version is created
• #18: The magic here is in the headers we send via the code to the server – specifically Vary.
• #19: Vary is a complicated subject but the basic idea is illustrated in this infographic The main idea is that there’s a URL and based on the vary headers that are requested, the content – or in this example, the accepted format of the content – might have variations for that URL. In our case, we’re varying on Geolocation – identified at the CDN layer – and interest – which is sent to the CDN via the interest cookie
• #20: So, what’s the SDK? Because Pantheon owns the stack, we can come at the idea of Personalization from a different angle, specifically using our edge layer and VCL through Fastly to deliver cached variations of pages provided the CDN knows what to vary on and it knows how to send data back to the application server. The CMS needs to know how to ingest this data being passed back via headers and how to send data to the CDN to create variants. That’s where the SDKs come in. Pantheon supports both Drupal and WordPress sites, so a core component of our approach should address both CMSes in a system-agnostic way, while CMS-specific integrations can be built separately. In both cases, the CMS needs to be able to inform the CDN of any identified interests based on behavior.
• #21: In the end, we built two SDKs – the Pantheon Edge Integrations Drupal SDK and the Pantheon Edge Integrations WordPress SDK. Each package was built as a metapackage
• #22: There’s not much to the SDK in itself but some functional documentation
• #23: Each SDK uses a composer-based dependency tree that calls in the appropriate external repositories based on the CMS Foundational to both SDKs is the Pantheon Edge Integrations global library. This is the “glue” that handles interacting with the headers sent to and received from the CDN.
• #24: For the WordPress side, there’s a plugin that was designed to be immediately extensible. We specifically built in opinionated functions like get_interest and get_geo that wrapped around the core functionality in the global library so ​​WordPress developers would have an easier time implementing personalization functions. There is no built-in admin UI. As an SDK (and an MVP at that), we wanted to focus on allowing development teams to build out the interfaces that made the most sense to them.
• #25: However, proof of concepts have been built to show how one might display personalized content using CMB2 and Advanced Custom Fields.
• #26: Analytics are core to understanding how your personalized content is performing, so integrating with Google Analytics and Google Tag Manager was an easy early requirement. We’ve built an option in the admin to add your GTM code, but this can be completely disabled if you’re using an existing plugin or solution for Google Analytics. When I polled a couple former colleagues of mine about how they typically handle adding GA, the consensus was that sometimes they add a setting, sometimes they hard-code the code, and sometimes they use a plugin, so I wanted to make sure we supported any of those options.
• #27: Working with our professional services team, I was able to request a specific shape for the data to come back to the server in from the CDN that aligned with how we’d architected the code. The VCL rules are fairly simple and the headers we get back are formatted in a way that’s easy to read and parse. Geo data is determined by where you are located in the world and what your connection is. We can vary down to the city level and we include things like connection speed and connection type, as these might be useful in some use cases. Interests are categories by default, but any taxonomy can be used. An interest threshold is set to 3 by default, and when you hit three pages that are tagged with a specific category, a cookie is stored in your browser that defines that term as an interest. A running count is being stored to localStorage to keep track of all the terms on pages you visit, so if a term takes the lead, that becomes your interest. Right now, only one interest at a time is supported, but we are thinking about how we might support multiple interests in the future.
• #28: Composer as a foundational technology From the beginning, I knew I wanted to use Composer as a core component of what these SDKs would become. Part of this was informed by my previous experience with Altis, which itself is a metapackage composed of multiple other packages, which themselves are metapackages of the different modules and features of the platform. While Composer hasn’t been embraced in core within WordPress the way it has in the Drupal community, being able to link everything together into a single `composer require` despite them potentially being located in different repositories was a key feature. However, for WordPress, the plan is and always has been to provide a ​​plugin for the WordPress plugin repository – so we spent a fair amount of time developing a build process which compiles all the dependencies and assets into a single zip file that you can download and get everything already packaged. When we’re ready to launch, we’ll be pushing a version of this to WordPress.org, but for now, those packaged releases live on the Releases page of our WordPress plugin.
• #29: Documentation is the product From the beginning, we were very opinionated that documentation was the product. The code was nice, but great code without documentation is virtually useless. Especially in a complex and confusing environment like content personalization. All of the functional code is documented in the SDK, and we have a detailed guide on Pantheon.io/docs about Edge Integrations, how it works and how to set it up.
• #30: Test all the things We use three different kinds of test suites for different things PHPUnit Does not use WP_Unit but does pull in a WordPress instance to run various tests that use WP functions Robo Task runner to run e2e tests using Terminus commands and curl requests TestCafe Javascript-based e2e testing similar to Cypress to test user interaction and experience things
• #31: Gather feedback This is where we are now We’re working primarily with partner agencies that are interested in personalization and encouraging them to use our SDKs and give us feedback so we can continue to build this out and expand our product offering Some future things that I’m particularly interested in are integrations with CDPs (customer data platform) like Segment to offload the segmentation and user profiling to a system that is built for that to reduce some of the granularity of the variants we get, particularly with geo Along the same lines, work at the CDN layer to possibly have more advanced VCL rules that are smart enough to use fallback rules to reduce granularity – e.g. I might be the only visitor from Salt Lake City, but I live in Utah, which could be a regional grouping, or fall back to the country if a cached variant for utah doesn’t exist Insights metrics Your “interest” page is doing particularly well in “geolocation” Integrating with the Pantheon decoupled solution that’s currently in development All our tools are open source, we use Fastly, so this isn’t meant to specifically sell you on a Pantheon product, specifically, but I think it’s interesting the angle from which we’re looking at personalization in contrast to what’s out there today and I’d love to chat more about it!
• #32: And that’s it! Thanks again for letting me share this stuff that I’ve been working on and back to you, Brian!