How to build a pricing engine for e-commerce promotions and offers

The convergence of in-store and online retail operations in recent years has put significant new demands on the management of pricing, offers, and promotions as compared to pre-omnichannel times. The outdated, simplistic approaches to determine the price of products and promotional offerings are still prevalent in the industry, leading to loss in revenue, market share, and margins.

Some of the issues may be traced to the use of old products like Oracle ATG as the core pricing system, suffering from a lack of critical capabilities such as predictive analytics, machine learning, real-time modeling and personalization, to mention a few. As a result, the business cannot accurately predict the results of any specific campaign, optimize discounts, or identify lost margins when discounts combine to dip the sales price below the cost.

We have had the privilege of working with a number of large retailers on the design and implementation of these next-generation pricing engines, which are built with open source technologies and run on the cloud. In this blog post, we will describe our approach to building microservices architecture for a modern offers, promotion, and pricing engine learned over several large-scale implementations with the leading omnichannel retailers.

Different retailers have different business models when it comes to pricing. Some may focus on luxury products, some may position themselves as having an everyday low price, and others may have an expensive list price, but provide a lot of discounts, promotions, and offers to entice shoppers. While the specific mechanisms of customer engagement for those types of retailers may be different, all of them have the goal to offer the right product to the right customer at the ideal price point. Matching the product with a customer via personalized search is something that we touched on in this article. Understanding customers by collecting and analyzing data is a topic for a different discussion and requires many systems, collectively called customer 360, to work well in concert. Matching the customer with price, and more specifically, managing various components of prices, is going to be our main focus in this post. The reason we are doing an examination about prices, offers, and promotions in a single discussion is that all of them are components that contribute to the price that a customer pays for products in a shopping cart. This can be seen in the infographic below:

During the shopping experience, a customer gets exposed to prices, offers, and promotions in different areas. Product discovery starts with product search and browse, and sometimes even earlier, in external search engines via SEO, or emails via marketing campaigns. Then, when building a shopping cart, a customer needs both up-to-date and correct information about the total cost of the cart, including base prices, discounts, offers, promotions, coupons, taxes, etc. In all cases, it is crucial to show accurate information and calculate the total cost in real-time, since latency in a shopping cart is especially important due to the consumer frustration it causes.

Keeping that in mind, let’s review the major components that make up the total value of a shopping cart:

• Price: Although price is the most straightforward component, it may be difficult to manage as well. Prices can vary by channels, regions, and stores. They may also vary by time, often with changes carefully planned and managed in advance by the business team. Prices can additionally have multiple tiers such as base and clearance.
• Promotions: Promotions can come in a multitude of different forms. Depending on a retailer, these may be simple or extremely complex. Similarly to prices, promotions are configured by a product or a family of products, and may vary by channel, region, store, and time. The complexity typically comes from managing interactions of promotions tied to the price of products, other promotions, offers, etc. For example, a retailer may have a promotion on a product until the product sells at a base price. When the product goes to clearance, the promotion may be disabled to prevent an unnecessary loss of margins. Common promotion types include, but are not limited to:
  • Buy one (or X), get one (or Y) free, also known as BOGO: A product offered for free may be the same or a different complementary product compared to the original one that triggers the promotion.
  • Tiered promotions: when a customer spends $100, the promotion is 10%, but when a customer spends $150, the promotion is 15%.
  • Dollar off or percent off a product or shopping cart: May be combined with tiered promotions.
  • A variation or combination of the above categories: This could be something like a dollar off or percent off when buying a certain number of specific products.
  • Free shipping.
• Offers: We distinguish between offers and promotions, while some retailers do not. We didn’t see a very consistent terminology, but the general theme is that offers are typically customized and given to specific customers or customer groups. Offers are also typically applied on a shopping cart level. Offers may be multi-use or single-use, may come as paper coupons or available digitally on the website, or be given as a result of a loyalty program or as an incentive to shop or return to an abandoned cart. Similarly to promotions, depending on the retailer, offers may follow simple or complex business rules, may vary by region, store, channel, customer, and time, and may work differently in the presence or absence of other promotions and offers.
• Taxes and shipping costs: Although the most self-explanatory, the business logic may be complex, especially when it comes to shipping costs, and the various offers and promotions associated with it. Fortunately, there are plenty of libraries for tax calculation, like Taxware, which provides straightforward mechanisms of applying taxes directly to the shopping cart.

From the customer perspective, the pricing, promotion, and offers engine should provide a consistent and fast experience by correctly calculating the total value of a shopping cart and an order. From the merchandising perspective, besides offering a rich set of offers and promotions, and being able to configure flexible business rules, it is very important to have a single authoring interface where all prices, offers, and promotions can be set for all channels and stores.

The non-functional requirements for the system are typically very strict, including:

• Support catalogs of several million SKUs.
• Support up to several thousand stores.
• Handle billions of current and future prices.
• Support millions of current and future promotions and offers.
• Scale horizontally to handle thousands of transactions per second with milliseconds latency in peak.

Keeping in mind the requirements above, the conceptual architecture of the service becomes clear:

This architecture consists of several major components: the core platform, the analytics and machine learning to augment business decisions, and the clients of the system. We assume that the core platform is omnichannel and the client services are also omnichannel. These consist of the wallet that may require information about available offers, the search & browse services that show pricing information during product discovery and export it for external search engines, and most importantly, the cart & checkout services that use the core platform to calculate the total value of the order. We will get to the analytics and machine learning components later in the article. For now, let’s zoom into the core platform.

The core platform may consist of several microservices, and the scale of each microservice is often dictated by the specific retailer’s environment and the complexity of the business logic. Major components, however, stay the same. The core platform needs to provide an authoring web user interface to the merchandising managers, a transactional service for downstream components, and integration services, including batch and in-stream synchronization. The architecture of such a system can be seen in this infographic:

In order to fulfill strict throughput and latency requirements, we recommend implementing a caching layer with a proven and high-performance technology, such as Apache Ignite. The bonus of this approach is that it can be used as a side cache or run calculations inside. When the caching and computation fabric is implemented correctly, the choice of a database is less important. Cassandra may be the natural choice for cloud deployment, but to support transactional capabilities and convenient SQL semantics, MySQL or PostreSQL are better. In many cases, even databases provided and hosted by cloud vendors may be a good option to save on deployment and configuration effort, such as Amazon RDS, Google SQL, Google Spanner, Azure SQL, or Azure Cosmos.

One of the biggest challenges in implementing omnichannel pricing, offers, and promotions is ensuring that the in-store experience is consistent with the online experience, and that the merchandising team can use one user interface to configure information across channels and in all of their stores. In the best case scenario, when a retailer doesn’t have many physical stores, or when a stores’ connectivity to the Internet is very good, the simple method may work. In this case, the services can be deployed in the cloud, and every time a customer needs to check out in the store, the point of sale or kiosk will call the central cloud instance of the service.

Unfortunately, not all retailers can always provide a fast and reliable Internet connection from stores. Even if basic connectivity for payments may work, the bandwidth, latency, and stability may be too poor for advanced services, such as pricing, offers, promotions, and an updated catalog. In this case, we recommend deploying local instances of the system in stores. That way, availability of up-to-date data and low latency is guaranteed. Store deployment comes with its own set of challenges, but with the proper implementation, most of them can be successfully mitigated. A reference architecture of this system is shown below:

Let’s get through the typical challenges and approaches to in-store deployment one by one:

• First of all, the store hardware may be different, and local IT operations personnel may not be well-versed in deploying and supporting modern applications. To mitigate this challenge, the services may be designed to support simpler technologies when deployed in-store. For example, local in-process cache and a simpler database, such as MySQL or PostreSQL. While it may look like a requirement to support different codebases for the cloud and store, in practice the codebase can be the same for each, and the build process may produce different configurations, which will be tested in their own respective environments. We used this technique in our projects, and it worked well.
• Second, data replication between the cloud instance and the store instances may be a challenge. To work correctly, several best practices should be followed. First, only data that is needed for a particular store should be copied to that store. Second, only deltas should be copied during synchronization by default. Next, the self-correcting mechanism should be implemented to constantly check for data consistency. And finally, secondary synchronizations should be requested when corruption is found.
• Third, in case the store gets disconnected for a long period of time, the business team should support configuration of future prices, offers, and promotions. Therefore, the platform should replicate not only current data, but also data for several days or weeks in the future.
• Last but not least, to deal with all of the challenges above, a management portal should be implemented to allow the central SRE team to observe the status replication and the health of each in-store instance. While the system should self-heal and self-correct when possible, the portal should provide status updates and an alerting mechanism, and allow requests for full synchronization for specific stores when needed.

While it may seem unnecessary in practice to support the store-level deployment of pricing services, most of the infrastructure that needs to be implemented is required for deployment of store components anyways. These components include point of sale, inventory and order management, and other customer facing and back-office systems. So, the base infrastructure and deployment process will be reused across all applications and services that need to be deployed in the store. Frankly, deploying and operating store applications is a separate interesting topic, and we hope to discuss it in greater detail in a future blog post.

During our past projects with large retailers, we have accumulated a number of techniques, tools, and processes to help optimize pricing systems. These range from foundational approaches, to deployment and monitoring, to utilization of advanced analytics including AI and ML for troubleshooting, identification, and resolution of issues.

Promotion-driven retailers typically have a large number of offers and promotions configured in the system at any given time, and this information changes frequently. At some point in time, the amount of offers and promotions may become too large to efficiently manage for a few merchandising managers. As can be seen in the infographic below, this traditional setup also leads to a reactionary approach rather than one that can respond in real-time:

In the best case scenario, the traditional approach with a handful of merchandising managers leads to an increased effort. Unfortunately, in our experience, it may also lead to lost margins and revenue due to excessive promotions. The challenges typically come from two sources:

• Offers and promotions may overlap, inadvertently creating deeper discounts than needed.
• Merchandising managers may enable promotions with big discounts, when in reality, customers would buy the product with smaller discounts.

In both cases, analytics and machine learning may find overlapping and unintentionally deep discounts. Ideally, business users operate on the level of business objectives and constraints, and the machine learning is implemented to generate granular decisions, such as creating individual promotions and offers for specific customers, as seen below:

A full-fledged algorithmic platform can cover the majority of business decisions. We won’t cover all of the possible use cases in this article, but if you are interested in learning more, we recommend taking a look at our book on this subject. While such implementation on a full scale may take significant effort, the first step is to implement a safeguard that can analyze combinations of promotions and offers, and alert business users when discounts are too deep. The threshold for alerting can be set either manually for various channels, regions, or product categories, or analyzed based on historical demand. In our experience, even such a basic implementation may save millions of dollars in margins.

Another small step towards an algorithmic approach is implementation of real-time offers for specific use cases. One of these use cases is that of an abandoned shopping cart. Abandoned shopping carts are one of the biggest problems in retail, and reducing abandonment rates may increase revenues significantly. In our experience, even a simple implementation of real-time offers to customers with a high propensity to abandon their carts may decrease the cart abandonment rate by several percentage points. The exact strategies of what offers to give, how to give them, and when to give them may vary. If behavior analysis shows the customer has a high propensity to abandon the cart, the offers may be given while the customer is still on the site, to prevent this from occurring. Or, if analysis shows that there is a high chance that the customer returns to a previously abandoned cart when given a certain incentive, the offer may be sent via email after they’ve already abandoned it. While the results of such offers may be positive, we advise implementing machine learning models carefully to avoid giving offers to customers who already have a high probability of conversion, as doing so would reduce margins.

Implementation of a modern pricing engine, offers, and promotions service enables a rich, omnichannel experience for customers. Equally important, it provides omnichannel management experience for merchandising managers and business users, increasing their efficiency, and leading to higher revenue and better margins. Conversion rates should increase, and cart abandonment rates should decrease, giving proof of pricing being at a better place for consumers.

Initial implementation and replatforming of base services from ATG allows the closing of immediate gaps, such as omnichannel experience, richness, and flexibility of business rules required by a specific retailer. In the long run, such replatforming opens opportunities for further improvements, such as embedding advanced analytics, machine learning, and an algorithmic approach. These improvements may improve the quality of prices, offers, and promotions, and improve the bottom line by an even greater margin.

The solution we laid out should help with creating a more modern omnichannel pricing engine for any retailer, as it uses open source technologies and a microservices approach. Many retailers move off ATG because they want consistent and highly personalized prices and promotions, which are not otherwise provided.

If you’d like to learn more about ATG replatforming, be sure to check out our upcoming posts in this series. In future articles, we will discuss building other services of the digital customer experience platform, including a customer profile, cart, and checkout. We will also touch on how to implement a product information management system, content management, and customer 360 capability. If you want to learn more, give us a call or leave us a comment!