It is the end of the year. The holiday season. The moment of truth for many retailers and the technologists that work for them or have them as customers. Black Friday is over, the last-minute shopping is over. Now it is time to judge if you made book or not and plan for next year.
As a customer, I purchase a decreasing amount of things from “not Amazon,” and generally when I do it is because the retailer offers specialized expertise, service, or selection. The only other reason I generally buy elsewhere is because I need it “right now” and am willing to drive there to make that happen (generally, that means food). An informal poll of my friends reveals that I’m not alone.
So, retailers are both specializing and having to understand their customers better and develop a closer relationship with them. This means excellent in-store service but also excellent online service. Luckily, specialty retailers have a home-court advantage in both places.
Think about it: For personalized recommendations or targeted promotions, Amazon has to create a “general use” algorithm that works for all products all the time. Any tweak either has to be tested against all product lines or against a general subset. But a specialty retailer can work within “camping equipment” or “high-fashion clothing” or “automotive parts” and realize that just because you bought a car cover doesn’t mean you’re a “car cover enthusiast.”
(Full disclosure: I work for Lucidworks, which sells search software that does much of this stuff for retail sites.)
There are six issues a modern retailer must work through to be successful.
Issue 1: Providing relevant search
Simple search is matching keywords (aka terms) in a user’s search to keywords in a product description. Next to no one just does that any more. Modern search takes into account multiple things. First, how rare the term is in all your product descriptions. Second, whether it is in a short field like the product name or a long field like the technical specifications or description (this is called TF-IDF). Moreover, modern algorithms (like BM25) smooth this to prevent more mentions overly boosting a result after a point.
Another key issue is “faceting,” or limiting search to a department or relevant category. This is especially important for retailers with many different departments or product lines. Often, the best customers know exactly what they want (“15-inch screen, i7 processor, at least 1TB storage”) and facets let them filter that search. There are other factors to consider in search relevance like phrases versus keywords, spelling, and synonyms.
However, offering a really personal experience means actually personalizing the results for your most loyal customers. Moreover, acquiring new customers is a lot more expensive than selling more to your existing customers, so retailers put a lot into customer loyalty. Heck, the trade shows and magazines usually have “loyalty” in the name.
Issue 2: Capturing and aggregating user behavior
The first issue that modern retailers have to face is capturing user behavior. I’m not talking about your nervous tick that makes your chew your nails. I’m talking about:
- Searches (what did they search on, and did they find anything)
- Clicks (especially related to searches or promotions)
- Add to cart (what did they add to a cart or wish list)
- Purchases (what actually got purchased)
- Paid for (not everyone’s payment goes through)
- Returns (what did they not only dislike or have issues with but to the point of going to the trouble of returning it)
These kinds of behaviors are a type of time-series data. Moreover it tends to have a shelf life. If you were looking for a new oven and didn’t buy one, six months down the line I don’t want to keep showing you ovens. It just isn’t relevant any more.
These behaviors should then influence what gets shown to users in multiple places from the home screen to the product detail page and even affect the rank of things in searches.
Issue 3: Targeted promotions
One of the worst things a customer can call you is “that company that sends me lots of email.” It means your communications are going unread. Promotions tend to come in multiple forms:
- The bone-headed promotion just sends users things they clicked on but didn’t buy. This can be helpful, but the “you just left our site, didn’t you mean to buy XYZ?” ten minutes later actually makes me unsubscribe.
- A smarter promotion sends complementary products.
- Also some promotions are for a specific item or set of items that the retailer is trying to push. Sending Honda parts to Mazda owners or women’s clothing to people who only buy men’s clothing is probably not a great idea. Some of this can be discovered with simple demographics that were gathered at account creation. However, bigger, better more targeted promotions look at behavior. There are machine learning algorithms for making recommendations based on user behavior. You can also reverse that and find the best users likely to buy an item that you’re trying to unload.
Issue 4: Personalized home screen
Run your favorite web analytics on your retail site (or any internet site) and you’ll find that the front page is the highest-traffic page. This means that your best opportunity to grab the user is on the very first page of your site. What a retailer offers there should be heavily tailored to what a user is likely to buy.
In a way, this is similar to the targeted promotions—but in reverse. You want to know what items that you have that a returning user is most likely to buy. Sure, you want to offer things they were looking at last time or items in the same category as their previous searches. However, there may be a reason they didn’t buy those things. The computer can do what a personal shopper would: Observe what a user looks at and actually buys, then adjusts recommendations to that user. These are the kinds of recommendations a retailer should put on its front page next to any targeted promotions.
Issue 5: Personalized complementary items
If you look at Amazon and see its “frequently purchased together” suggestions, this is a simple statistical analysis of which item made the cart next to another item. These types of automated recommendations are incredibly effective. These help the user with choices like “Yes I want a brush to clean the coffee grinder I’m buying, good idea” and “Oh, I need bike chain oil because I bought a new bike.” This isn’t just good business (because they’re more likely to buy from you and maybe look a little less closely at price) but good service (because getting something only to discover you don’t have everything you need is annoying).
There are automated recommendations that go beyond this, however: What other things are other users interested in that thing also interested in that thing? Again, this is an automation of what happens in high-touch customer service in stores. For users already in buying mode, this is a key opportunity to upsell.
Issue 6: Scale and resilience
Remember ATG? It had a lot of features that were personalized and ahead of their time. However, it had a lot of services and they all pegged an RDBMS in client-server fashion. The way it dealt with data (a terrible implementation of an object-relational mapping system) was slow and error-prone (if you tuned it to be fast it tended to blow out memory and core dump). The search in it wasn’t great and all of the personalization features were slow.
Modern e-commerce software needs to scale to multiple nodes. It needs to be able to distribute the data and stay up if a node (even data node) goes down. The world has also gotten more unpredictable. Who knew a hurricane could hit New York City and take out the retail capital of the world? So, modern e-commerce software needs to be cloud-ready and run on multiple availability zones or datacenters.
Also it is safe to assume any user waiting more than a second or so for a page to draw has already gone to any other site on the internet.
All of this means abandoning a traditional client-server architecture. It means that you need some kind of shard-able NoSQLish store. You need clustered communications and data replication. You need WAN replication. You need a lot of asynchronous processing. Yet losing data isn’t really an option. This goes beyond the old ATG-style architecture and really demands a kind of modern Amazon or Google-like architecture.
What a modern retailer must do to be modern
Modern retailers need close relationships with their customers. They need to offer better service than Amazon in particular.
One of the key ways to do that is by capturing user behavior and using modern machine learning algorithms to make recommendations and custom-tailor results.
Second, a modern retailer has to have an architecture that scales, is cloud-ready, and is resilient.
If you didn’t make your goal this year, next year is more competitive and these modern technologies are essential to making your e-commerce site more compelling.