Programmatic Success: Everything Interesting Happens at the Margins


Adam Epstein

Programmatic buying doesn’t eliminate the need for humans. If anything, it has actually increased the value of curious, competent employees.

Full disclosure: adMarketplace hasn’t always been programmatic. A couple of years ago we built our platform for granular custom optimization across 15 different dimensions – and we learned the hard way that humans simply cannot efficiently optimize across so many different dimensions in real time. As a result, we defined best practices and got to work building a bid optimization algorithm. After we switched to programmatic, bid changes were up 10 times. This produced results for advertisers, but also generated a tremendous amount of new data, which we had to figure out how to analyze.

Make Big Data Smart Data

If you’re only analyzing top-level data, you may see trends, but you won’t learn anything interesting. Choosing the right metrics to analyze is important, but it’s only half the battle – you still need to dig deep to find useful insights. Turns out, to get smart insights from big data, we learned to find small data sets.

Identifying “outliers” is the best place to begin granular analysis because the most interesting stories are found at the extreme points of a data set. The stories this data can tell can be good or bad news for your business.

Data outliers are statistical anomalies that fall outside the “normal” range, or defy expectations. Whenever we observe an anomaly — for example, an unexpected spike in click-through rate (CTR) or a drop in conversions – we depend on humans (remember them?) to dig into the data and figure out what the heck really happened.

Smart Insights Come From Curious People

Curiosity is an innately human skill. We can write programs to algorithmically bid and match ads to publisher queries. We can write programs to track and report data. We can even write programs to spot the anomalous data outliers discussed above. However, to analyze these anomalies, and actually learn something from them, we need the creativity and insight of a curious human mind.

Consider the following real-life examples:

Volume Outbreak

Our platform tracks and reports publisher query volume by category in real-time. Last week, we noticed a three to four times volume spike in a relatively obscure category: medical supplies. At the time, we weren’t matching ads to these queries. But something interesting was happening at the margins. We assigned an analyst to investigate.

  • The uninformed analyst looks at this data and thinks: Great! Look at all this volume! and connects some advertisers. This is dangerous.
  • The semi-curious analyst looks at the same data, connects some advertisers, and then investigates the clicks and conversion volume. If there are no conversions, they may conclude – this traffic is worthless - and then block the publisher altogether.
  • A truly curious analyst digs deeper. They will notice the volume spike and before doing anything, start looking for indirect or external causes.

In this case, our data science team looked at the individual search queries for this category. If you’ve been following the news from your quarantined bunker, you wouldn’t be surprised about the source: Ebola. When the first patient hit New York City, local query volume for anything Ebola related exploded. As a result, we added “Ebola”-related keywords to advertisers who offer medical supplies and related products. Relevant advertiser offers saw an average 10 percent increase in volume. Algorithms caught the surge, but we needed human analysts to figure out what action to take.

Missed Connections

Whenever we add new publishers to our search partner network, we create certain expectations based on experience. A few months back, some of our account managers noticed that a new publisher was producing click volume well below our expectations. Even though query volume was where we expected, we were averaging 20 clicks per day for an advertiser when we expected more than 1,000.

An algorithm would have stopped matching this publisher to this advertiser, and the non-curious analyst may have approved. In our case, a curious analyst dug in and found that keywords were not matching a relevant account because the keywords mistakenly loaded as negative keywords. Algorithms are great at matching thousands of keywords per second, but they cannot recognize simple human error (takes one to know one).

Programmatic buying does not eliminate the need for humans. If anything, big data provided by programmatic elevated humans. Marketing is a mix of art and science, and while programmatic handles the science, creative and curious humans are best able to find the cause of data outliers. If you want to use programmatic efficiently, you need to hire and reward curious people who are eager to discover and investigate your outlier data.