Amazon at ICML: Industry and academia meet at Expo Day

At this year’s International Conference on Machine Learning (ICML), Alice Zheng, a senior manager of applied science in Amazon’s display advertising organization, was one of the cochairs of Expo Day, a chance for the conference’s corporate sponsors to report their latest research and demonstrate their latest technologies.

“ICML Expo talks are pretty much like research talks that you would see in the main conference, and here is where academic conferences differ a little bit from industry conferences, based on my observation.” Zheng says. “I used to be part of a small local startup, doing machine learning. And we attended a lot of industry vendor conferences, for marketing. I had come from an academic background, and I walk into this giant pavilion with row upon row upon row of vendors with booths that have people dancing aerobics to disco tunes every afternoon from 3:00 to 4:00 to draw a crowd. I was like, ‘What is happening here? This is such a circus.’”

At academic conferences like ICML, Zheng says, sponsor presentations have a much different tone. “In academic conferences, sponsors are generally companies trying to hire,” Zheng says. “That changes how they present themselves and what they talk about. The Expo chairs are responsible for reading proposals and making accept or reject decisions. We want to uphold a certain level of quality: They should be research talks. They should be informative. They should be relevant to the topics of the conference.”

ICML’s Expo Day, which took place on Sunday, was organized into three tracks, Zheng explains. One track consisted of hour-long research talks and demos. The other two consisted of four-hour workshops in which attendees could gain first-hand experience working with new technologies developed by conference sponsors.

But while Zheng and her cochair, Hsuan-Tien Lin of National Taiwan University, upheld rigorous scientific standards in their evaluation of submitted talks, there are, she says, differences in emphasis between industry and academic research.

“One of the disconnects that I see between industry and academia is that academia focuses way more on modeling and math,” she says. “The full cycle for machine learning development and operations starts with ideation: you come up with an idea, or there’s a problem that you want to solve. You formulate the problem, you propose a solution, you test it, both offline and online. You iterate on the process, and you eventually end up with something that works well. You work with engineering to get a robust implementation — and that can take a while — you deploy it, and you monitor it, to make sure that it’s continually working as it should.

“That’s the full cycle. In academia, I would say that oftentimes even the ideation is fairly rote — people work on pre-established problems. The focus is on the proposal of new methods, new solutions, followed by very light testing. It’s like one out of five steps of the entire cycle.

“Several years ago, I created a talk about operating machine learning models, and I highlighted the importance of evaluation and metrics. It’s tempting to say ‘Oh, I’ll just measure the AUC [area under the curve].’ But in reality, it’s more complicated than that, because in many, many application areas, it’s not one homogeneous set of data, one monolithic model. The data can be subdivided, and you should look at each slice separately. How do you slice the data? How do you create metrics that are stable enough to operate on and still sensitive enough that if something were to go wrong, it would appear abnormal?

“So I created this talk, and I gave it at our intern symposium, and people loved it. Students were raising their hands and saying, ‘How can we learn more? Where can we go to do more of this?’ This is the kind of thing that industry research excels at. Academia does not have the same amount or variety of application data.”

Explainable AI

Of course, industry researchers stay abreast of the latest academic research, and from her vantage as ICML Expo chair, Zheng can see which academic trends have recently begun to take hold in industry, as well. One of these, she says, is model explainability.

Model explainability is a question of fundamental scientific interest: neural networks are black boxes, and it’s natural to want to understand how they do the remarkable things they do. But, Zheng explains, it’s also a question with immediate business implications.

“Sometimes the business needs insights, in which case an explainable model is the thing that they’re looking for,” Zheng says. “Oftentimes you need some way to examine if something is going wrong. It gets back to model operations.”

The need for model explainability, Zheng says, frequently arises in the context of her work at Amazon, which focuses on the design of algorithms for automatically bidding on advertising space on other sites and responding to bids for space on Amazon sites.

“Digital advertising, or programmatic advertising, operates in a high-volume, high-velocity environment,” Zheng explains. “Websites send bid requests out to ad exchanges, and then the ad exchanges contract with different bidders. And the bidders say, ‘Okay, I’m willing to pay this much for an opportunity to show an ad.’ Our algorithms need to make split-second decisions about where to bid and how much to bid.

“The advertisers want to know, ‘What can I do to improve the performance of my ad?’ They’re trying to understand this black-box system. They are always looking for insights in terms of, ‘How can I better set up? What actually drives conversions?’ 

“And then when we operate our model, we are also interested in general model analysis. Say our conversion rate prediction model is giving a lot of weight to ads that are selling consumer products. Because customers may buy toilet paper every month, whereas they buy a camera maybe every two years. Is that helping advertisers reach the right audience? By being able to understand what the model is doing, we can detect problems, and we can improve it.”