Embedding machine learning reranking inside Apache Solr improves search performance
Parth Vasa · How to Train Your Search Engine // Parth Vasa, Bloomberg (FirstMark's Data Driven) · May 24, 2017 · at 10:59
Parth Vasa, Head of Data Science at Bloomberg Engineering, explains why Bloomberg integrated machine learning search reranking directly into Apache Solr.
0:00 / 0:05exact quote · 5.0s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“And also, it reduces the hop between two services, so your performance gets much better.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Parth Vasa
Bloomberg chose Apache Solr over Elasticsearch to maintain open-source control
“One was, we were looking for something that was truly open source, where we had a lot more control over open source and, sorry, source code, and a lot more saying how the direction goes.”
Bloomberg cannot rely on large-scale A/B testing or usage data
“So a lot of the luxury that other companies have, like doing large scale machine learning based on massive amount of usage data, or doing large scale A-B testing to decide which model works better, doesn't work for us.”
Learning-to-rank algorithms continuously improve search models without manual human tweaking
“We had a huge improvement once we started using it, but more than that improvement, the good part about learning to rank is it always continues to improve your model without constant need of people tweaking it.”
Bloomberg limits unsupervised learning to data bootstrapping due to precision needs
“We use unsupervised learning a lot to Sort of bootstrap our data. For example, word to whack, right? It's a perfect unsupervised learning algorithm. We use that a lot to, for query reformulation and all that, but, ah, it's a little risky for something as high …”