Visual question answering is harder than captioning because it requires reasoning
Antoine Bordes · Artificial Intelligence at Facebook // Antoine Bordes, Facebook [FirstMark's Data Driven] · Nov 9, 2016 · at 9:15
Antoine Bordes, AI Research Scientist at Facebook AI Research (FAIR), presents on the evolution from computer vision object detection to visual question answering at Data Driven NYC in 2016.
0:00 / 0:23exact quote · 23.1s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“So, what people try to do now is that to move to caption what's called caption generation, which was actually super promising, but actually people realized that actually the machine wasn't that good, to what's called now visual question answering, which is more about trying to ask questions about images. This is much harder, actually, and we, the task has been very popular for the last two years. Actually, it's too hard, which means that there is a big ceiling”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Antoine Bordes
Bordes: 2016 image recognition AI uses Yann LeCun's 1993 neural architecture
“This is exactly the same architecture of what's being used right now.”
Bordes: Direct machine reading will eventually match IBM Watson's performance
“The goal is that, ah, eventually, by trying to understand the text directly, you can actually at least equal, all the information by Watson is in text, in free text.”
Real-world applications must balance machine learning with rule-based extraction pipelines
“In terms of developing real application, I think you should really try to balance both, because it's true that in the example I showed, If you have a very good process to basically already isolate all the entities and if you want basically to use something lik…”
Most 2016 production computer vision systems lack bounding box capabilities
“This is, right now, I would say that the best systems are doing this in production. Most of them are doing this. And some are not even doing the boxes.”