Morris: CycleGAN mapping aligns disparate model embeddings without paired data
Jack Morris · Information Theory for Language Models: Jack Morris · Jul 2, 2025 · at 46:25
Cornell Tech PhD researcher Jack Morris discusses his research adapting CycleGAN from computer vision to align different text embedding spaces without paired training data.
0:00 / 0:33exact quote · 33.1s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“We took it and we applied it to model embeddings where instead of zebras and horses, we have like BERT embeddings and GPT embeddings, or like two completely different models with different architectures. So I think these are GTR, which is a T five based retrieval model and GTE, which is based on BERT. So they have different training data, different architectures, different downstream objectives, different embeddings. But yet when we do this cycle GAN in the embedding space, they just perfectly sort of snap to the same place, which is amazing and has some pretty deep implications of like the platonic stuff.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Jack Morris
Morris: ChatGPT could likely have been built using RNNs instead of Transformers
“And I think like, we honestly probably could have gotten this with RNNs. I know like the scaling laws paper shows that RNNs have worse curves for scaling, but probably people would have been like, I bet you could have built chat GPT with a very sophisticated R…”
Jack Morris: Fundamental AI science shifted to companies due to academic compute limits
“That's when I think things really started to change in terms of the types of questions you wanted to ask can't always be answered with academic resources. So a lot of the like fundamental kind of like boundary pushing and AI science moved into companies.”
Morris: AI paradigm shifts are driven by novel datasets, not architectures
“I think, like, all of the things that I would consider paradigm shifts in the Kuhnian sense came from a new technique, but trained on new data, and I think the new data is super, super important”
Jack Morris: Most AI research was previously open, but is now closed
“Most stuff was open. Now most stuff is not open.”
Morris: Re-implementing new paradigm shifts quickly is the best AI grad strategy
“Honestly, if I were to give advice to a younger grad student, I think the way to do it would be literally just like sit and wait until the next kind of paradigm shift and then just immediately start working as fast as you can to like re-implement it. Like, I d…”