StarZero All shows Try the podcast clipper →

Dec 1, 2024 · 41m · latent-space

[Paper Club] Embeddings in 2024: OpenAI, Nomic Embed, Jina Embed, cde-small-v1 - with swyx

0:00 / 0:00
▶ Watch on YouTube →

gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions

In this Paper Club session, swyx leads an in-depth analysis of the 2024 embedding landscape, examining architectural innovations like Matryoshka learning, task-specific LoRA adapters, and contextual conditioning across key models from OpenAI, Nomic, and Jina AI.

How this conversation actually went

Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →

The hosts as informed peer 5.6 Guest teaching 2.6 Guest disagreement 1.0 The hosts pushing back 0.7
05100:0015:0030:000:00–2:15 · The hosts as informed peer 6/10 Overview of 2024 Embeddings and the MTEB Benchmark swyx opens the session with a broad, confident walkthrough of the current state of text embeddings in 2024 and the Massive Text Embedding Benchmark (MTEB). He outlines trade-offs between model size, efficiency, and benchmark rankings without any guest intervention.2:16–9:33 · The hosts as informed peer 5/10 OpenAI Embeddings and Matryoshka Representation Learning swyx explains OpenAI's Matryoshka Representation Learning (MRL) dimension reduction. Eugene educates the room on production latency constraints, pointing out that 1024 dimensions are often unfeasible for real-time ANN lookups, while 64 or 128 dimensions provide viable latency.9:34–15:54 · The hosts as informed peer 6/10 Nomic Embed and Fully Reproducible Training Pipelines swyx highlights Nomic's fully reproducible pipeline, noting details like masking percentages and standard BERT architectures. Attendees discuss the lack of dedicated code embedding models in the open source ecosystem.15:55–27:23 · The hosts as informed peer 5/10 Jina Embeddings v3 and Task-Specific LoRA Adapters swyx discusses Jina Embeddings v3 and task-specific LoRA adapters. Eugene steps in to correct swyx's mental model regarding LoRA layer placement, explaining that LoRA weights modify all attention and MLP projection layers rather than just replacing the final classification layer.27:42–30:27 · The hosts as informed peer 6/10 Multimodal Embeddings and Jina CLIP v2 Architecture swyx walks through the Jina CLIP v2 release, highlighting how freezing text embedding backbones and training vision adapters creates performant multimodal representations.30:28–35:09 · The hosts as informed peer 4/10 Discussion on Biomedical and Domain-Specific Embeddings SPEAKER_03 inquires about medical domain embeddings and protein pathway retrieval. swyx and participants discuss the practicality of fine-tuning generic base models versus using niche domain foundations like MedSAM.35:10–41:43 · The hosts as informed peer 7/10 Contextual Document Embeddings and Two-Stage Adaptation swyx explains Contextual Document Embeddings (CDE) and two-stage corpus conditioning, distinguishing it from traditional fine-tuning by clarifying that it avoids gradient updates and operates more like contextual KV caching.0:00–2:15 · Guest teaching 0/10 Overview of 2024 Embeddings and the MTEB Benchmark swyx opens the session with a broad, confident walkthrough of the current state of text embeddings in 2024 and the Massive Text Embedding Benchmark (MTEB). He outlines trade-offs between model size, efficiency, and benchmark rankings without any guest intervention.2:16–9:33 · Guest teaching 5/10 OpenAI Embeddings and Matryoshka Representation Learning swyx explains OpenAI's Matryoshka Representation Learning (MRL) dimension reduction. Eugene educates the room on production latency constraints, pointing out that 1024 dimensions are often unfeasible for real-time ANN lookups, while 64 or 128 dimensions provide viable latency.9:34–15:54 · Guest teaching 2/10 Nomic Embed and Fully Reproducible Training Pipelines swyx highlights Nomic's fully reproducible pipeline, noting details like masking percentages and standard BERT architectures. Attendees discuss the lack of dedicated code embedding models in the open source ecosystem.15:55–27:23 · Guest teaching 6/10 Jina Embeddings v3 and Task-Specific LoRA Adapters swyx discusses Jina Embeddings v3 and task-specific LoRA adapters. Eugene steps in to correct swyx's mental model regarding LoRA layer placement, explaining that LoRA weights modify all attention and MLP projection layers rather than just replacing the final classification layer.27:42–30:27 · Guest teaching 0/10 Multimodal Embeddings and Jina CLIP v2 Architecture swyx walks through the Jina CLIP v2 release, highlighting how freezing text embedding backbones and training vision adapters creates performant multimodal representations.30:28–35:09 · Guest teaching 3/10 Discussion on Biomedical and Domain-Specific Embeddings SPEAKER_03 inquires about medical domain embeddings and protein pathway retrieval. swyx and participants discuss the practicality of fine-tuning generic base models versus using niche domain foundations like MedSAM.35:10–41:43 · Guest teaching 2/10 Contextual Document Embeddings and Two-Stage Adaptation swyx explains Contextual Document Embeddings (CDE) and two-stage corpus conditioning, distinguishing it from traditional fine-tuning by clarifying that it avoids gradient updates and operates more like contextual KV caching.0:00–2:15 · Guest disagreement 0/10 Overview of 2024 Embeddings and the MTEB Benchmark swyx opens the session with a broad, confident walkthrough of the current state of text embeddings in 2024 and the Massive Text Embedding Benchmark (MTEB). He outlines trade-offs between model size, efficiency, and benchmark rankings without any guest intervention.2:16–9:33 · Guest disagreement 2/10 OpenAI Embeddings and Matryoshka Representation Learning swyx explains OpenAI's Matryoshka Representation Learning (MRL) dimension reduction. Eugene educates the room on production latency constraints, pointing out that 1024 dimensions are often unfeasible for real-time ANN lookups, while 64 or 128 dimensions provide viable latency.9:34–15:54 · Guest disagreement 1/10 Nomic Embed and Fully Reproducible Training Pipelines swyx highlights Nomic's fully reproducible pipeline, noting details like masking percentages and standard BERT architectures. Attendees discuss the lack of dedicated code embedding models in the open source ecosystem.15:55–27:23 · Guest disagreement 2/10 Jina Embeddings v3 and Task-Specific LoRA Adapters swyx discusses Jina Embeddings v3 and task-specific LoRA adapters. Eugene steps in to correct swyx's mental model regarding LoRA layer placement, explaining that LoRA weights modify all attention and MLP projection layers rather than just replacing the final classification layer.27:42–30:27 · Guest disagreement 0/10 Multimodal Embeddings and Jina CLIP v2 Architecture swyx walks through the Jina CLIP v2 release, highlighting how freezing text embedding backbones and training vision adapters creates performant multimodal representations.30:28–35:09 · Guest disagreement 1/10 Discussion on Biomedical and Domain-Specific Embeddings SPEAKER_03 inquires about medical domain embeddings and protein pathway retrieval. swyx and participants discuss the practicality of fine-tuning generic base models versus using niche domain foundations like MedSAM.35:10–41:43 · Guest disagreement 1/10 Contextual Document Embeddings and Two-Stage Adaptation swyx explains Contextual Document Embeddings (CDE) and two-stage corpus conditioning, distinguishing it from traditional fine-tuning by clarifying that it avoids gradient updates and operates more like contextual KV caching.0:00–2:15 · The hosts pushing back 0/10 Overview of 2024 Embeddings and the MTEB Benchmark swyx opens the session with a broad, confident walkthrough of the current state of text embeddings in 2024 and the Massive Text Embedding Benchmark (MTEB). He outlines trade-offs between model size, efficiency, and benchmark rankings without any guest intervention.2:16–9:33 · The hosts pushing back 1/10 OpenAI Embeddings and Matryoshka Representation Learning swyx explains OpenAI's Matryoshka Representation Learning (MRL) dimension reduction. Eugene educates the room on production latency constraints, pointing out that 1024 dimensions are often unfeasible for real-time ANN lookups, while 64 or 128 dimensions provide viable latency.9:34–15:54 · The hosts pushing back 0/10 Nomic Embed and Fully Reproducible Training Pipelines swyx highlights Nomic's fully reproducible pipeline, noting details like masking percentages and standard BERT architectures. Attendees discuss the lack of dedicated code embedding models in the open source ecosystem.15:55–27:23 · The hosts pushing back 1/10 Jina Embeddings v3 and Task-Specific LoRA Adapters swyx discusses Jina Embeddings v3 and task-specific LoRA adapters. Eugene steps in to correct swyx's mental model regarding LoRA layer placement, explaining that LoRA weights modify all attention and MLP projection layers rather than just replacing the final classification layer.27:42–30:27 · The hosts pushing back 0/10 Multimodal Embeddings and Jina CLIP v2 Architecture swyx walks through the Jina CLIP v2 release, highlighting how freezing text embedding backbones and training vision adapters creates performant multimodal representations.30:28–35:09 · The hosts pushing back 1/10 Discussion on Biomedical and Domain-Specific Embeddings SPEAKER_03 inquires about medical domain embeddings and protein pathway retrieval. swyx and participants discuss the practicality of fine-tuning generic base models versus using niche domain foundations like MedSAM.35:10–41:43 · The hosts pushing back 2/10 Contextual Document Embeddings and Two-Stage Adaptation swyx explains Contextual Document Embeddings (CDE) and two-stage corpus conditioning, distinguishing it from traditional fine-tuning by clarifying that it avoids gradient updates and operates more like contextual KV caching.

speaking balance: gold is the hosts, purple is the guest (3 minute bins)

0:00 · the hosts 0% · guest 100%0:00 · the hosts 0% · guest 100%3:00 · the hosts 0% · guest 100%3:00 · the hosts 0% · guest 100%6:00 · the hosts 0% · guest 100%6:00 · the hosts 0% · guest 100%9:00 · the hosts 0% · guest 100%9:00 · the hosts 0% · guest 100%12:00 · the hosts 0% · guest 100%12:00 · the hosts 0% · guest 100%15:00 · the hosts 0% · guest 100%15:00 · the hosts 0% · guest 100%18:00 · the hosts 0% · guest 100%18:00 · the hosts 0% · guest 100%21:00 · the hosts 0% · guest 100%21:00 · the hosts 0% · guest 100%24:00 · the hosts 0% · guest 100%24:00 · the hosts 0% · guest 100%27:00 · the hosts 0% · guest 100%27:00 · the hosts 0% · guest 100%30:00 · the hosts 0% · guest 100%30:00 · the hosts 0% · guest 100%33:00 · the hosts 0% · guest 100%33:00 · the hosts 0% · guest 100%36:00 · the hosts 0% · guest 100%36:00 · the hosts 0% · guest 100%39:00 · the hosts 0% · guest 100%39:00 · the hosts 0% · guest 100%
Sharpest disagreement ▶ 27:08 Debating LoRA definition and rank semantics

swyx questions whether multi-layer LoRA can truly be considered low-rank, prompting Eugene to directly defend the mathematical compression definition of LoRA dimensions.

 Hardest push from the hosts ▶ 41:25 Pushback on contextual embeddings being mere fine-tuning

swyx rejects an audience question asserting that two-stage corpus conditioning is just fine-tuning, emphasizing that no gradient updates occur.

 Biggest teaching moment ▶ 26:28 Eugene corrects swyx on transformer LoRA implementation

Eugene gently corrects swyx's assumption that LoRAs are merely applied to the final output layer, detailing their application across transformer MLP and QKV projection matrices.

 The host holds their own ▶ 6:00 swyx quantifies Matryoshka dimension vs accuracy trade-offs

swyx demonstrates deep technical familiarity by citing precise benchmark numbers showing a 94% storage reduction with only an 8% drop in top-1/top-5 retrieval accuracy.
the scores for every segment, with the reasoning behind each
ChapterTopicThe hosts as informed peerGuest teachingGuest disagreementThe hosts pushing backWhy
Overview of 2024 Embeddings and the MTEB Benchmark 6000 swyx opens the session with a broad, confident walkthrough of the current state of text embeddings in 2024 and the Massive Text Embedding Benchmark (MTEB). He outlines trade-offs between model size, efficiency, and benchmark rankings without any guest intervention.
OpenAI Embeddings and Matryoshka Representation Learning 5521 swyx explains OpenAI's Matryoshka Representation Learning (MRL) dimension reduction. Eugene educates the room on production latency constraints, pointing out that 1024 dimensions are often unfeasible for real-time ANN lookups, while 64 or 128 dimensions provide viable latency.
Nomic Embed and Fully Reproducible Training Pipelines 6210 swyx highlights Nomic's fully reproducible pipeline, noting details like masking percentages and standard BERT architectures. Attendees discuss the lack of dedicated code embedding models in the open source ecosystem.
Jina Embeddings v3 and Task-Specific LoRA Adapters 5621 swyx discusses Jina Embeddings v3 and task-specific LoRA adapters. Eugene steps in to correct swyx's mental model regarding LoRA layer placement, explaining that LoRA weights modify all attention and MLP projection layers rather than just replacing the final classification layer.
Multimodal Embeddings and Jina CLIP v2 Architecture 6000 swyx walks through the Jina CLIP v2 release, highlighting how freezing text embedding backbones and training vision adapters creates performant multimodal representations.
Discussion on Biomedical and Domain-Specific Embeddings 4311 SPEAKER_03 inquires about medical domain embeddings and protein pathway retrieval. swyx and participants discuss the practicality of fine-tuning generic base models versus using niche domain foundations like MedSAM.
Contextual Document Embeddings and Two-Stage Adaptation 7212 swyx explains Contextual Document Embeddings (CDE) and two-stage corpus conditioning, distinguishing it from traditional fine-tuning by clarifying that it avoids gradient updates and operates more like contextual KV caching.

Statements from this episode (0)

Nothing in this episode matches those filters. clear them

Made with StarZero

Turn any episode into a week of clips.

This entire site, over 200 episodes transcribed, diarized, checked and made playable, runs on the StarZero media pipeline. Drop in your own episode and the podcast clipper finds the moments worth sharing, cuts them, captions them, and reframes them for every feed.

Try the podcast clipper Learn more