StarZero All shows Try the podcast clipper →

Dec 21, 2023 · 22m · mad

How Moody’s Analytics Is Using AI to Transform Credit Risk | Cristina Pieretti & Yimei Fan

Cristina Pieretti · 12m spoken Yimei Fan · 5m spoken Matt Turck · 3m spoken
0:00 / 0:00
▶ Watch on YouTube →

gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions

Host Matt Turck moderates a Data Driven NYC panel featuring Moody's Analytics executives Cristina Pieretti and Yimei Fan discussing how traditional financial enterprises leverage generative AI, RAG architecture, and internal copilots to transform credit risk analysis and workforce productivity.

How this conversation actually went

Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 16% of the talking time here. How this is scored →

Matt as informed peer 2.6 Guest teaching 3.0 Guest disagreement 0.2 Matt pushing back 0.6
05100:0010:0020:000:08–4:09 · Matt as informed peer 2/10 Welcome and Panelist Introductions Matt opens the panel by establishing context on enterprise adoption of AI, asking the guests to introduce Moody's Analytics. Cristina explains the origin of Moody's Analytics and her history meeting Matt in 2017.4:09–6:26 · Matt as informed peer 1/10 Evolution of Machine Learning at Moody's Matt asks about Moody's legacy with machine learning. Yimei and Cristina detail the transition from quantitative risk models and tree-based algorithms to automated feature engineering with transformers.6:26–11:06 · Matt as informed peer 4/10 Adoption of Generative AI and Research Assistant Launch Cristina passionately describes their new GenAI platform and RAG architecture. Matt politely interjects to steer her to explain the actual domain use case, noting that credit risk analysis involves high stakes where hallucinations are unacceptable.11:06–15:15 · Matt as informed peer 4/10 Technical Architecture, LLM Evaluation, and RAG Mechanics Matt drills into the technical stack, asking how they evaluated GPT-4 against alternative LLMs and how their custom RAG operates. Cristina and Yimei walk through intent identification and data retrieval mechanics.15:15–18:52 · Matt as informed peer 2/10 Internal Enablement and Moody's Copilot Impact Matt asks about internal employee adoption and external ecosystem partnerships. Yimei clarifies that 'Moody's Copilot' is an internally developed proprietary platform rather than Microsoft or GitHub Copilot.0:08–4:09 · Guest teaching 2/10 Welcome and Panelist Introductions Matt opens the panel by establishing context on enterprise adoption of AI, asking the guests to introduce Moody's Analytics. Cristina explains the origin of Moody's Analytics and her history meeting Matt in 2017.4:09–6:26 · Guest teaching 3/10 Evolution of Machine Learning at Moody's Matt asks about Moody's legacy with machine learning. Yimei and Cristina detail the transition from quantitative risk models and tree-based algorithms to automated feature engineering with transformers.6:26–11:06 · Guest teaching 3/10 Adoption of Generative AI and Research Assistant Launch Cristina passionately describes their new GenAI platform and RAG architecture. Matt politely interjects to steer her to explain the actual domain use case, noting that credit risk analysis involves high stakes where hallucinations are unacceptable.11:06–15:15 · Guest teaching 4/10 Technical Architecture, LLM Evaluation, and RAG Mechanics Matt drills into the technical stack, asking how they evaluated GPT-4 against alternative LLMs and how their custom RAG operates. Cristina and Yimei walk through intent identification and data retrieval mechanics.15:15–18:52 · Guest teaching 3/10 Internal Enablement and Moody's Copilot Impact Matt asks about internal employee adoption and external ecosystem partnerships. Yimei clarifies that 'Moody's Copilot' is an internally developed proprietary platform rather than Microsoft or GitHub Copilot.0:08–4:09 · Guest disagreement 0/10 Welcome and Panelist Introductions Matt opens the panel by establishing context on enterprise adoption of AI, asking the guests to introduce Moody's Analytics. Cristina explains the origin of Moody's Analytics and her history meeting Matt in 2017.4:09–6:26 · Guest disagreement 0/10 Evolution of Machine Learning at Moody's Matt asks about Moody's legacy with machine learning. Yimei and Cristina detail the transition from quantitative risk models and tree-based algorithms to automated feature engineering with transformers.6:26–11:06 · Guest disagreement 0/10 Adoption of Generative AI and Research Assistant Launch Cristina passionately describes their new GenAI platform and RAG architecture. Matt politely interjects to steer her to explain the actual domain use case, noting that credit risk analysis involves high stakes where hallucinations are unacceptable.11:06–15:15 · Guest disagreement 0/10 Technical Architecture, LLM Evaluation, and RAG Mechanics Matt drills into the technical stack, asking how they evaluated GPT-4 against alternative LLMs and how their custom RAG operates. Cristina and Yimei walk through intent identification and data retrieval mechanics.15:15–18:52 · Guest disagreement 1/10 Internal Enablement and Moody's Copilot Impact Matt asks about internal employee adoption and external ecosystem partnerships. Yimei clarifies that 'Moody's Copilot' is an internally developed proprietary platform rather than Microsoft or GitHub Copilot.0:08–4:09 · Matt pushing back 0/10 Welcome and Panelist Introductions Matt opens the panel by establishing context on enterprise adoption of AI, asking the guests to introduce Moody's Analytics. Cristina explains the origin of Moody's Analytics and her history meeting Matt in 2017.4:09–6:26 · Matt pushing back 0/10 Evolution of Machine Learning at Moody's Matt asks about Moody's legacy with machine learning. Yimei and Cristina detail the transition from quantitative risk models and tree-based algorithms to automated feature engineering with transformers.6:26–11:06 · Matt pushing back 2/10 Adoption of Generative AI and Research Assistant Launch Cristina passionately describes their new GenAI platform and RAG architecture. Matt politely interjects to steer her to explain the actual domain use case, noting that credit risk analysis involves high stakes where hallucinations are unacceptable.11:06–15:15 · Matt pushing back 1/10 Technical Architecture, LLM Evaluation, and RAG Mechanics Matt drills into the technical stack, asking how they evaluated GPT-4 against alternative LLMs and how their custom RAG operates. Cristina and Yimei walk through intent identification and data retrieval mechanics.15:15–18:52 · Matt pushing back 0/10 Internal Enablement and Moody's Copilot Impact Matt asks about internal employee adoption and external ecosystem partnerships. Yimei clarifies that 'Moody's Copilot' is an internally developed proprietary platform rather than Microsoft or GitHub Copilot.

speaking balance: gold is Matt, purple is the guest (3 minute bins)

0:00 · Matt 42.4% · guest 57.6%0:00 · Matt 42.4% · guest 57.6%3:00 · Matt 8.2% · guest 91.8%3:00 · Matt 8.2% · guest 91.8%6:00 · Matt 6.5% · guest 93.5%6:00 · Matt 6.5% · guest 93.5%9:00 · Matt 22.9% · guest 77.1%9:00 · Matt 22.9% · guest 77.1%12:00 · Matt 9.2% · guest 90.8%12:00 · Matt 9.2% · guest 90.8%15:00 · Matt 18.3% · guest 81.7%15:00 · Matt 18.3% · guest 81.7%18:00 · Matt 7.6% · guest 92.4%18:00 · Matt 7.6% · guest 92.4%21:00 · Matt 9.3% · guest 90.7%21:00 · Matt 9.3% · guest 90.7%
Sharpest disagreement ▶ 15:21 Distinguishing Moody's Copilot from commercial tools

Yimei explicitly reframes the terminology to ensure the audience does not confuse their custom internal platform with third-party products like GitHub or Microsoft Copilot.

 Hardest push from Matt ▶ 9:30 Matt interjects to clarify product domain

Matt cuts in while Cristina is explaining RAG mechanics to force a clear definition of what the underlying product actually does for users.

 Biggest teaching moment ▶ 5:30 Yimei details transition from handcrafted features to transformer models

Yimei provides an technical breakdown of how data science shifted from manual feature engineering to automated transformer extraction, demonstrating why Moody's was prepared for GenAI.

 Matt holds his own ▶ 10:22 Matt highlights the zero-tolerance stance on hallucinations in credit risk

Matt demonstrates sharp domain awareness by explaining that credit risk decisions involve billions of dollars, making LLM accuracy and containment far more critical than in consumer AI apps.
the scores for every segment, with the reasoning behind each
ChapterTopicMatt as informed peerGuest teachingGuest disagreementMatt pushing backWhy
Welcome and Panelist Introductions 2200 Matt opens the panel by establishing context on enterprise adoption of AI, asking the guests to introduce Moody's Analytics. Cristina explains the origin of Moody's Analytics and her history meeting Matt in 2017.
Evolution of Machine Learning at Moody's 1300 Matt asks about Moody's legacy with machine learning. Yimei and Cristina detail the transition from quantitative risk models and tree-based algorithms to automated feature engineering with transformers.
Adoption of Generative AI and Research Assistant Launch 4302 Cristina passionately describes their new GenAI platform and RAG architecture. Matt politely interjects to steer her to explain the actual domain use case, noting that credit risk analysis involves high stakes where hallucinations are unacceptable.
Technical Architecture, LLM Evaluation, and RAG Mechanics 4401 Matt drills into the technical stack, asking how they evaluated GPT-4 against alternative LLMs and how their custom RAG operates. Cristina and Yimei walk through intent identification and data retrieval mechanics.
Internal Enablement and Moody's Copilot Impact 2310 Matt asks about internal employee adoption and external ecosystem partnerships. Yimei clarifies that 'Moody's Copilot' is an internally developed proprietary platform rather than Microsoft or GitHub Copilot.

Statements from this episode (13)

Disclosure
Moody's Analytics Launches Generative AI Tool 'Research Assistant'
“One of the ones I'm more particularly excited, because we just launched it on Friday, is, ah, one, ah, leveraging Gen AI, which is called Research Assistant.”
Cristina Pieretti Dec 21, 2023 ▶ 5:18
Disclosure
Moody's automated feature engineering using transformer models on text data
“Then we actually automated the feature engineering process, for example, using transformer models to extract features from massive, massive text data.”
Yimei Fan Dec 21, 2023 ▶ 6:03
Disclosure
Moody's Analytics Deploys Internal AI Copilot to All 14,000 Employees
“We did roll out Our co-pilot to all the employees in Moody, so that's 14,000 people”
Cristina Pieretti Dec 21, 2023 ▶ 7:13
Disclosure
Moody's Research Assistant Uses GPT-4 Alongside Custom RAG System
“We use ChatGPT-IV as the LLM, although we've tried a lot of, we talked about in our prep, we have tried different, ah, large language models, and we've also developed our own RAG.”
Cristina Pieretti Dec 21, 2023 ▶ 8:14
Disclosure
Moody's Research Assistant currently only uses rating agency data
“So as of now, it's leveraging only the information that comes from the rating agency. So it's all the ratings and the research that come from the rating agency. In the future, we plan to expand.”
Cristina Pieretti Dec 21, 2023 ▶ 10:08
Disclosure
Moody's designs AI architecture to avoid vendor lock-in
“I'm kind of obsessed, and I think my team is obsessed on making sure that we're not dependent on one technology, but we have the flexibility to switch as time goes by, right?”
Cristina Pieretti Dec 21, 2023 ▶ 11:41
Assertion Not checkable as stated
Moody's Found OpenAI Outperformed Competing Models in Internal Benchmarks
“ChatGPT was the one, OpenAI was the one that was performing better in a, in over Different parameters, right?”
Cristina Pieretti Dec 21, 2023 ▶ 11:55
Disclosure
Moody's Spent Most of 7-Month AI Dev Cycle Evaluating Model Accuracy
“We first started working this seven months ago. I would say a big, big chunk of the time we spend it evaluating that as opposed to building more features, right?”
Cristina Pieretti Dec 21, 2023 ▶ 13:40
Assertion Not checkable as stated
Fan: Moody's Copilot has over 10,000 active users
“We've seen now actively, I would say more than 10,000 users are using this”
Yimei Fan Dec 21, 2023 ▶ 16:38
Assertion Not checkable as stated
Moody's Engineers Saw 40% Efficiency Gains Using GitHub Copilot
“We measured that there's 40%, ah, performance efficiency improvements for our engineers.”
Yimei Fan Dec 21, 2023 ▶ 16:47
Disclosure
Moody's Analytics partnered with Google to extract data from unstructured financial statements
“We have another partnership with Google where we're, ah, leveraging, we're working together to leverage the NAI to, ah, be able to extract more information from financial statements. Ah, financial statements, not the ones that are, you know, published at the 1…”
Cristina Pieretti Dec 21, 2023 ▶ 17:49
Assertion Not checkable as stated
Moody's Observes Up to 80% Efficiency Gains in Data Retrieval
“And what they do is, first, they have to retrieve a lot of information, and we've observed up to 80% efficiency gains on that retrieval of the information. From there, once you retrieve the information, you're gonna do analysis, and we've observed up to 50% on…”
Cristina Pieretti Dec 21, 2023 ▶ 19:26
Assertion Not checkable as stated
Moody's units can deploy AI apps within a day via Copilot architecture
“One example actually, so Moody's Copilot already has some core components, so, and the infrastructure and architecture's already set up, so if Moody's other operation unit wants to build their own application, they can come, use the existing ones, and they can…”
Yimei Fan Dec 21, 2023 ▶ 21:26
Made with StarZero

Turn any episode into a week of clips.

This entire site, over 400 conversations transcribed, diarized, checked and made playable, runs on the StarZero media pipeline. Drop in your own episode and the podcast clipper finds the moments worth sharing, cuts them, captions them, and reframes them for every feed.

Try the podcast clipper Learn more