Jun 15, 2023 · 1h 20m · catalyst
AI for climate: a real world test
Duncan Campbell · 27m spoken Seyed Madaeni · 21m spoken Shayle Kann · 20m spoken Daniel Waldorf · 15s spoken
0:00 / 0:00
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Host Shayle Kann and energy trading expert Seyed Madaeni put generative AI to the test by tasking non-programmer Duncan Campbell with building a wholesale battery storage dispatch optimization algorithm using ChatGPT. The experiment demonstrates how iterative prompt engineering and human domain oversight can compress weeks of specialized quantitative development into hours.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Shayle holds 28.9% of the talking time here. How this is scored →
Shayle as informed peer 5.1 Guest teaching 4.2 Guest disagreement 1.0 Shayle pushing back 1.9
speaking balance: gold is Shayle, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 33:30 Seyed rejects relying on simplistic price forecasts
Seyed firmly pushes back against relying on basic regression forecasts by emphasizing that poor price forecasts produce garbage in, garbage out results in trading optimization.
Hardest push from Shayle ▶ 20:22 Shayle questions whether software engineering is turning into essay writingShayle directly challenges the conventional boundary of coding by asking Seyed whether software engineers will simply be relegated to writing prose prompts.
Biggest teaching moment ▶ 43:05 Seyed breaks down the 90 percent complexity reductionSeyed provides a masterclass on wholesale battery commercialization, explaining how excluding real-time market bidding and ancillary services removed 90 percent of the engineering complexity.
Shayle holds their own ▶ 32:55 Shayle details multivariate regression requirements for LMP forecastingShayle demonstrates deep quantitative understanding by outlining the exact input parameters and regression mechanics required to forecast day-ahead wholesale electricity prices.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Shayle as informed peer | Guest teaching | Guest disagreement | Shayle pushing back | Why |
|---|---|---|---|---|---|---|
| Mid-Roll Sponsorship: Bloom Energy and Engie | 0 | 0 | 0 | 0 | Sponsor advertisements followed by the host introducing the premise of the episode as an experimental test of large language models on battery dispatch optimization. | |
| Defining the CAISO Battery Bidding Challenge | 6 | 5 | 1 | 2 | Seyed defines the technical constraints of CAISO wholesale bidding while Shayle demonstrates familiarity with acceptance criteria, market parameters, and battery cycling limits. | |
| Traditional Software Workflows vs. Non-Coder Abilities | 5 | 4 | 1 | 2 | The panel discusses the software engineering timeline for building dispatch algorithms, with Duncan acknowledging his non-coding background and reliance on Excel. | |
| The First Attempt: The 'Buckshot' Prompting Failure | 5 | 4 | 1 | 2 | Duncan describes the failure of pasting the prompt whole-cloth into ChatGPT, concluding that LLMs require bottom-up structural guidance rather than top-down delegation. | |
| The Second Attempt: Step-by-Step Prompting Strategy | 5 | 5 | 1 | 3 | Shayle probes whether prompt engineering is turning coding into essay writing, while Seyed emphasizes software architecture and human quality assurance. | |
| Debugging Physics and Mathematical Constraints | 5 | 5 | 2 | 2 | Duncan explains how physical boundary conditions like simultaneous charge and discharge and daily cycling constraints required explicit human intervention to resolve bad math outputs. | |
| Scaling Across Multiple Days and Skipping Price Forecasting | 6 | 5 | 3 | 3 | Shayle articulates how multivariate regression models would need to be specified for LMP forecasting, while Seyed notes that flawed forecasts create garbage in, garbage out optimization. | |
| Formatting Data Outputs and the Human-AI Partnership | 4 | 3 | 0 | 1 | Duncan shares how wrestling with visualization libraries felt like working with a junior analyst, highlighting the collaborative feeling of prompting. | |
| Mid-Roll Sponsorship: Bloom Energy and Engie | 6 | 6 | 2 | 2 | Following the mid-roll ad break, Seyed clarifies that by omitting ancillary services and market bidding, Duncan reduced the challenge's overall complexity by roughly ninety percent. | |
| Code Review and Section-by-Section Grading | 5 | 7 | 1 | 2 | Seyed performs a code review, praising the clean modular structure and linear optimization formulation while noting technical gaps like missing integer programming decision variables. | |
| Quantifying Productivity Gains and Economic Value | 6 | 4 | 1 | 3 | The panel compares the four hours spent against traditional engineering cycles, with Seyed estimating that producing equivalent groundwork would take two engineers several weeks. | |
| Production-Grade Reliability vs. Analytical Pre-Construction Use | 5 | 4 | 1 | 2 | The discussion distinguishes between real-time production-grade software and exploratory pre-construction modeling, identifying the script as valuable for analytical workflows. | |
| Live Model Execution and Dispatch Behavior Analysis | 6 | 3 | 0 | 1 | Duncan executes the live notebook, showing the resulting dispatch schedule and price curve charts while the panel observes how the model optimized state of charge across multiple peaks. | |
| Macro Implications of Generative AI in the Energy Sector | 6 | 4 | 1 | 2 | The speakers reflect on broader industry implications, agreeing that generative AI serves as a powerful labor multiplier for systems thinkers rather than an immediate replacement for coders. | |
| The Holy Grail AI Use Case: Parsing Utility Tariffs and Regulations | 6 | 4 | 0 | 1 | Duncan proposes using LLMs to ingest and standardize complex utility tariff structures, a use case that Shayle and Seyed enthusiastically endorse before concluding. |
Statements from this episode (20)
Kann is skeptical LLMs will fundamentally change climate tech and energy
“I have generally been a bit more skeptical than most, I think, that it would change, fundamentally change at least, the things that I spend my professional time on, which is to say climate tech and energy.”
Madaeni: Energy storage market bidding is far harder than thermal generation
“And traditionally bidding in conventional assets like thermal plants has been relatively straightforward, but energy storage creates a whole host of challenges for market bidding, because at the end of the day, energy storage is a use limited asset.”
Madaeni: Building automated energy trading software is a year-long endeavor
“You can come up with something over the course of 12 hours, but trust me, it's going to leave a lot of money and value on the table, or you can actually systematically procure software or build software Which is a, in my view, a year endeavor.”
Campbell: Building complex software with ChatGPT requires iterative prompting
“It's very hard to kind of, like, top-down build it, because there's a bunch of little errors everywhere. It's, like, hard to track what's going on. And I pretty quickly reverted to, like, bottom-up building it. Like, building one element, like, slowly working …”
Campbell: ChatGPT operates like a fast junior engineer needing domain guidance
“ChatGTP is not like a ten-year industry veteran with deep domain expertise, right? It's a decent sort of like Junior software engineer, and its benefit is that it's very fast, right? But you can't just sort of like let it take the reins and run with the proble…”
Madaeni: AI will shift software engineering toward prompting, code review, and QA
“At the end of the day, we're flying a plane that's, we're souping up the autopilot system, but I think we still need pilots. And, you know, a pilot with an autopilot system is a different kind of a pilot. So it's mostly about, I wouldn't say essay writing, but…”
Madaeni: Non-domain engineers miss battery physical constraints just like LLMs do
“Batteries can't charge and discharge at the same time, and it's rather obvious to us, but the same applies if we bring a software engineer out of, as an example, out of Uber, where they've never heard about energy problems, can't distinguish between megawatts …”
Campbell: Blindly prompting LLMs to debug code leads to nonsensical rabbit holes
“In debugging, if you kind of just, like, keep prompting it, like, fix this, here's the error, like, here's the message I'm getting back in error, and you just kind of, like, assume it's gonna be smart. You can wind up in, like, rabbit holes that just don't mak…”
Madaeni: Full-year vs daily analysis makes no difference for 4-hour battery dispatch
“If the battery had enough, you know, energy to kind of cycle for a couple of days, if we're thinking about long duration storage distinguishing between a full year analysis versus a daily analysis is important, but if your battery has four hours, then it reall…”
Madaeni: LLMs are not equipped to build and train neural nets for forecasting
“But it's clear that I don't think that LLMs are equipped with, you know, building a whole neural net from scratch and training them and do some form of supervised, unsupervised learning for Forecasting.”
Campbell: Generating Plotly charts with ChatGPT took longer than the optimizer
“Just the fact that the optimizer worked is one thing, but like actually parsing all of that, that schedule you've created and present creating outputs that are useful and charts is its own whole like finicky journey. So then I started digging into all of that …”
Madaeni: Building a production battery dispatch optimizer takes a team one year
“If I think about the initial problem, which takes a year for a souped up team to build and productize, I originally reduced the complexity to 50% by ignoring all of the bidding stuff. And from the 50% that was remaining the insular services are probably like 4…”
Madaeni: Engineers with both backend software and optimization expertise are extremely rare
“Software engineers do a lot of the back-end data engineering, post-processing of results, putting it in the architecture. That's why you go and hire optimization engineers, people with OR backgrounds and math models that can write Optimization level software. …”
Campbell: Built battery dispatch optimization script in four hours using ChatGPT
“If I had to guess, I'd say all this to like, from prompting to where we are right now, probably like four hours of continuous work, something like that.”
Madaeni: Equivalent battery optimization code manually takes two engineers one month
“Once you want to like grab, put some software foundation, what Duncan has presented today, I would say you need two engineers, more like a backend software engineer and an optimization engineer. And depending on how well they get along with each other, because…”
Madaeni: AI battery dispatch code is ready for analysts, miles from production
“There's a lot to make a software code production great, and this is far from reality. So I would say, like, we are miles away from using this in production, but if you think about an analyst just trying to make You know, informative decisions as, you know, sou…”
Campbell: ChatGPT's 'permaflow' eliminates human context-switching penalties
“Every time you go back to you know, chat.openai.com and go into this chat, It has a perfect state of where it was previously, right? So I said four or five hours of continuous work, but in reality, it was a bunch of 45 minute segments. And I didn't have to lik…”
Campbell: LLMs will empower Excel and systems thinkers across the energy industry
“Energy industry is full of people who are like great at Excel and really good at systems thinking, but are not programmers. So I think you could see a lot of people empowered by this.”
Campbell: AI will not replace software engineers but make individuals team-scale
“I don't see this replacing software engineers. Just as it empowered me, like I very much see a great software engineer just using this to save a ton of time, right? Telling it exactly what to do with a level of specificity about programming that I could never …”
Campbell: Parsing utility tariffs is a holy grail LLM energy use case
“A persistent issue in the electricity industry for as long as I've been doing this job is utility tariffs, right? Every utility has some 700 page document that outlines massive The complex nuances of how users will pay for electricity. And we're talking about …”