The Exchanges

Q If you go down that next level, if you're on the financing need, good momentum, what might a next question be to determine in that subset which companies are likely to outperform?

A Typical questions would be about volatility. We tend to find momentum works better when it is consistent. When the stock price is rising in a consistent manner, it leads to better outcomes than companies that have one giant price move driving the momentum measurement. Company age also comes into account there. We find that momentum typically is more meaningful when you're looking at newer companies than companies have been around for a long time. They're generally higher growth businesses. They are more often in industries that are evolving. Knowing that the sentiment is strong around those companies is an even more positive Indicator of future returns than knowing that a company that's been around for a hundred years had a good quarter.

AI assessment note: “Typical questions would be about volatility. We tend to find momentum works better when it is consistent.”

Q When you've been working with machine learning models for a long time, what does the introduction of Chad GPT change, if anything, in the way you've approached what you've done?

A Large language models and ChatGPT specifically are not anything that we're presently making use of in our modeling. One of the big challenges for folks who are trying to use those types of models in a stock picking context is the problem of in sample versus out of sample. Especially if you're using a commercial model, you don't have any control over What data that model was trained on. When you're running a back test through the better part of the last decade, ChatGPT knows that Nvidia became a multi-trillion dollar company. ChatGPT knows what the mega trends were in the economy and the market over those timeframes. It's not realistic to trust a back test that ChatGPT generated. That said, there are exciting things going on in the AI space, and we use a lot of proprietary software and tools in our investment process. One area in AI that is really appealing to us is the idea of software development co-pilots. The idea that AI can make and enhance software development at an organizational level, We're a small team with a lot of software, and any ways in which we can improve efficiencies there are valuable to us.

AI assessment note: “ChatGPT specifically are not anything that we're presently making use of in our modeling.”

Q And in one individual tree, to get from the top to the bottom, how many different decision points and nodes are there?

A Typically we ask between two and five questions in each tree. The reason we don't ask more questions is we found that as you ask questions deeper and deeper in the tree, you're working on smaller and smaller pools of data because the trees are customized to the branch of the tree that you're working down. If you think about trees breaking up fifty-fifty at the second layer of the tree, each question is motivated on half of your original data. Down another layer, it's a quarter. Down 10 layers, each question is going to be motivated on one 1000th of the data. Down 20 layers, you would be operating on one one millionth of the data. You can quickly see that there's a sharp limit to how deep you want to make these trees. Fortunately, we have another approach to asking more questions about companies, which is rather than relying on a very deep tree, Relying on a forest of relatively shallow trees.

AI assessment note: “Typically we ask between two and five questions in each tree.”

Q Which two people, other than your wife, have had the biggest impact on your professional life?

A I've worked at MDT my entire career, and I was really fortunate to have two mentors from day one, David Goldsmith and Sarah Stahl. David was the founder of the Quant Group and the CIO. Sarah was one of David's first hires who led analytical and portfolio attribution effort here for many years. What was great about the two of them was that they were incredibly different from one another in terms of mentors. David was the mad scientist of our group. He would be thinking about algorithms, twenty-four-seven, come in and tell us about the idea he had while he was in the shower. Sarah was also very brilliant in a less wild and unconstrained way. She was very meticulous, very focused on craftsmanship and Understanding precisely what was driving the returns of our models. They were both great mentors and helped me appreciate that success in investment management. It's not all about being the brightest and having the most genius ideas. There are a lot of geniuses who failed. It's not just about meticulousness and craftsmanship, but both of those things are very important. It's a success in this business. I'm really indebted to David and Sarah.

AI assessment note: “I was really fortunate to have two mentors from day one, David Goldsmith and Sarah Stahl.”

Q How much of those model inputs come from your own qualitative insights of what should matter?

A The selection of factors is driven by the potential questions that can be asked, is driven by the investment team. That's a major area of focus for us on the research side. Once we present that list of factors to the algorithm, it's Completely mechanically determined. A lot of times we'll have an idea about a factor as a new idea, help the model improve its forecasting, and the decision trees will simply say, nice try, guys, but I don't find a lot of profitable questions to ask about this factor. I'm going to ignore it. In terms of how does it decide to use the factors in relation to all the other characteristics, that's 100% driven by the algorithm.

AI assessment note: “The selection of factors is driven by the investment team... Once we present that list”

Q model, I'm envisioning peering into the glass box and seeing a huge piece of paper on the wall with all these decision trees and different questions and nodes that gets down to a signal somewhere. What does that ultimately look like in the sense of how many theses are at the top and work their way down into different nodes if you could actually visualize what's inside this glass box?

A We started our decision tree journey with OneTree. Over the years, with faster processing power and more advanced algorithms, we've been able to improve the forecasting by relying on algorithms that employ a forest of trees. Back in the OneTree day, we would print out the tree and tape it on the wall of our trading room. Every time we were reviewing a trade, we would simply walk through the sequence of questions On that paper tree on the wall to help inform what specifically was motivating every trade that happened in our portfolio. As you move to a forest of trees, we can't put a thousand paper trees on the wall anymore. We've built some tools, some analytical helpers to synthesize and summarize what's happening across the thousand trees. At the end of the day, you could go through that exercise, it would be tedious. Walking through tree by tree, whether anything has changed, specifically what, and dig in on the data updates that are driving every decision that happens in the portfolio.

AI assessment note: “As you move to a forest of trees, we can't put a thousand paper trees”

Q I'd love to tease through how you go about the investment process, and we'll just go top to bottom. As you're building a model to try to understand what stocks are likely to outperform, how do you come up with the ideas that you want to test quantitatively to see if it makes it into your model?

A There are two big sources of research ideas for our process. Certainly we read all of the academic and practitioner literature in the investment finance space, and occasionally we get some good ideas out of seeing what's published. More often than not, we test an idea and either it's not replicable when we look at it with our data set or something else in our model essentially captures the same underlying effect. Where we find More value typically is when we generate ideas that are driven by our own observations on the behavior of our strategies. That's one of the advantages of having the long history. We've been investing our strategies over 30 years now. The observations that we've made across multiple different market cycles over those decades have informed meaningful enhancements to the process.

AI assessment note: “There are two big sources of research ideas for our process.”

Q And what is it about the analytical approach that leads you to believe that?

A When you think about how do you construct a portfolio that is going to be able to perform well in lots of different market environments, there's two approaches to doing that. One is to be able to predict what the market environment is going to be with a fair degree of accuracy and then tilt your portfolio ahead of time to be in the right stocks, the right sectors at all times. Global macro crystal ball approach. There are investors who do that. It's not in our wheelhouse as quants. The other approach is very much on the other end of the spectrum of leaning on diversification, of not having reliance on any one company, any one sector, any one type of stock to be able to drive your portfolio outcome and to diversify across companies with differentiated alpha drivers Gives you that opportunity to have a portfolio where you'll have a fighting chance at performing well, no matter what market environment comes.

AI assessment note: “diversify across companies with differentiated alpha drivers Gives you that opportunity”

Q In the process of going from book to price being an important factor to not being in the model, what's the process to toggle on and off compared to decreasing its importance into the construction of the model?

A It's very data-driven. The process of removing a factor from the model, it's just the inverse of the process of adding a factor to the model. When we have a sense that a factor is working less well, generally that sense comes from the fact that we don't observe decision making being driven by that factor on a day-to-day basis. When we review our trades every morning, Year after year, we see fewer and fewer trades that are being driven by this one factor. That's the value of the glass box of being able to understand what's driving the decision making. When we have that intuition that a factor has decreased in efficiency, we'll run that research project and say, well, the model seems to be making less use of this over time. What if we made zero use of it? What if we removed it from what we present to the algorithm? How does that impact our research results? How does it impact the returns and the risk that we generate from our back test? And if we see that we can remove a factor from the model and have very little or no impact on portfolio outcomes over the course of decades, that gives us confidence that that's a factor that no longer needs to be there.

AI assessment note: “The process of removing a factor from the model, it's just the inverse”

Q In several of the things you've mentioned along the way, there is human judgment that's coming into play, whether that is the risk constraints in the model or news coming out about a company and say, well, that's not what the model's trying to signal. How do you think about the degree to which your human judgment should override anything that comes out of the model?

A We take a data oriented view on that. We try to put all of the potential overrides that we might make to the decision making of the model as much as possible through the lens of data. When we're thinking about trades, we're thinking about specifically what data inputs lead into what factors that are driving the decision making. When a company that we're trading has reported great earnings, We want to dig into, okay, well, how are those great earnings going to impact all of the factors in our model? At the next level, how will those factors changing impact the decision making that comes out of the trees? It's often the case that we're trading something and they've just reported great earnings, but we are buying them for reasons that have nothing to do with Analysts forecast. Whether the analysts raise their forecasts a ton or whether they make modest updates can be irrelevant for certain of the trading that we're doing in our portfolios. That's the value of the glass box is being able to see how the decision making is being made allows us to be precise in terms of how we think about potentially stepping in and overriding the model.

AI assessment note: “We try to put all of the potential overrides that we might make... through the lens of data.”

Q Would you walk me through what that means, a decision tree approach applied to stocks?

A So a decision tree, these are things that people have probably seen. It's just a series of yes and no questions about characteristics that lead to a forecast or an outcome. A common place that they're used is in an insurance setting. In the life insurance industry, you may want to build a model to predict longevity. Decision trees are often used in that space The first question might be on age. Depending on how you answer that question, whether you're above or below a certain age, there will be differentiated questions that are asked to help provide the most precise decision making as possible. For folks who are above the age of 65, the risk factors tend to be different than for people who are under the age of 65. If you are a smoker, There will be questions about how much do you smoke? How long have you been smoking? That won't be relevant of people who don't have that characteristic. Translating that back to the stock world, instead of asking about risk factors for longevity, we're asking about the characteristics of companies, and depending on how those questions are answered, the lines of questioning will evolve based on what's relevant of those types of companies.

AI assessment note: “Translating that back to the stock world, instead of asking about risk factors for longevity”

Q What are some of those important pitfalls that you've learned along the way?

A There are a pair of very related issues in the data science space, which are overfitting and underfitting. Obviously, they're two sides of the same coin. It's easy to not build a model that's overfit just by having a very simple model, but that very simple model is going to leave a lot of explanatory power on the table. It's going to be underfit. That's a problem that gets less press than overfitting, but is a significant one nonetheless. Figuring out what techniques can allow us to strike the right balance between having a model that's too complex Versus having a model that's not complex enough is something that we have put a lot of thought into and evolved significantly over the decades. Our view in the machine learning space is that transparency is exceedingly important to understand precisely how these models are working. A common epithet that gets thrown at us in the quant investment management space is that we're using black boxes. At MDT, that is not the case. We like to position our investment strategies as being a glass box. There's a lot of machinery on the inside, but we can see into it. We can see how it's working and understand what's driving all of the decision making on a day-to-day basis.

AI assessment note: “There are a pair of very related issues in the data science space, which are overfitting and underfitting.”

Q How do you go through the process of retesting a factor that's working? To see if the market catches up or it no longer works.

A We do occasionally remove factors from our modeling. The reasons that you do it are that first one where the factor no longer works for one reason or another, whether you were mistaken or whether markets have evolved. Occasionally we'll remove a factor if we add something new that captures a correlated underlying effect. An example of that first factor, we used book to price in our models. Going back to version one point O in 1991. But as markets evolved and more importantly, as the economy has changed, we saw less and less explanatory power to incorporating that in our model. And we had an intuitive sense of why that factor seemed to explain returns in data through the 19 seventies or eighties, but maybe doesn't work. Given how the intangible economy that's arisen in the decades since changes how companies trade on their book values.

AI assessment note: “we saw less and less explanatory power to incorporating that in our model.”

Q How do you think about the reflexivity of other quantitative participants in different models as it impacts what you're doing?

A When we think about that reflexivity in the quant space, There's a tendency to conflate natural fluctuations, good performance, bad performance, with quant strategies, which can be true of any investment strategy, with the impacts from running a strategy with leverage. When people talk about the most famous quant blowups of all time, long-term capital management, the quant quake in August of 2007, what they're highlighting are events that were caused by a period of underperformance for a quant strategy, but were magnified by the use of leverage in those strategies. If long-term capital management hadn't been running a fifty-x leveraged strategy, They wouldn't have ended up in the trouble that they ended up with. Similarly, in the quant quake, the paper that was published on that was written by Andrew Lowe, who's famous from MIT, and a gentleman by the name of Amir Kandani, who, you know, the world is small anecdote, happened to be my roommate at the mobile internet startup that I worked at in college. But the run on quant strategies that happened It was predicated by the fact that statistical arbitrage strategies had gotten more crowded over the years leading up to 2007. In response to that, certain managers began running those strategies with additional leverage, and leverage doesn't just blow up quant strategies. It's equal opportunity. No one panics out of traditional portfo…

AI assessment note: “When we think about that reflexivity in the quant space, There's a tendency to conflate”