The Exchanges

Q Was it as a consulting kind of way almost?

A Exactly. Exactly. We worked for companies like Airbus, um, many federal authorities in Europe, um, software providers, uh, in Germany. Building all kinds of systems, all time and material, as you say, um, it was consulting. We went in, we sculpt the problem. We, you know, looked into what do we find in open source communities? Which tools are around? What can we use? How can we assemble and build a solution for them? And then really we started building that solution. So it was great learning experience. And at the same time, also our financing strategy. And while we were doing it, um, end of 2018 already, actually, our hypothesis became somewhat reality, right? You are well aware about what happened. Google released BERT, the first of its kind of transformer. Uh, we were among the early contributors also into hugging face transformers. And this was when we felt, hey, this is, this is somehow, you know, this is a new level of technology maturity, and this is what we were waiting for. So we exposed ourselves a lot to this technology. We only built applications for customers with transformer models. And then end of 20 19, Um, we somehow, you know, reflected and tried to find a common denominator in the way we were building these transformer-based NLP systems. And this is how we came up with Haystack, because Haystack was, um, we understood that, look, there is somehow always a set…

AI assessment note: “Exactly. Exactly. We worked for companies like Airbus... it was consulting.”

Q a VC. Uh, but, uh, the, how do you find the transition from consulting to, uh, building products? Was the company small enough that it didn't really matter? Uh, you know, a typical way of thinking about those things is that, okay, the first few years you create the DNA of the company, and if DNA is consulting, evolving the DNA into product is, is hard. What was your experience?

A Um, we were, I would say it was, it wasn't as, um, as hard for us. Um, on the one hand, the team wasn't super big at that point in time. I think we were like around 10 or 12 people, but this was because we used already a lot of the, of the money, not to invest further into consulting, but actually to build out product and early versions also, also of Deepset Cloud, actually early versions were already built while we were bootstrapped. Um, and then it's a bit, you know, the initial, The initial aspiration. Why do you start it? Uh, and for us, it was always clear that we're looking for a way to actually capture some big opportunity that wasn't really around at that point in time. You know, people weren't really aware what we're talking about. No one knew the abbreviation NLP LLM. This abbreviation wasn't even born. Um, so, but we always had this aspiration. This is why I have to say it was comparably easy to in the end, um, pull the trigger on the shift.

AI assessment note: “it wasn't as, um, as hard for us.”

Q Uh, the fact that you have, uh, effectively selling to several personas. You have the, uh, you have the ML AI people, of course, but also the software engineers and then the product people. What, what, what have you found, uh, so far, uh, works best in terms of like where you want to land and who's the key audience and, and, and buyer ultimately for a solution like Deepset?

A Um, Look, uh, even while being in consulting, um, even then afterwards, when we launched, uh, Haystack, um, we see, we see, um, we see strong adoption really like from, from enterprises, um, from large companies. And this is pretty much what DeepSight Cloud as a product also, you know, has evolved and has been built. This means if we think about the segment, it's really like an enterprise platform and enterprise product where you have, you know, multiple teams, plenty of use cases, um, You know, really high requirements to your application. And from a sales standpoint, that means, um, we're often not really selling to our community members, right? So our community members are adopting Haystack and maybe they work for one big bank or big media company, whatever it is. Um, but what they, what they in the end do is they, you know, They are the technological trust, right? This is, they say this is a standard and this is a framework. This is technology we can really build our applications on because it's robust. It's reliable. All of these things, right? All of these tick boxes. Um, the buyer is of course, usually someone higher up the ladder. And here we see that, um, to sell into the enterprise, what you need is you really need a use case, right? And you really need to have clarity about the use case. And the best person to sell to is the person that is responsible for serving a u…

AI assessment note: “the best person to sell to is the person that is responsible for serving a use case.”

Q And in terms of, uh, LLMs, I'm sure that list is evolving very quickly as well. What are some of the key ones you support?

A Um, of course, the OpenAI models, um, which are in particular, very well performing for, um, all generative use cases, uh, but we see, uh, the Hugging Face models, um, from the Hugging Face Hub, um, the full hub is supported, um, are still very popular, um, opens, of course, Haystack is an open source framework. We're big fans of also open source models, more transparency we can create, um, also around, you know, how models have been trained, You know, the more research we have around the model, of course, the more trust it creates and easier it is for companies to adopt. Um, uh, so yeah, also hugging face quite popular, but look, um, who here models, um, anthropic models, all of those are also supported.

AI assessment note: “OpenAI models... Hugging Face models... anthropic models, all of those are also supported.”

Q a wonderful conversation, maybe, maybe to, uh, close and sort of zooming out of, uh, of, of deep set. Um, what do you find, um, particularly fun and interesting in generative AI in particular, but also AI in general these days, you know, like cool, I don't know, products, projects, companies, uh, whether they obvious ones or small ones, uh, that, uh, people should Should know about and look into?

A Um, there's, uh, there's a lot, as you know, right? It's, uh, it's, um, it's, uh, yeah, it's really, it's really a lot of, a lot of, uh, a lot of stuff out there. Look, in particular, I think, um, yeah, we, we, we love this, um, this somehow new space that is forming also in, in the area of LLM, um, uh, observability, right? So pretty much what I was also talking about with DeepSight Cloud, um, this idea of, you know, how to monitor your applications in production, make sure that, that, you know, everything works as you want. And I think, I mean, looking at the observability market itself, I'm always fascinated because there are so many players, so many big companies, New companies that are founded, right? Uh, and, you know, Observability is an endless market, it seems. And I'm very excited about LLM Observability because I'm not sure if it's like just, you know, like a small sub-segment or if it's not actually, you know, an own, an own massive market itself. Um, and, um, it's definitely something where we see more of the younger, newer players coming in one company. Um, uh, one company I like is, for example, Context from London. Um, But, um, but yeah, there are, there are plenty of players around, but I think that's, that's something that, that, um, yeah, that excites me where this is going, you know, how the product will look like, how they will differentiate, because I thin…

AI assessment note: “one company I like is, for example, Context from London.”

Q So maybe to close on the product part, uh, how do you think about what goes into open source versus what goes into the paid, uh, cloud product?

A Um, it's really, Right away, we're not following this open core approach, right? Where we say, um, where we say there are, so everything you build on Deepset Cloud, every LLM application that comes out, you technically can also build it just by using the open source framework, right? Because the architecture, the technical architecture, everything you need from API point, from the one API point, which is query, and the second API point, which is upload files, data, All of this in between, that's open source. This will always be open source. Everything that is Tooling for a workflow, right? So how do we, how do we support, um, evaluating? Uh, how do we, um, what's, you know, where, how does the backend look like to track, um, AB tests? Um, all of these, all of these, if you will, organizational challenges, right? All of this management layer, these are things that, um, that we simply see as, as platform features and, uh, where we think it's, it's, um, now that's, that's simply, of course, You know, usually also something a certain group of companies requires, right? I think not everyone in the, in, in, in the community probably will also use it because look, if, if you care so much around about the factfulness of your applications, you're probably not a startup, right? Um, startups, you know, you want to ship fast. You do whatever works best. You assemble something, it's shipped…

AI assessment note: “All of this in between, that's open source. This will always be open source. Everything that is Tooling”

Q How difficult is it actually to bring in a database or model? Like, do you need to create, uh, like a custom integration each time or is that, uh, standardized to some level?

A Um, to be honest, it's, um, you know, the, as, as, um, as the capabilities of the database is also mature, it becomes more and more individualized. Um, this is why we really, you know, invest also a lot in partnering with, um, with, for example, with the database providers, such that it's somehow, you know, like a co-ownership of the integration where also the database provider support us a lot to make it a good experience to always keep the integration up to date. But, um, yeah, the products, uh, all these products are maturing and also differentiating, right? Um, so it is becoming more and more of an effort, to be honest, to really manage this. But this is also, you know, pretty much where the value proposition of something like Haystack comes in, right? That somebody takes care for the user about it and the user can simply, um, pick what they want or what works best for them for a particular use case.

AI assessment note: “it becomes more and more individualized”

Q uh, Gipset cloud, uh, semantic search, summarization, Q and A, all the things, uh, this one that you alluded to a few minutes ago that I think is particularly cool and interesting. Uh, that's the hallucination detector retrieval augmented generation. Uh, do you want to talk about, um, maybe the hallucination problem General and the way people have been fighting it and more specifically what you guys do about it?

A Yes, of course. Um, so hallucination in general means, um, you have your model and you want to, um, let's say question answering system. You want to You, you want, you have a question, you want to know a fact, right? And now you want to make sure that this model gives you the proper fact. And what we see is that models maybe don't give us the right fact. Maybe they, you know, even don't know this fact. We often don't know it, right? Because it's often not really clear what has the model been trained on. And this means in the end, hallucination is more or less like, hey, does this say something that is simply not factful, right? This is something that is not True. Right. Or not. Yeah. Not, not real. That doesn't make sense. Um, the first, the first solution to this is to augment the model really with what you, with your data. Right. So when you want, again, like if you ask chat GPT, um, today about a question about your podcast, it might hallucinate.

AI assessment note: “the first solution to this is to augment the model really with what you”

Q So that's Haystack. Let's talk about, uh, Deepset Cloud, which is the commercial product, um, a SaaS platform for NLP teams. So what, what does Deepset Cloud do?

A Um, When you think about this, uh, let's stick to the example we had in Haystack, right? We have this question-based, question-based search for, um, for your podcast. Now, the first thing you have to do is, right, you have to develop this application. That's the first big life cycle phase you're in. And in order to develop it, what you have to do is, and that's common in all of machine learning, right, is experimentation, right? So this means you start to somehow evaluate the performance of these applications, and you try to, you try it out with different parameters. For example, you will try out different models, right? You want to see, hey, for my podcast, what's the best option? Is it GPT, 3.5, or is it, um, is it, uh, is it, I don't know, uh, one of the T five models from, um, from the hugging face model hub. And in this moment, you need some process. You need to follow some process, right? To evaluate this and to make that choice. Now in traditional machine learning, this is traditional machine learning, uh, whatever that is, but think about a time series problem, right? Time series prediction. It is kind of straightforward because you simply take a past, a past, um, a past time series that you observed about the phenomenon, let's say a stock price, you run predictions against it, and this is how you can iterate over Your model, right? But how do you do it for a question a…

AI assessment note: “in order to develop it, what you have to do is... experimentation”