DN #28: Runtime Economics, The AI CFO Gap & Why Expensive Models Are Cheaper (w/ Joakim Hauge)

Niklas:Hi, and a huge welcome to you, my lovely listener. So glad you're here. Today you're joining me for a chat with Joachim. Joachim, so nice to have you.

Joakim:Thank you, it's an honor to be here.

Niklas:Joachim is the co-founder of Monetize. He's building economic governance infrastructure for AI native firms. That is really interesting. I think we're not really seeing agent to agent transaction that much ⁓ yet. We will probably see them explode though. So this is I think a really good topic. Before we dive into that, maybe a few words on your background. How did you end up building ⁓ infrastructure for agents?

Joakim:⁓ That's a good and ⁓ long question actually. ⁓ early I understood that economics spoke to me, so I like to... figure out how to improve and to kind of optimize business ⁓ and margins. so I graduated in finance, in quantitative finance, specializing in risk management. And I've been in the ecosystem of startups for a long time and I've thrived in trying to solve new and important problems. So that... When AI came, like big after the chat GPT came into the market and stuff, you know, were talking about this. And I soon discovered that what's normally is deterministic behavior in software runtime. It was actually non deterministic as you couldn't actually just predict what was going to happen. And this kind of volatile behavior was extremely interesting. And, and. At the beginning nobody cared. It was like, interesting, but nobody cared. But as these tools became more more autonomous, people were starting to give me more more attention. So I figured out that this was so interesting that it became a business idea. And here we are. I'm about to now launch ⁓ Unitflow, which is the governance layer for enterprises coming later this or next month.

Niklas:You're documenting your journey, I think, from start to going supersonic as you would say. ⁓ why why did you decide to go this way and not build in silence?

Joakim:Building in public is actually a bit of a scary choice I made. It wasn't because I wanted to, it's because I felt that it would help people, help me validate, you know, making people aware of the problem and get that necessary feedback because I wasn't, even though I felt that this was an important problem, I wasn't sure that my solution would be the ideal solution for the audience. So I wanted to get that product market fit. I felt it was necessary to get that feedback from the very start. So, and then I heard about, was on IndieHackers and I found this building in public on X and it was really easy to get in touch with people and I got feedback and so yeah, and that's how I got started. It wasn't planned or anything, it just happened because I was, you know, was easy to get that feedback and people were really interested in knowing more.

Niklas:It's a a pretty nice community, I would also say. So I've been that's kind of how my account grew on on X over time, being involved in that. And I'm I'm now ⁓ the account has gotten so big that I'm getting a bit out of this ⁓ this world. But before, like it I always felt that this was a really nice subculture to be in. And it also is extremely valuable, you're able to get feedback fast. I think like this on the things you're doing, ⁓ especially from users. So I think it's great to to still build in public, even though building software has become really easy. When ⁓ when you look at the idea of runtime economics, I I think we are seeing this shift from traditional DAS thinking to kind of cost patroken metrics or something like Like that. When did you notice that this was actually a thing?

Joakim:⁓ well... It's it's and these were concepts at first I felt it was really when I was speaking about like runtime economics I was mainly concerned with the volatility the fact that there is a some uncertainty associated with whatever you're doing that you're not accounting for was what I was focusing on but when I spoke to the market they were more concerned about cost optimization and model rerouting and more like and this was and it became a little bit like how do I kind of communicate with that and I noticed early that when I speak to the like the CFO and the executive managers of a business they immediately get it but when I speak to the established people who engineers and who were the stakeholders in most of these projects now you know when I was doing validation they would refer to more you know ⁓ AI infrastructure type of mindset where it's like cost optimization, model rerouting, prompt trimming and that kind of orchestrating the AI workflow. ⁓ And then it became immediately, ⁓ over time it became clear to me that there is a stakeholder issue here where the current management is the engineers and the stakeholders who fund these operations are a little bit in the dark. and then these concepts are also... ⁓ kind of from the engineer perspective, you and the focus here is that your backend needs to be running, it needs to be operational, it shouldn't have too many errors and then you know the lights should be on. It's about optimization for keeping things up. Well on the stakeholder side where the CFO is involved or the CEO, they want optimization, they want competitive edge, they want to make more money, they want margins, they want a profitable operation, they want return of investment. So those two paradigms are kind of different. And I was kind of in the mix. My concept was a little bit here and a little bit there. So I had to kind of make a clear choice. And I think those concepts of the tokenomics, when that became a thing that the market was talking about. It became apparent to me that I have to make a clear choice. I, you know, build, improve the current orchestration environment and go head to head against like line chain and all those very established market, by the way. And, and, know, the tools are great. Or do I try to carve out this new category, you know, in emerging, you know, as these autonomous tools become more autonomous and this market becomes more and more mature, try to carve out a new category and define that as something. new and value bringing. And I chose that and it became like runtime economics. And it's not that it's different or anything. It's just simply we're looking into the runtime telemetry data rather than looking at logs and traces and using that data to build an understanding of the value creation and the risks associated with these autonomous systems. So that's the Namia.

Niklas:If you put it again into one sentence, so what what exactly are runtime economics very crisp and precise?

Joakim:We're using runtime telemetry data to understand the value creation and the risks associated with running autonomous AI.

Niklas:You're I think at a good point in time because currently companies are starting to get obsessed with tone costs. I think like the first first few months of this new technology, especially with colored code and so on, it was just let's get it up and running. And now I feel there's a huge shift. I think Huber made huge headlines on trying to somehow cut Koston Token spent. Then because they say they cannot measure the impact, like they don't see it. Like there's they don't see the value. ⁓ this is really really interesting, I think, also in a traditional finance sense, right? So ⁓ because you you would typically kind of learn to to price something and manage the volatility. Instead of kind of trying to completely eliminate it or something to go to go back on it. And ⁓ what what is it what is the difference here? What's your view and also maybe your take on the current situation with these large companies? Are they right in cutting so much cost?

Joakim:That's a complex question. can't answer on each. There are good arguments for both, I guess. And I think the common denominator for why these costs are allowed in the first place is that It's usually in you know, where the early days is still very young market. And we, we see this huge potential on the upside of AI. We can, you know, completely replace huge teams. So thinking that you can replace like a hundred people, 200 people in your organization, you immediately see the value that could bring on your balance sheet. Right. So, so that's ⁓ a lot of money because, you know, usually one of the largest costs in an organization is salaries. So the upside here is very clear. But as the market becomes more more mature, you will see that you need to replace, someone needs to manage these systems and someone needs to build them and someone needs to do this. So you're not actually getting rid of too many. In fact, in most cases, they're still there. You just now have this additional cost. And so once the organization starts to see that, OK, this kind of naive understanding of ⁓ upside is might be somewhere in the future, but right now I have to defend my investments. when the return of investment is coming. And I think that has kind of put a ⁓ little bit pressure on the... on the cost situation and also is involved with the CFO. I think visibility here is a huge problem because the CFOs were kind of left in the dark. They just signed off on these huge budgets because everyone else was doing it. And they were left in the dark because everyone spoke to the guy who understood these systems. And that's like the engineers, right? So they have been kind of the focal point up to now. the minute it becomes uncertainty and problems and now everyone's pointing to the CFO, what's going on? Where is my return on investments? And they have no visibility. So we're kind of in a place now where tools need to speak to the... stakeholders that are signing on these checks. we're, you know, seeing the kind of feedback when we out and talk to customers that we, when I say it simply, we're not trying to sell the product before, but I'm simply saying we're giving you visibility into, you know, the risks and the costs that you're, you know, you're enduring now in, when you're, you know, embracing these AI transformation projects. And when can I get the product? When can I test it? You know, I'd like to see it. How does it work? You know, how can we started. It's tremendous and this wasn't the case only like two three months ago. People were like you know this is nice to have, know this might be a problem in the future but right now we're not too concerned with costs. This was April now, people can't wait to get started. So this cost has scared people and I think it's mainly because they're starting to understand that this naive upside might not be as clean cut as they thought in the beginning.

Niklas:Yeah, so different points on this one. I think at least the so going back to the Uba question, so I think token spend is ⁓ justified. ⁓ maybe not all of it, but at least in parts. it's just really hard to measure. And I would I think that is goes back to like the classical a classical problem in IT. It's very hard to measure productivity gains at the end of the day. So value creation is hard to quantify. Now we have the second side of it, which is at runtime, right? And we are looking and I think this is also very different. We kind of used of the pricing module models of the hyperscalers. I know there are a lot of people who are afraid of it, especially like smaller entrepreneurs, something like a DDoS attack can kind of cause a significant budget impact. It's even worse, I think, with agents as you already said because they don't necessarily behave deterministic. So somehow there are some some loops in it, some retries, some tool calling that doesn't go as planned. There's some replanning. So they you don't see the same cost for the same task, right? So how how you do you deal with something like that? I think that's really interesting to measure.

Joakim:This is an interesting challenge and we're looking into this and since we are not completely finished, I'm not going to say exactly this is how we're going to do it, but... It's all to do with the universal, know, how who you're speaking to right and I think the the context here in when they talk to you know, CFOs it's some data and cost or income and some uncertainty and it's usually over a period of time a day or a month or a week or three months So so so we are not going to invent those concepts. We're going to use those concepts to try to explain this. So it will be usually within a month or a day and then we'll add some sort of ⁓ more granular to kind of capture... what is important in this value creation. But try to keep it as financial and non-technical as we can. I think that's important because it has to be... Not because it's technically complex, it is, but if you can't see these problems or explain them very simply and clearly, it's very hard to make ⁓ good inference and good decision and good strategic... choices based on it. I think that it's always going to be like, know, we're looking into concepts like value at risk and risk adjusted returns and these borrowing from financial services. And if we end up using something like that, that at least will kind of give a picture of the value it brings and how to deal with it. So it will be like, one day, one hour and so forth. is what we think is the easiest for the stakeholders we're talking about to understand and to do something about. ⁓

Niklas:When when you look at it, I think what what is now different is that we we will see a lot of of volatility in cost, which is very hard to associate to value, right? So how do you deal with s with a scenario like that where you where you have kind of a volatile cost structure which which is hard to associate to to the exact value, yeah.

Joakim:I mean, today it's really, I mean, you have a lot of things going on during runtime. In the beginning, it was more, we were more focusing on these cost spikes that happen where you have like, what would be considered as token waste where, know, runway loops, know, this continued reasoning tool chains that kind of spin out of control. ⁓ But you can also see normal, where these models are doing what they're kind of supposed to, and you still fall out of outside of the budgets. so, there's a lot that indicates that, you you need to be inside runtime to adequately govern these tools, especially when they're multi-model, because they'll go from tool to tool to model to model. And it's difficult for the orchestration layer to kind of control that. to some degree also be inside runtime to intervene to make sure you stay within budgets, but it's also to understand the risks. ⁓ I mean, we don't have, I mean, we need to test this more to, be a hundred percent certain of, know, what, what represents volatility and what represents just, you know, trying to do a good job and, and, you know, all job, all, because the context window keeps changing. You're not essentially from a, you know, a, you know, AI point of view, you'll never ask the same question twice because you will, you know, it will be a different circumstances. The context window will be changed and et cetera. So I know we don't fully understand the consequences. those things. it's difficult to say when is it wrong and when is it right. We can say from an economic perspective, if you're making money that's good, if you're not that's bad, but other than that it's not that straightforward. But what we are doing now is to kind of... govern those things and to help with a more comprehensive policy engine to kind of gracefully downgrade, reroute and to use human in the loop if that's necessary to prevent further damage. But at the same time, do it in a graceful way so you don't incur bad user experience and higher churn because you're now trying to save money and then the users are kind of left with a... bad experience. So this plays into it as well. yeah, so we haven't a full down detail. This is the risk category. We see when you the volatility, the variance in the pricing is what we measure. What actually takes place in the AI, ⁓ we don't have the full picture of that. I mean, it's a lot ⁓ and we don't have enough data. So it would be speculation at this point. Yeah.

Niklas:When you put it at the core, like and try to summarize ⁓ how unit flow builds on the idea of runtime economics. So what what exactly happens? How can I imagine that?

Joakim:That's a good question. What we are doing now in testing is to... I mean, we're looking at telemetry data and what we're doing is we understand the margin. So we have like this margin. The three pillars that we're trying to build is like visibility, control, governance. So when you have visibility and you can do something about it, that's control, you can move from experimentation to implementing strategies. So now you can, you know. align your operation with business objectives and that's the governance part. So within visibility we have what we call like margin intelligence that's like where we look at profitability and the margins and you can set some kind of you know ⁓ targets and cutoff points and then we have some exposure analysis where we look at how bad can this go. ⁓ And then we have on the control side we have like this budget guard where it intervenes when you it thinks you're about to break your budget. It predicts you know how many steps are left in the session and if it believes you're going to break the margins it will start to downgrade and try to kind of avoid that. And then we have the policy engine which describes all the aspects of what can we do to try to prevent the cost. as gracefully as possible without protecting the user experience at all costs. And then we have the final, which is like a reporting tool, which reports on all the activities to the CFO and the CEO in a way that they're familiar with. So that's kind of the basics of what we're building.

Niklas:If you if you think about it even further, and I I would kind of like to separate it into three parts, like cost value and risk. what what do these three actually l look like in in a to or for a tool like Unitflow? And do you take a look at all three or do you focus on costs like what what is the what is important here?

Joakim:I mean, when you look at both the value and the costs and the risks, and the risks are ⁓ financial, economic risks. We don't look at persistent memory or hallucination or the other challenges during runtime or security itself, someone wanted to cause harm. We look at the economics. ⁓ And that's sort of the risks here is that you don't make money and you're actually running at the deficit. And so when we look at the costs, we want the true cost per successful outcome. So we don't and that's not tokens, right? That's actually the reality, you know, what hits your cash flow, your bank account. And so that is actually An important part of it is to kind of link it to the financial operation, your cash flow statement, your, you know, to your overall operation and not necessarily the amount of tokens, even though it plays a part, it's a different language. And I think the language and the framework and the context here is important. And that's part of the problem why these, the CFOs now is in the dark because he, he will, you have to follow, go into, you know, leave his current framework and go to the, to the tech center and look at dashboards with his engineer friends and he's not getting it right he sees tokens and it doesn't speak his language and I think the translation into ⁓ a more traditional income statement you know balance statement cash flow statement kind of framework is important for these stakeholders to kind of see the true value and the true risks of their operation and so that's the essence yeah

Niklas:think it's really challenging because like a year ago I talked to friends about when I talked to them I always said if you build to solve a problem, use the most expensive model on the market and assume costs will come down. Like just optimize for output quality, get it as good as it can be, right? So now at this point in time I think smaller models like G Mini Flash or so on for for a lot of tasks they had just clearly good enough, right? You have some long document, you want to pass it and extract some data. It's good enough. You don't have to take the the state of the art. So you now have the tasks that require like frontier level models and you have a lot of more routine tasks where even open weight models will come into play. And I think we'll see the idea of keeping your data private and on like infrastructure you control and not sharing it so much with the with the two big players, OpenAI and Entropic, we would also see see this more. So what does all of this ⁓ mean? Do you also try to make these cost optimizations? Do do you stay away from recommendations? What how would would that actually work?

Joakim:think cost is a big part of it, obviously. You don't want to waste tokens. I mean, we've tested it and in many ways it is actually cheaper to upgrade to an expensive one. We know that for a fact. That's because some of these recursive practices where it's kind of tries and tries again, doesn't happen when you move to a more advanced mode. That doesn't mean that's the case in all situations. Sometimes it's the opposite. And to fully understand... What is what? I think a lot of it goes on in how these context windows are devised and how they're updated and how they kind of approach. And we don't have enough data. don't. We're too kind of it would be speculative. So we try to stay. out of the kind of cost optimization. But we do, I mean, we could do the same thing as a lot of these API providers do like, and rerouting and recommending spot prices and do some sort of kind of inference on that. But to be honest, I don't see the value of that. It's not, mean, looking at the spot price and keeping that, I mean, it's relevant data. I'm not saying it's irrelevant. It's just that it... doesn't correlate really well with the bottom line. So you need more inference. And I think that is one of the things that has caught the market a little bit off guard. thought cost optimization would... one of the things that has caught the a bit guard. thought cost optimization would... ⁓ mean what it actually says, But in reality it doesn't. mean what it actually says, But in reality it doesn't. It doesn't optimize the cost at all. It reduces cost maybe, but to call it optimization it would be misleading. And I think we read about those stories in the news all the time. ⁓ So yes, we will provide a lot of those services, but it's not a focus, no.

Niklas:It's a really interesting takeaway, right? Because actually it's counterintuitive, as you said, right? So if you take if you reduce the amount of tokens used, it can be cheaper to use a more expensive model in total on the way you created. I think this is this is also something that will be discussed more in the future. If you look beyond that, you have published something on GitHub, it's called circus circuit breaker, ⁓ which is kind of a runtime Governance. Why would I need something like that?

Joakim:For us it was a way to start testing, know, and get people, get feedback and to learn and to talk to engineers and to kind of, you know, see how this, you how did they receive products like that. It was very incomplete in many ways, but it got us started. You would use a tool like this to make sure you stay within some budget parameters to show, I mean, for validation, if you want to scale something, And this is going to be, in my opinion, best guess is going to be tools like this will be important as a part of validation because... If you can scale a product just by kind of, you know, spending tokens and spending, you know, that's completely different than scaling something with margins. So, and also validating that you have a customer base and you're able to make money on a small group of people is a stronger argument to scale into a larger business and something that you haven't validated, whether it's you can make some margins or not. So I think in the terms of validation, I think a tool like this could be extremely useful. But also we have like you know this policy side you know to understand your users what can they accept and what do they you know what can they tolerate and what do they want you know in terms of margin protection you know if you downgrade what kind of interventions can you do without hitting churn you know it would be a balancing act. It's a bit like you know I like to compare that with portfolio management where you know you want some more You want some risks, but you don't want the bad ones and you don't want too much. But if you don't have any risk, you're not going to make any money. same here, right? So you can't just kill the session every time you are threatened to lose some money. You have to find a way to balance it. And by doing it graceful, you're still accepting some risks, but you're trying to balance it and not to trigger the churn. That would be the things that these tools could help with.

Niklas:You want some asymmetrical upside, right? So you want to flip a weighted coin. That's at the end of the day, that's it. But you will lose sometimes. That's the whole idea of risk management. If you I think it will get even more interesting when we, as I said in the beginning, have the agent to agent payments that happen on blockchain, whether it be on an institutionalized one like the Stripe or something like that, but we will see agents paying to agents. And I think then it would become even more relevant to kind of avoid that the agent just drains your bank account kind of. So ⁓ I think this is also something maybe in the future I I think probably not part of circuit breaker yet, right? But something you could think about for future use cases. ⁓

Joakim:Okay.

Niklas:When when you look back and when I've also built companies before, when you look back and think about the mistakes you have already made, what would you recommend for people who are actually starting out the first time? What can they already learn?

Joakim:Wow, that's a good question. There's a lot of things. But I think if there's one thing is that for me now, and I'm leaving that right now, is that... I mean, there's a lot of focus on ship files now. There's a lot of people there, because AI is help speeding up things tremendously. And I think shipping files is a good thing, right? You fail files, you win files, you do everything faster. So that's a good thing. But ultimately, you think of it, it's highly nonlinear as well, think, because quality of service is like, what is important? And I think... Or if you sell something, explain something, if you're going to raise capital or if you're going to do whatever you're to do, if you really truly understand the problem. from the user's perspective, whoever you're going to sell to, if you understand their story and you really do that well, your solution is going to hit sooner. It's going to hit better, cleaner. And if it doesn't, you'll know because you'll get better feedback because they understand what you're trying to achieve. And if that's a miss, you will get that feedback. And I think the shortcut is if you do that really well. That is the time saver and that's the definition of speed. Don't try to rush that, try to do that part well and then everything else will just move so much smoother and so much smarter and so much faster. So take that and it's not what you believe, it's what whoever is going to buy from you believes. I wish I kind of understood that ⁓ part bit sooner. ⁓

Niklas:Yeah, that's one hundred percent right. So the customer is right. And there's no like and I think that's also large value business building in public. Don't worry, people will build the same stuff as you build. ⁓ you might be a little earlier, but if it's a really good idea, latest when you're on the market, people can sign up and use it. They will see it. The worst thing you can do is build something. Nobody wants. I think AI has made it a lot easier to to build something fast and get it in front of people, which makes this validation a lot easier than it was before. But I think at the most important step. The worst thing you can do is build one and a half years in a quiet room, then go out and realize you have built something nobody wants. Like and you had this really great idea, but you'd find no customers. Like it's really important to find people that actually buy it, that use it, and then scale from there on. And I think this is this is a a general basis. What I would also, and that's maybe the last question, find very interesting is to understand when runtime costs can vary quite widely. what is the impact of this on pricing? What should founders keep in mind when they now price their product and you can have such a high variance in cost.

Joakim:That's actually a very tricky question. And part of the challenge is that we're coming into this from a deterministic, like SaaS world where you bought seats, where the costs were fixed and you had customer acquisition costs, but it was relative, at least deterministic. So I think the pricing mechanism now needs, I mean, you have to be usage-based, but even usage-based has a challenge because it happens if you're using it based on log data, then obviously by the time you understand the costs. It's too late. you'll have to. you and we're considering also building something there that helps with the pricing. So instead of, you know, cutting the session, we'll just add pricing data to the user session. So now that it gets priced correctly, this would be highly experimental for many reasons. And we haven't actually tested it. So, but it would be nice way. And that's actually an idea that someone could look into. I think that would be something worth experimenting on because right now I feel like you know people say you know we're going towards usage base but it's only the bigger enterprises that have the mean to kind of do that in a meaningful and useful way everyone else will be have to guess for the time being so something along those lines a very good pricing engine that actually helps in with real-time data that would be something for people to look into and to solve and if it's not there, build it. ⁓ Otherwise, I think it's a bit of a challenge actually because you can't put the price too high because then people won't buy it and if you go too low you're in the risk of losing money. So this is a really big challenge right now.

Niklas:Joachim, thank you so much for being on the podcast. It was a pleasure having you and to me, you, my listeners, see you next time.

Joakim:Thank you.