AI Observability Metrics: Quality, Risk, and Business ROI

Amir: On this episode of the show, I have with me Jeetendra Soneja. He is the VP of engineering AI platform at iHerb. And we're gonna be talking about AI observability. We're gonna be talking about the ROI, quality, scale, some of the concerns around AI and observability, as well as some skills gaps and a couple other things if we can fit in there as well.

Jitendra, thanks for taking the time.

Jeetendra: Yeah. Thanks so much, Amir. pleasure to be, talking to you. Thank you so much for the invitation. yeah, I'm looking forward to our conversation. There's obviously a lot going on in this space. I think we are all learning, we are all adapting, as we go along.

absolutely. All right, before we dive in, iHerb, what do you guys do over there?

so iHerb, I actually recently joined iHerb about three months ago, as mentioned, VP of engineering. so it's a wellness e-commerce company. it operates in 180 countries, selling wellness products, supplements, vitamins, minerals, among other things.

that's what iHerb is about.

Amir: Okay, awesome. So this episode, obviously, we're gonna be talking about, some of these components of, I-I-AI within that observability within the platform. you have a deep background in this space. Uh, ri-right now, the state of AI observability, talk to us a little bit about, what do the tools look like?

o-obviously saying maturity is, is a tough one 'cause e-everything is new and everything's changing. But just where's, what is the state of AI observability in terms of, meeting goals right now?

Jeetendra: Yeah. So I would say, it has evolved quite a bit in the last, two, three, years now since, everybody started using AI, building AI agents.

observability, I think more and more people are, starting to be more aware, the need of observability, how important it is to have observability. so I would say it has evolved quite a bit and continues to evolve, both from model providers' perspective, I think they are starting to provide more transparency and, monitoring and, observability into how their models are thinking, what tools they are calling.

some of them even, provide like industry standard metrics like OTEL metrics and traces. but as a whole, like taking a step back, I would say, AI observability is different from like traditional software observability, because it also not only it has to monitor like more deterministic things like errors and latency and availability and so on and so forth, but also now you have to deal with like sto-stochastic nature of the AI models where, uh, they can hallucinate, they can make things up.

So you wanna be able to monitor not just availability, but also more importantly, the quality and the groundedness, uh, and whether your AI agent or AI chatbot, what-- as the case may be, depending on the use case, are they able to accomplish the task? And, uh, also I think, yeah, also I think one thing that's also becoming clear, cost is an important metric to monitor as well.

'Cause it's one thing to get it done, it's another thing to get it done economically, feasibly, so, so you can actually convert that into a ROI for your company. those are some of the things. Of course, there's a security angle as well, which is very important, if you wanna get into it, but, things like prompt injection, things like, safety, like, toxic responses or going off track or, having permissions that it shouldn't have, executing, destructive commands, irreversible commands.

Those are some of the risks that you have to watch out for. And, yeah. So at the end of the day, you, you wanna make sure you understand how your agent is kind of getting where it needs to get. So that's where observability, becomes really important. there's also one last thing I would say, and I'll pause, is there's also this concept of compound engineering, where observability really is your feedback loop for the agent.

'cause it is gonna make mistakes. It is non-deterministic. So it's, once you have observability, it can use those traces, it can use those logs to feed it back and understand, okay, what it did wrong, what actions it took, where it went off track. So, so it's really an important, feedback tool as well to continue to improve your systems.

Amir: I, I guess, you know, you, you, you walked us through th- yeah, obviously some of the, the challenges. when you're looking at those challenges and you're looking at... Ob-obviously you also mentioned the tools have been evolving, the platforms are evolving. It almost feels like as soon as we hit a, a, a SWAT one milestone, another one's breaking, uh, you know, another enhancement here, another enhancement there.

Does that represent a challenge at all when it comes to some of the, the risks you outlined?

Jeetendra: Yes. so there's I think couple of aspects too, like to your point, like platforms are also moving fast and evolving. one is the models themselves, right? They're getting more capable. so historically, again, teams have, built more, I would say scaffolding, around the model because they were less capable, quote-unquote, less capable, let's say even a year ago or even six months ago.

so as models get more capable, a lot of that becomes obsolete. So again, observability can give you that data. Like once you have the right evaluation datasets where you can test a model with and without scaffolding, and you s- you can continue to kind of measure is the scaffolding really still needed?

Is a layer around the model, you know, how heavy it needs to be versus just having a lightweight orchestration layer around the model to get your task done. so yeah. So that's one aspect of platforms evolving. The other aspect is also, as we are seeing in the, industry, I would say both like harnesses are evolving, but also like things like MCP and tools and, you know, memory management, context management, all these.

but at the same time, I would say some of them are still unsolved, right? Like how do you manage memory? How do you kind of make... kind of find that sweet spot where you are giving model the right information so it can make a informed, uh, decision about its next step.

At the same time, you're not bloating the context, but not providing too much information. So not too little, not too much, kind of finding that, sweet spot. so that's where I think platforms will continue to evolve, and that's where because, uh, you know, you need to do that context management on top of the model, that's where companies are investing when they are building, whether it's in-house platform or purchasing third-party platforms, is making sure, uh, everything around the model, whether it's tools, whether it's MCP server, whether it's context management, whether it's observability, those are the capabilities that are, kind of on top of the underlying platforms or underlying models are also evolving in parallel.

Amir: W-when you think about observability, you think about, you know, obviously metrics. Um, w-what are, what are metrics right now that, that you find are probably most important within the AI observability space?

Jeetendra: Yeah. So the way I would phrase that is, there are some table stakes metrics that you just ought to have.

and again, these are, you know, basics like, as we mentioned earlier, availability and errors and latency. Again, depends on your use case. For, for, a agent working offline, latency is not as important. But something user-facing, users waiting for the response, latency becomes more critical. Um, so those are some of the table stakes metrics.

The other table stake, I would say again, I mentioned earlier, is cost. Uh, you want to be able to, attribute how much e-each task, is taking in terms of dollars spent by the agent, token spent, to accomplish a given task Uh, the other metrics that are-- I think this is the area where I think industry is continuing to evolve, and at the end of the day, it's very domain-specific and business-specific, is around quality metrics, right?

Quality is very subjective, and also depend-- varies from use case to use case. So to give you a concrete example for, a chat that's a e-commerce chatbot, all right? Like the quality is more about making sure, um, all the responses around your product recommendations, around your products are grounded in your catalog, in your inventory, and all the product, product description is not making things up.

So, so it's like groundedness is important. the goal is really to drive conversions. if you have a chatbot, let's say, that's doing, like financial planning for you or, or even an agent that's, let's say, doing code reviews or writing code, like, you know, we see all these coding agents. The metrics, uh, the quality metrics there are very different in terms of like you-- that, that's where you need to develop a-almost like domain-specific evaluations to kind of tell agent what good looks like, so that when it completes the task, it's able to verify and, compare and validate its output.

this is where we also hear, verification is really important, the criteria. The, the more Concretely and deterministically you can give agent what the end goal looks like, the better outcome you're gonna have. Uh, so, so that's where I think, in terms of, metrics and evaluations are really important 'cause that, that help, uh, agent understand what good looks like, what right looks like for your given business,

Amir: to touch on as well, obviously we're talking about ROI. I mean, w- it is hard to measure. what, what are some concrete, like, examples? What are you seeing out there in terms of measuring ROI when it comes to AI?

Jeetendra: Yeah. it's one of the hot topics these days, right?

Like, as we are seeing, different comp-- like hearing from different companies where employees are starting to token max and spend a lot of tokens. so you ob-obviously wanna understand, did you get the ROI from all this AI spend that you are, are kind of, you know, allocating to your org? so there's few ways to do it.

One, I think is in terms of like cost savings. So this could be in the form of, for example, productivity, which by the way is notoriously hard to measure. It's tricky to measure. So you wanna measure like time to task completion, for an exam- as an example. It can be done, but, but just kind of flagging it, it, it being, uh, one of the tricky things to measure.

So productivity is one where, you know, a lot of, uh, organizations are using AI to automate internal business workflows and, you know, they, they wanna see productivity savings. This, this, uh, this task that u-used to take humans a week, now it's taking, one day with the A- with AI, so you can kind of, do the math in terms of, okay, you know, I was paying this, human for for five days, and now I'm getting the same thing done in one day.

So, so that's one way to measure productivity savings. for your, like, coding tasks and engineers, again, idea is similar where how long it's taking to ship things. One important thing I would like to highlight, because again, this is where we see, you know, a lot of Vanity metrics being thrown, thrown out on LinkedIn and social media around, like number of PRs going through the roof with AI.

Uh, yes, that's great. That means people are kind of, I guess, writing more code, but that doesn't really mean in terms of y- bottom line or top line for your business. So I think you wanna measure what we-- what you really wanna measure is business outcomes. So productivity is one I me- measure- mentioned.

Another example would be like if this is user-facing AI workflow or, um, uh, customer-facing, this could be customer satisfaction score, this could be like are, are they buying more products? if, if it's like, you know, the goal of the AI agent is to help your customers buy more product. Is it, kind of closing more support tickets without human escalation?

So, so those are some of the examples in terms of measuring ROI, and that I think should be part of observability, and you should connect all of your operational metrics like cost. Like I spent a million dollars in the last month, what was my ROI either to top line or to my bottom line through the use of AI?

Amir: Absolutely. I, I think you-- I mean, you actually touched on, obviously verification, evaluation components. When you're looking at a- AI projects a- and you're obviously observing behavior, I mean, slightly different than maybe what you were observing ten years ago in, in, in, in the cloud and applications.

When you're looking at that observability piece and, and you're looking at models, you're looking at, you know, sentiment, hallucination, I mean, drift, all these different factors, like when you're working on tolerances for a company, I, I mean, that's-- I, I don't know if that's art or science yet, quite yet.

There's a, there's... I, I, I'm gonna ask you, h- how do you tr-- how do you start of determining what the tolerances look like for an organization?

Jeetendra: Yeah. So, I would say, again, like many answer, it depends, right? Like, depending on the domain, what the organization is operating in. so for example, in a healthcare domain, the tolerance for not having grounded responses is very little, right?

So, ri-risk management, it, it does need to be part of designing your agent, evaluating your agent, having the right observability there. Versus something like, e-commerce or some, getting financial advice, again, depending on, you know, what actions you're gonna take, it can be...

The, the-- I would say the stakes, if the stakes are high, it becomes really important to ha- take that into account and, again, like develop the right guardrails, right, evaluations both, And this is where one, aspect of observability is o-- In the past, a lot of the evaluation, or at least a lot of evaluation e-even today are happening offline in the sense agent, you don't want to block the agent because user is waiting for the response, especially if it's, again, latency critical.

You don't want to block and, and evaluations are expen- one, they're expensive, and they are also kind of slow, because you are running essentially most of the times another LLM to look at the response and asking, "Okay, is this response grounded? Is this response have the-- does this response have positive sentiment?"

And so on and so forth. so again, depending on the domain, you may wanna have what, what we call like blocking, evaluations or online evaluations that can modify the response or block it. so if it's again, a healthcare domain, or, like, you know, you're, you know, giving, therapy is another one, right?

Like, you wanna make sure, responses are within your guardrails and so on and so forth. So there's offline guardrails, there's online guardrails, and you really need to, again, to your point, like understand what your risk tolerance is and define those concretely in the form of evals, in the form of guardrails, be it security or otherwise.

and that needs to be accounted for, in terms of both, not just observability, but also in your overall ROI calculation, right? Like, is this risk worth it? Like, the... Because again, models are not deterministic, so there is an element of risk. You have to balance that with the outcome, with the ROI that you're gonna get, and you have to measure, your ROI within that risk governance framework, to see, okay, is this worth the investment, worth the automation, for the benefit that I'm getting at, at, at the, at the same time it's presenting some level of risk.

Amir: Yeah. And I guess when we're looking at speed, you know, you, you can, you can, you can move quickly, but obviously there's dangers to that. If you move too slowly right now, there's dangers to that. When you look at that speed trade-off, obviously I know every industry is different, but just looking at what's happening and how quickly companies want to move, yet everything you've said about observability, I mean, i-i-i-it's essential if you wanna actually execute at, at, at any type of scale.

W-what, what are, what are your thoughts when it comes to this concern about we need to move fast, but we also have all these other issues to, to kind of c-consider and be worried about?

Jeetendra: Yeah. I would say it's, again, every organization needs to kind of, one, adapt and understand where they are in the, AI adoption journey, right?

Like, I think it's okay to move fast or faster. I, I think the goal is to move, always move fast, no doubt, right? But, but to your point, you know, you need to account for like risk tolerance and guardrails and so on and so forth. Observability, that, that does require some effort. So I would say first you wanna prove the value of AI before you, kind of invest in a lot of infrastructure.

good news is there's a lot of, observability tools, especially in the buy space, where, you know, you can get started in a matter of, you know, less than ten minutes. so it's, it's not something that, you know, you always have to build. It's something that down the road you may decide, okay, now I'm in-investing enough across the organization in the AI and, you know, I may w-want to build a custom solution here or custom integration even, 'cause a lot of these observability providers also provide like MCPs, so you don't have to, kind of you can build your internal workflows around observability as well.

So I would say again, kind of going back to your question, it's more of crawl, walk, run approach, where you want to prove the value of the AI and keep an eye on these kind of long-term investments at the same time. Things like observability, things like reliability, things like cost, cost controls, cost governance.

This is something, again, you probably don't have to worry on day one, but as you're starting-- start to adopt AI, uh, in more and more workflow, then you start seeing the value and you wanna kind of operationalize it across the organization at scale. Those are non-negotia- non-negotiables, be it observability, be it governance, be it security controls and guardrails and so forth.

Amir: Absolutely. Yeah, I, I guess something I was thinking about, o-obviously with the, the boom of every new technology comes, you know, the gap of trying to find people that can work with that technology. I mean, we, we've, we've seen it through various iterations. The cloud was a big one. when you're looking at the s- the skill set required to be successful in, AI platform observability work, what, what, what are you seeing as the gaps right now?

And as you said, there's a lot of buy tools, but, but still, I mean, you know, everyone has different experiences and exposures to what AI is defined. It's, it's a wide-ranging term right now.

Jeetendra: Yeah. Yeah, and I think if you look at the most successful companies out there, I think, to me, I think they have the really solid understanding of the foundations like how the AI like You just don't want to take the model on its face value.

Yeah, you know, I'm gonna send it a prompt, and I'm gonna get a, a response, and yes, I'm gonna evaluate it. But you really need to understand, like, things like, you know, memory and context engineering. So I think the skills that are highly valuable are, you know, what we are calling AI engineering skills, right?

So these vary all the way from, you know, having a solid foundational understanding of how large language models work. you know, what are the best practices in terms of like context engineering and context management? how do you build evals? That's a mindset that historically used to belong in the classic machine learning data scientist space.

But as an AI engineer, you need to develop some of that muscle to be able to kind of do evaluation, run experiments, be it offline experiments like with your test data set or be it online AB testing where you are kind of, testing features live with a certain percentage of your users to see, okay, I just launched this new prompt or, launched this new tool for my AI agent, is it performing better?

so I, I think there, there's a couple of things that come to my mind. One, I think we are seeing engineers, and I'll put that in air quotes because be it engineers or product managers or designers, I think we are seeing this cross-pollination of skills where AI allows you to cross those boundaries. And I think it's very important these days.

If you really wanna be successful in this space, I think you need to be able-- you need to be, willing to wear multiple hats, and AI allows you to do that. You don't have to be domain expert in every single field. as a engineer, you can now, like, if you've done back-end engineering all your life, you can start doing front-end with the help of AI, and that's, you know-- 'cause once you-- So engineering, I, I would-- other thing I would say is the engineering foundations are even more so important.

Like we hear all this, oh, is AI taking away all engineering, uh, like coding jobs? I would say again, you know, you probably heard people say over and over, coding was really not the job. It was a tool to get, get things done to solve problems. So those solid engineering founda-foundations like, you know, understanding, uh, overall architecture, understanding the trade-offs, even making the call, is this the right thing to build?

Because with AI you can build so fast the, the really, the judgment as we talk about, like human judgment becomes important. Is this the right thing to build? And then, yes, you wanna move fast with AI. the other thing I would say is where software engineering also matters is on the long-term maintenance of these systems, right?

Like it's one thing to just, you know, write ten thousand lines of code within a day. It's another thing to maintain it and evolve it and, you know, have right testing and make it production ready.

Amir: Absolutely. I, I guess a question for you. y- you know, we talk a lot about, you know, obviously productivity, we've talked about ROI, we've talked about even the new skill sets required.

F- for you, what, what you're seeing within the space, h- are, at what stage does AI observability come into projects, right? So i- is it moving earlier in the conversation? I- is it waiting to get to a certain, I guess, stage of development? I... When, when do you start engaging, in this conversation around the observability and, and the platform piece?

Jeetendra: I would say you wanna think about it from day one. it do-- Again, you don't have to have, you know, a, a fancy system or like, kind of advanced, sophisticated, observability system on day one, but you want to make it part of your thing. For example, when you're even prototyping and testing, an AI agent, let's say you're building for a new use case.

Let's say you wanna automate an internal business workflow around, let's say, data analysis and reporting. You wanna generate a report or a slide deck for your, CEO or your executive, and you are us-- like building a automated AI agent. You-- And even a simple skill, right? Like you want to be able to understand how that skill is working or how that workflow is working in order to have confidence in the actual deliverable.

So you do need to have, even at the time of development, I would say you do need to have some basic level of observability, so you can have that confidence and, um, verify the outcome. and if things do go wrong, again, observability shines because it allows you to kind of debug and reverse engineer, okay, what were the steps taken?

Was the context missing? Was the information that I provided to the agent out of date? did it call the wrong tool or did it pass the wrong, arguments? And so on and so forth. So I would say really, I, I think you're gonna be ha-- handicapped, is the word I was looking for. You're gonna be handicapped, if you don't have the right observability set up from day one.

So I would invest in observabili-- at least think about it and, and invest in basic observability tools. The good news is, again, there's plenty of free tools available. You don't have to, like there's free tool o-- free and open source tools available like Phoenix is one comes to my mind from Arize AI. Langfuse is another one which is, open source.

so you can start using these, free tools that require like on, on the order of minutes to set those up. A-and, yeah, then it, it also kind of makes it as you start scaling, you-- that's where again, you start to realize, the importance of observability e-e-even more so.

Amir: I, I guess maybe a final a-area to kinda dive into is, when we're talking about production, we're talking about scale.

I mean, we're still early, in seeing applications, deployed, and obviously we're still learning through some of these processes. when you are starting an AI project and you're thinking about scale How do you rightsize that? Like, what do you-- where do you start? Because obviously it will grow. you have to obviously, anticipate some capacity, but also models might change, which shifts again, some of those variables.

But how-- what, what do you think about scale and how do you have to think about that?

Jeetendra: Yeah. I would think about scale. yes, it's important, but again, I'd go back to proving the value first, that it's working, it's providing the value. This is where I think A/B tests are really handy, or you can even test out your feature with a small group of beta testers to prove product market value that the, the AI agent or AI workflow that you're building has real value.

and then, yes, you need to think about scale, both in terms of, like the AI, like the AI model that you're using, the cost. we talked about observability. We talked about like, you know, if you're gonna call tools, or rack pipelines if you're gonna provide context. So there's multiple facets.

and this is where, again, goes our, our earlier discussion around software engineering. this ties back to, kind of having that strong foundation in software engineering, which allows you to think about these, systems in, in terms of scale. So in terms of scale, I would say, again, start by focusing on product market b-- fit and product market value.

start with, a workflow that is relatively easy, easy, or, and low risk to automate. And then you wanna kind of give more agency to the agent and automate more and more. And eventually, yes, there's gonna be a time where you wanna scale this out to all of your organization or to, you know, all of your users.

That's where observability really matters. so that's how I would think, of it. it's, it's basically again, going back to like crawl, walk, run approach. and the good news is with AI, you can iterate and build POCs fast, and you can iterate really fast. So again, I would say, you know, focus on iterate, iterate, like iterating fast.

and then, yeah, keep, keep scale in mind. figure out, I would say a-as you-- once you prove product market value, the other thing you wanna know is what are the biggest bottlenecks. Is it the AI model? Is it, you know, like data, data pipelines that are providing context to the model? is it the tools?

So I think that's another angle to kind of look at from, is, is what are your biggest, bottlenecks as, as you scale this like ten X or twenty X.

Amir: Fantastic. Jitendra, I was gonna say thank you for coming on and, and sharing with us about, AI, observability. I think it's been fantastic. I'm, I'm sure, there might be questions, that have come up through this discussion.

Is, is there a good way of connecting with you to, ask you a follow-up question?

Jeetendra: Yes. I think the best way to reach out would be, through my LinkedIn profile. I'm, pay attention there. So, so yeah. please, you know, hit me up on LinkedIn, would love to hear if you have any thoughts, if you agree, disagree.

Again, we are all learning. I'd love to learn from, you know, folks if they have different thoughts. So yes, please do reach out if you have any feedback or any, any, anything that, I guess you wanna kind of learn more about. so yeah.

Amir: Absolutely. All right, fantastic. That's it for this episode. Be back again, different guest, different topic.

Until then, two things. One, I'd love to hear from you guys. I mean Jitendra as well. He said, I'd love to be in the discussion about AI observability, so share this episode with somebody that might benefit from it. He covered a lot of ground for us, and I think, some great points and insights, so that would be greatly appreciated.

Also, like, subscribe, comment, leave me a review on Apple Podcasts. I'd love to hear from you. Until next time, thank you and goodbye.