AI Workflow Orchestration, Engineering Productivity, and Hiring

00:00 | On this episode of the show, I have with me Viren Baraiya. 00:03 | He is co-founder and CTO of Orkes, and we're gonna be talking about AI workflow in orchestration. 00:09 | We're gonna talk about team usage, productivity. 00:12 | We're gonna talk about token economics as that is what's happening out there, and we'll ask Viren where he's seeing some of the productivity gains for his team, maybe something about hiring as well, given AI's impact on the hiring process itself. 00:26 | Got a lot to cover, Viren. 00:27 | Thanks for taking the time. 00:29 | Yeah, absolutely, and thanks for having me on your show. 00:32 | Orkes, what do you guys do? 00:34 | We are a platform for orchestration. 00:35 | That's kind of in general. 00:37 | We started out a few years ago based on the open source project that I had written at Netflix called Conductor. 00:46 | We brought Orkes as an enterprise version of the open source project. 00:51 | In a nutshell, we are a workflow engine. 00:53 | We allow developers to add durability to their code. 00:56 | So it's a durable execution platform. 00:59 | And lately, we also have been helping companies orchestrate their agents or build agents and orchestrate tool calling and other things. 01:07 | So when it comes to AI workflow, and we obviously could talk about orchestration, and obviously you guys are deep into that space, where is AI making an impact within orchestration itself? 01:21 | So in general, we always talk about AI workflows, right? 01:24 | Agents inherently are workflows at the end of the day, minus the fact that instead of a human building those workflows, an LLM is building that workflow, and it is essentially building that workflow as it continues. 01:42 | In the pure technical term, those are essentially late bound sagas where, just like saga as an orchestration pattern, the agents are essentially sagas except that they are bounded as they are progressing. 01:58 | At every step, every iteration, LLM is thinking about and reasoning about what to do next. 02:05 | It builds that and continues until it completes. 02:08 | One thing that has happened with AI is that orchestration has gotten into the front seat. 02:15 | When you are building applications and leveraging AI to build your harnesses, you have to orchestrate across agents. 02:25 | You have multi-agent scenarios. 02:28 | Your agents itself could be orchestrating different tools and services. 02:33 | There is also orchestration there. 02:35 | Even if you are combining a non-deterministic flow with LLM plus the deterministic aspect of workflows, there is still orchestration. 02:47 | So orchestration has become very critical to overall AI adoption. 03:02 | We hear a lot about how teams are starting to leverage different products within different spaces. Orchestration itself obviously has a lot of room to automate because there are repeated tasks. When you look at some of the best use cases for teams to automate, what do some of those look like? 03:25 | I think when you think about tasks which are purely non-deterministic and open-ended in nature, that's why we see a lot of proliferation of coding agents because at the end of the day, they are very non-deterministic. 03:38 | If I sit around today versus tomorrow to solve the same problem, chances are that I'm going to write different code. 03:48 | Those are some of the obvious ones. 03:51 | Second is a lot of things around productivity and one-off things where it doesn't make sense to build an application for that one-off scenario, but it's a perfectly great use case for an agent to automate that. 04:06 | A good example is if you are monitoring your cloud cost and every day you could have a different request. 04:14 | Today I want to know more about overall cloud cost. Tomorrow, maybe I'm focusing more on my databases. 04:19 | If I want to build an app, that app would look like a juggernaut of different screens and features. 04:28 | An agentic system helps quite a bit here, simplifies overall, and I can do a lot more complex things. 04:37 | The third one would be wherever you can improve human productivity. 04:41 | If there are repeated tasks that somebody's doing, applying judgment and context, those are perfect ones to be automated through LLMs, and you put them as a reviewer at the end of the day as a human in the loop. 04:52 | It overall improves productivity and removes the monotonous aspect of it. 04:57 | You mentioned productivity. When you're looking at teams that are automating the orchestration component of their workflow, where do they use that additional time? 05:19 | I can talk about what has happened to us. As a team, we have been invested very much into agentic coding practices. 05:29 | We write a lot of our code using AI. 05:33 | What happens when we have extra time available? One is we can ship features faster. 05:39 | Second is we have improved substantially our testing practices because we are no longer limited by how fast can we type and write the test. 05:47 | Now we can leverage LLMs to come up with more tests. 05:52 | We sit around, review those tests. 05:55 | Overall, creating systems to be a lot more resilient has become easier. 06:07 | Outside of engineering, things that otherwise would require us to prioritize and bring in additional consultants or services from other SaaS companies, we are able to build some of those things in-house. 06:25 | How do we, for example, monitor our cloud cost? 06:27 | We are starting to automate a lot of our customer engineering through both a combination of Orkes as well as AI. 06:38 | It allows us to be better at customer engineering, like can we beat our SLM rather than trying to think about can we meet our SLM? 06:42 | It has gained into internal company productivity, but also shifting those gains back into the customer's thing. 06:49 | We can be a lot more proactive in terms of catching issues and bugs. 06:55 | We can monitor a lot more stuff. 06:59 | We can do a lot of productive analysis to see what's going on there. 07:02 | Those things otherwise would have required a lot of investments. They have become much easier to do. 07:10 | When you're looking at how your team has been using these agentic tools, there is a lot being discussed right now about the cost of the tools themselves. How do you track against ROI? 07:44 | When it comes to engineering, we are looking at it in multiple angles. 07:54 | One is what's our velocity to ship product features? Are we getting faster? 08:00 | Second is, are we able to catch the bugs before they show up, therefore improving product stability? 08:18 | The third part is coming up with new features and product ideas that otherwise would have taken a lot more investment. Can we do that much quicker now? 08:28 | Three angles: new innovation, stability, and velocity. These are the three areas I would measure, and that's how we are measuring our investment and ROI into AI. 08:40 | When we're looking at the human in the loop, a lot has been made about engineers now responsible for managing multiple agents. How do people like managing agents? 09:19 | I think there's definitely an adjustment period because you are starting to look more like a tech lead rather than purely an IC now. 09:27 | You are no longer responsible for spitting out the code. You are more responsible for designing what you want to build. 09:49 | What is it that you want to build, how you want to build it, how you want to architect. 09:54 | In some sense, a lot of software development is becoming more like a waterfall. 09:59 | You spend a lot more time coming up with the specs, design, overall design, review with the agents, review with the team, bring that feedback. 10:10 | Then agent writes the code, and then you again come up with a test design. 10:14 | So a lot more verification. 10:16 | The role is shifting from being purely a programmer to becoming a designer and a verification engineer. 10:24 | That's where we are spending most of our time. 10:26 | That's where now you are coordinating across multiple agents. 10:32 | You have to change your practices. 10:36 | It's no longer iterative because there is a real risk of drift from the original design. 10:50 | You are relying on an inherently non-deterministic system. 10:59 | Depending upon how you're optimizing your token usage, you are shifting across different models. 11:07 | Now you are also becoming an orchestrator. 11:11 | You're orchestrating across different agents, what to do, how to do, when to switch gears from a low-cost model to high-cost model. 11:19 | How are you keeping your design decisions somewhere because that's important. 11:24 | Agents need to know why certain things have to be done this way so they don't forget. 11:38 | Talk to us a little bit about optimizing token usage and determining model level. 11:57 | I will split into two categories: there's a coding part and the non-coding part. 12:03 | On the engineering side, we tend to leverage models like Opus or high thinking models. 12:17 | When it comes to designing, addressing difficult problems, finding bugs which might be very hard to reproduce, things rooted in concurrency and parallelism, use that model to come up with design and implementation guide, and then let a smaller model like Sonnet implement those things. 12:41 | Then switch back to the high reasoning model to review the output and come up with feedback. 12:50 | It's orchestration between two models constantly happening. 12:56 | On the non-engineering side, if you are doing a repeated task, repeated task becomes workflow. 13:15 | Instead of asking an agent to review cloud cost for AWS for this set of services, the question becomes, build me a workflow to do this. 13:29 | Tomorrow I can rerun that workflow. I don't need to go to LLM. 13:32 | You pay cost one time and then reuse it. 13:40 | That has benefit longer term because there are only so many variations that you are going to ask questions, and then you can always reuse one of the existing workflows. 13:49 | We recommend customers think about it as a workflow and use AI and agents to build the workflow, and then that workflow becomes part of your tool stack that you can repeatedly use without burning tokens. 14:07 | When it comes to what you're seeing from your team and the way you ran projects, has anything changed in how projects are run now? 14:46 | The Scrum part has not changed. We still do Scrums, we still do sprints, except that what we could do in one sprint has changed. 14:55 | We can do more stuff in one sprint. 15:16 | That has become more like mini waterfalls. 15:19 | Every sprint has its own mini waterfall that builds on top of the next sprint and next sprint. 15:23 | At a macro level, nothing much has changed, but if you dive deeper into it, certain things have changed. 16:04 | Traditionally, if you have a two-week sprint, there is only so much you can do in a two-week sprint. 16:16 | If you add coding agents, within a sprint you can do a lot more. 16:22 | The risk becomes, if you are not on top of things and don't understand deeply what's going on, you end up shifting your entire pattern to vibe coding, which doesn't work for systems like ours. 16:38 | The approach we started taking was instead of treating things like an iterative design, let's treat that like a waterfall. 16:49 | At the same time, we don't want to spend months designing and building things. So break it down. 16:54 | This sprint, for example, you might focus on design and architecture for the new feature and some proof of concept. 17:04 | In the next sprint that carries over and continues to implement with basic tests. 17:15 | Everything we build is shippable in one way or another, either via feature flags or, if it is small enough, production ready. 17:26 | It has the entire cycle: design, implementation, tests, design, implementation, tests across every sprint. 17:34 | Therefore, this looks more like a mini waterfall than a very iterative design. 18:46 | One question about hiring. A year ago, eighteen months ago, the conversation was around making sure no one's using AI to game the hiring process. Now I'm starting to see a shift. We want to see what you prompt, here are agentic tools, let's see what you can do with it. Have you seen a shift? 19:34 | No, we shifted. We shifted a while ago. 19:37 | Interviews have to represent your day-to-day life. 19:41 | In day-to-day life, if you are saying, hey, you must use coding agents, then why are you assessing people differently in your interviews? 19:48 | You can't do zero-shot prompting and build a system. You have to work with it. 19:54 | If you think about yourself as a tech lead and coding agent as a mid to senior level engineer, how you interact with them is really important. 20:04 | One of the rounds we do as part of our interview process is agentic coding. 20:08 | Traditionally you would ask a candidate to implement an algorithm. 20:12 | Now you're asking them to implement the entire system because in 45 minutes you can implement a mini system very well. 20:18 | The process is, how do you go about working with an agent to come up with a spec? 20:24 | Are you writing a spec or just vibe coding it? 20:27 | Once you build it, how are you verifying that that's correct? 20:30 | The other part that is very critical now is what's the formal proof that you did exactly what you were supposed to do? 20:37 | If you are asking an agent to write a test, what's the guarantee the tests are correct? 20:51 | Then the focus becomes more on system design. 20:57 | How well are you able to design a system? 21:00 | If I give you 1,000 lines of code, can you read? 21:03 | If your coding agent is spitting out ten thousand lines of code in a day and if you are the one reviewing the code, can you read ten thousand lines in a day? 21:15 | Can you read somebody else's code? 21:17 | Can you reason about it? 21:18 | Can you understand what's going on there? 21:30 | That has become a more important skill than actually writing the code now. 22:43 | I have a slightly different opinion about that. 22:47 | If I think about why do we hire senior engineers or why they get paid more, it's not because they wrote a lot of code. 22:57 | It's because they have seen a lot of things in production. 23:00 | You have seen what fails, what doesn't work, what has caused those late hours and 24-hour cycles of debugging things in production. 23:09 | That experience is a lot more important. 23:11 | When you see code that has an obvious smell that this is going to not work in production, that's because you have dealt with that code in production and have been on call trying to resolve that issue. 23:29 | Every engineer in my team and everywhere where I worked, they all have their own perspectives what not to do. 23:35 | A lot of this is rooted in their own experiences in dealing with production issues. 23:57 | Those are the battle scars which give you that ability to understand that this is the wrong thing to do because this is going to fail in production. 24:17 | There is a lot more code being written. You can ship faster, and if you don't have enough guards, it will fail a lot more than it used to fail. 24:24 | Maybe we will learn more now and learn faster and know what not to do. We are still early. 24:50 | A lot has been made about junior and entry level people struggling to find jobs. Does that still fit with agents writing code? 25:47 | I think the short answer is you still need them. 25:51 | The responsibilities are changing, but the need is not. 25:58 | Before agentic coding, you would give a junior engineer small bugs to fix, maintenance work, and quality of life improvements rather than tackling bigger problems. 26:20 | If a coding agent is doing it, a senior engineer still has to babysit that coding agent. That can be best offered to a junior engineer. 26:42 | Somebody has to do it. 26:43 | Junior engineers can still do it, except now they are probably not writing that by hand. 26:47 | They are still orchestrating agents and gaining experience, and then as they go through the process of putting that into production and understanding systems, they are getting more senior. 26:57 | I don't think it is going to create issues for junior engineers to be able to get a job or find roles. 27:07 | Of course, you don't need bigger teams anymore, so overall the number of engineers that you need may be smaller, but that's across the board, not necessarily in one particular area. 28:26 | I think verification is going to be more important, being able to verify and test for correctness becomes more important. 28:33 | Back in the day, people used to hand code optimization. If you're writing in C or C++ or even to some extent Java, you would make sure that most systems are optimal. 28:47 | Today nobody does that. You don't need to. Compiler is smart enough. 28:51 | Does it mean that you don't need senior engineer? You still need senior engineer. It's that you don't need to think about those things. Somebody else is doing it for you. 29:43 | The part of this process that is probably undervalued at this point is human verification. 29:57 | That's going to be what's scarce because expertise is going to mean you'll know how to understand if the LLM has provided something correct, incorrect, context is wrong, the depth of the context is incorrect. 30:18 | I appreciate it. Thanks for coming on. 30:33 | I think LinkedIn is the best way to connect. 30:42 | That's it for this episode.