Hello, everyone.
My name is Petra and today we are going to talk about teaching agents.
I'm heading up developer experience at warp and I'm sure a lot of you know warp.
You're pretty much the best place to run your agents and manage them and work with them.
So it's probably the terminal that you know.
You should check it out if you want to use CloudCon in there.
My team is responsible for telling the world how great warp is.
And we work a lot with the community, we create content.
And so today we are going to talk about how to teach agents through the story of how we
built Buzz, our social response helper little agent.
So this is Buzz.
Before I say a bit more about what Buzz does, a quick lay of the land.
I would like to ask all of you guys to put up your hand if you have built an agent before.
Right.
Obviously, that's a lot of people.
Now put up your hand if you're built an agent that is running on a weekly or daily basis
and is doing something, okay, already hands going up, a lot of you have done that.
And now put up your hand if that agent is still daily in production, shipped, and you're
very happy with the result.
That's a significantly less hands up.
So that gap is what we are going to talk about how to close today and help all of you guys
build better agents that actually work for you and do things for you on a daily basis.
So while we were building BOSS, we noticed, and I've seen from a lot of people that we
build agents, and we get to this point where a lot of you have put up your hand that you
have something that kind of sort of works and it kind of sort of does what you want to do,
but it's just not quite there for you to put it into production and just let it go and
and let it do its thing.
And that sort of 80% there, but just not quite,
is where I think a lot of agents die.
And it's sort of, end up almost worse
than if you would not have an agent,
because you end up spending time on tweaking it,
and prompting it, and spending time on trying to get it
tried, because you can kind of feel that it's almost there,
because it's not quite good enough to just let it run on its own.
And so that gap is what I would like to close today
through telling you about how we thought Buzz
to learn on its own, essentially.
So Buzz is our agent at Warp to help us respond
when users mention us.
It basically monitors our social mentions,
and it helps us figure out what to do with them.
Warp is a very popular product, and we
have a ton of people wanting to talk to us.
We have a lot of users who love Warp,
and they want to reach out to us.
They have questions, they have bug reports,
they want to learn about what features are coming up next,
and we want to talk to all of those people,
and we want to make sure that we can get back to them
and engage with them.
And so these replies are very important to us,
because they set the tone for the product,
for the company, for the community,
and all of these things are very important
for Warp as a company.
And so we're also start-up,
and we don't want to hire an army of people
to engage with this army of the community that we have.
And so we build Buzz, and Buzz monitors these mentions for us,
and helps figure out what to do as an action.
So it helps us decide between replying to something,
because someone had a question, or someone had a product
feedback item, or something along those lines.
Or it helps us just figure out that we should like something,
because it shows engagement with the user,
or we should just skip it, because it's not actually
about warp, or they don't expect a response,
they're talking to someone else. And if Buzz decides that we should respond to something,
it also helps us draft a message so that we don't start from scratch. And all of this
is a massive time save for us because we are a small team. And it allows us to focus on
the high ROI moments of engaging with the community. Buzz was built in a few days. It's
composed of about 15 or so skills, which you will also see in a little bit. There is essentially
to zero code written.
So all of these are skill files, and they build on these
agentic primitives that allow us all to build these really
cool agents that take things over from our place.
And it also connects to various services, from the X API to
Slack, and all of these things to be plugged into the rest of
the team and the context.
So the challenge that we faced when building Buzz is that
Something like figuring out social replies and what to do with them requires judgment and taste.
When to say something, what to say, how to say it, when to completely stay out of it because by
inserting yourself you're not actually achieving the goal that you have. And you probably also know
that you see all these replies on Twitter that are completely AI generated and you can notice those
from a mile away and you probably don't really want to engage with them because it kind of feels
is weird and it's very obviously AI generated.
So we wanted something that is better than that, but something that is not just not AI
generated, but that it really helps us engage with the community deeply because we care about
our users a lot.
And so the question was how do we build an agent that can do this well?
How do we build an agent that has good judgment and good taste to make these decisions, write
these replies make sense of what users are telling us about. And that gap is not just
we wanted it to be not just kind of okay, like you can put this on Twitter, but we wanted
it to be really good, like it actually sounds like it's coming from a human, and it actually
understands our product context and our company context to make good decisions about which
user mentions to engage with in what way and what is valuable for the user as well.
And so if we look at some of the approaches of how this is normally solved, we see a lot
of these agenetic loops. Agents are really, really good at figuring things out, basically,
and they are really good at going in loops when they can figure out if they are there
or not. And so, for example, you probably know the Ralph loop that you can see. And
why the Ralph loop works really well is because there is this external check that allows the
agent to know if it's there or not. It has a specific goal, and it can evaluate if it
has achieved that goal or not. Now, that works really great when it's about coding. It works
really great when it can run our unit test suite, and it works really great when it can
use, computer use and browser use, and it can check that, hey, I made this change, I
check it in the browser, I make an API call, I make a curl command. Is it doing the thing
that I wanted to do yet? Is it doing it? Yes. Great. Then I'm good. Is it not doing it
yet? Then, okay, what am I seeing? How can I iterate on it to make that achievement goal?
Now, the problem is that when an agent actually needs judgment and taste, and it's not that
have a unit test, it's very difficult to figure out how to do this loop, essentially.
Because you have something where with social replies, you would have to, for that external
check to work on its own for the agent, you would have to like sand a lot of replies live
and see what users are saying and how they are reacting, what the impact on the brand
perception is on the community is, and if it's like what other people are thinking when
they are reading those threads and those feedback loops are super complex and long.
And you can just set it up for the agent as an external check like you do with unit tests.
And so what we've, what the question here is how do we get that knowledge into that
agent?
How do we give the agent judgment and taste?
How do we teach it to do these fuzzy things that the humans really understand?
Well, we understand nuance and context.
When we look at a Twitter thread, we know that, hey, that was a pretty weird reply, or what
What should we say to help the user and how should we say it in a way that makes them
feel better about the product and also creates a good engagement?
And this applies to a lot of other things, from customer replies to helping you with
your own Slack messages to code review comments to all of these fuzzy things that we interact
with on a daily basis that need judgment and taste and are not as clear-cut as something
that you can unit test.
So when we build Buzz, we tried a few things.
We started, as I think a lot of you do, we're just trying to nail the prompt.
Seems like a pretty reasonable place to start at.
And usually you try to craft a prompt that encapsulates everything that you want the
agent to do.
And then you work with the agent to improve that prompt and you ask it to identify conflicting
information, you ask it to figure out what's ambiguous for it, and you ask it to identify
by gaps that it could fill, and then you work with the agent to improve that prompt.
I think a lot of us have done that, whether for an agent or just to nail out, plan out
what a feature implementation should look like.
What we saw with Buzzard is that the agent or the prompt ended up as pretty much a checklist.
It was just a list of rules of, like, if X happens, you should do Y. And the problem
with that was that it sounded like a robot because it couldn't really figure out what
to say and how to say, and it also broke the moment something new appeared, because it
just couldn't have that flexibility and it couldn't deal with that. The rules were just
too brittle. So we thought about how would we explain this to a new team member, because
working with agents is very similar as to trying to explain to a new team member how
and what they should do. And when it comes to that, you would probably explain to them
how to do things.
You wouldn't just give them rules
that if X-Y's what happens, you should do ABC.
You would explain to them how to think about things,
how to make good decisions,
how to reason about these situations,
what the purpose of engaging with the community is,
and you would give them guidelines like,
hey, you shouldn't get defensive
when users complain about the product
or they have an issue, you should be kind and empathic,
you should come across as a product builder
versus just someone who is processing support requests.
And you have all these principles,
and we switched our agent to work off of these principles.
So we ended up with, instead of these long list of rules,
we ended up with principles that really much better
encapsulated what we wanted the agent to do.
And so the result was pretty much that the skill file
was like a fifth of the original length
because you needed much less text,
much as lines of code, if I can put it that way in a scale file, to achieve the same or
even better result. And through these principles, because they were so much more flexible for
new situations, the agent could reason with them better so the output got better. So you
have a smaller scale file, you have better output, just because you switched from these
rules that the agent tends to do on its own, you switched from rules to principles. So
So that was the first thing that worked really well for us.
But there was still this gap that it wasn't quite as good as we wanted it to be.
It was like getting kind of there, but it wasn't like, oh yeah, this reply is amazing
and we're just going to send it live as is.
So the next thing we did as, I think, any good engineer, you start testing the thing,
you evaluate the results, and then you fix whatever you see.
I think it's a pretty usual loop of things as you build something.
And what this looked like in practice when we were working with the skill files and these
agents is that I collected a bunch of these responses that I wanted the agent to triage
and figure out what to do with, and it gave me whatever response it would have done or
whatever it suggested that we do, and I gave feedback on it.
I get feedback on, hey, this is good because of ABC.
This is not really good because of XYZ.
This is how I would do this differently.
This is how my reply would be.
This is why I wouldn't actually reply to this
because of XYZ.
And so this created this set of human feedback
on the agent generated output.
And I wanted the agent to learn from that.
Like, here is what you did.
Here's a bunch of feedback.
go and figure out how you can do better.
Now what the agent did was go back to these dangles
and we kind of know that that doesn't work.
So it had these principles and it started adding
these very specific rules like if a person is talking about,
is having some x-rays of problem with the product,
never mention pricing in the first line.
And it's like, sure it worked in that specific case,
but it's not something that is applicable
to most other use cases or most other situations.
A much better learning would be if someone is venting
about the product, don't try to pitch them
some other part of the product.
That is a much more flexible thing.
And so what we realized was that we had to teach
the agent to learn differently.
The agent sort of needed to learn how to learn.
And if we think about, again, how would we do this
with a new team member?
We would explain to them that we would sort of explain why something is better based on
our feedback, and we would ask them to take a look at what you did, take a look at what
I told you, take a look at your instructions, and what is the gap between the two?
What would your instructions need to be for you to have the same output that I gave you
as an ideal expected output.
And so we encapsulated this in another scale, and that worked actually really well.
I was really happy with how now we had two components.
Basically, we had principles which told the agent what to do, and they were very flexible,
and they applied well to new situations.
And we also had a way for the agent to learn, and so it could expand its own instructions.
Now, the next problem was who's going to keep teaching it, because it takes a bunch of time
to sit down and do all this back and forth and keep spending time on this, giving it
data, giving it feedback.
So I don't really want another team meeting.
I don't want another task assigned to someone on a rotation.
So how can we have the agent just learn from what the team is already doing?
And this was the last piece that really clicked everything together for us.
How do we have the smallest input from the team for the biggest output for the agent
to have a better result?
And so we designed a feedback loop that allows us to basically have almost no extra action
on our side on a daily basis and allow the agent to still learn from the team.
What this looks like in practice is that Buzz monitors these mentions and helps us figure
out what to do.
In the end, we still do everything manually when we actually interact with a user because
it's very important to us to maintain that authenticity and we actually care about the
user experience there.
But by Buzz triaging these mentions and helping us figure out what to do, helping us figure
out what not to care about, we save a lot of time.
And what happens in reality is, BUS monitors these mentions and sends us Slack messages.
We have a Slack channel.
We get a Slack ping.
Hey, there is this mention.
This is what you should do with it.
This is why.
So it also explains its thinking, which is really helpful for us to get in context quickly.
And then the team just monitors this channel as any other team channel and adds an emoji
reaction.
So basically, Boss says, hey, here's a mansion.
You should reply to it.
You should say something like this.
And then the team takes a look at that.
It's very easy and quick to skim through it.
We also leverage a lot of Slack's structured formatting
to make it easy to skim and make it the least effort
on the team.
And then the team adds an emoji reaction
with what action they actually took.
If they actually reply, they just add a check mark.
And through this, Buzz can take a look at what it suggested to do
versus what the team actually did and draw takeaways on what that gap was.
The team can also leave notes in slack threads.
I'm gonna show all of this in a few minutes on screenshots.
I think it's easier to see.
And so all of this allows Buzz to have a lot of contextual feedback
with extremely little extra effort from the team.
Because the team also uses these emoji reactions
to not step on each other's toes
and keep track of what was already handled and what wasn't.
And so this means that the sort of
branch crumbs that we leave for each other,
Buzz just simply learns from those
and draws takeaways on how to make its instructions better.
And then opens for requests.
All of these skills are in a Git repo
so that we can essentially handle them as code.
And then we just look at these for requests on a daily basis.
And the insight that really worked for us was, just keep it simple.
The trickiest part of designing feedback loops
is always the humans, because you have
to be very intentional about how you create some extra stuff
for humans so that they still do it.
If it's too complicated or if it takes too much time
or too out of the normal process,
they're just simply not going to do it.
And you want them to do it because the agent learns from it.
And so that was the first inside.
The other one was to make it feel like a teammate
because you just see humans interact more
and more meaningfully with agents that feel like teammates.
You can talk to them on Slack.
You can leave them notes.
You can leave them emoji reactions.
And it really helps the team give more valuable feedback
to the agent itself by just making it have a name
and have a little bit of personality
and a little bit of whimsy.
And so ultimately what closed the gap for us was this daily loop.
It allowed us to consistently make the agent better and allowed us to have consistently
better results and more time and effort saved on our side.
And all of these pieces need the others.
The principles are needed because the agent needs to know what to do.
the agent to learn is required because the agent needs to get better over time. And then
this day-to-day feedback loop is needed so that the agent actually gets the information
that it can use to improve itself. So let's take a look at what this looks like in practice.
This is one of Buzz's skill files. You can see this is just a normal GitHub UI. On the
On the left, you can see all the various skills.
As I mentioned, we have about 15, but consistently,
it always increases because we keep adding new stuff to Buzz
and it keeps doing new things.
But here you can see this is the warp reply skill.
This is what allows Buzz to draft messages
and it's an integral part of it,
making a decision on what to reply to
and what to skip and what to like.
And so here you can see the principles.
It doesn't say things like if person X mentions whatever feature you should say ABC or you
should link to this Docs page or something along those lines.
It has principles.
So it can take a look at the Twitter thread, let's say, and just understand what the user
needs and what they want and what they want to talk about and figure out what to say.
And as I mentioned, we have this Slack channel, and this is the channel that the team consistently
monitors.
You can see here the different actions that Buzz suggests that we do.
The top one here is a suggested reply with some reasoning of what the user is looking
for and a message that is drafted.
We didn't use this as the basis of what to say, but we usually sort of make it our own,
but it's immensely helpful to have the right tone, the right style, the right information,
the right content already there for us to start from.
It basically removes 90% of the effort and leaves us the 10% that is the most valuable
to give that reply, the most helpful for the user as well.
And then you see skipped replies here.
This is my favorite, honestly, because it means that we don't even have to care about
this.
We don't have to look at this.
We don't have to look at the Twitter thread.
Because we don't end up taking an action.
They're talking to someone else.
They're talking about something else.
And it makes no sense for us to insert ourselves.
And so we don't have to spend any time on these.
Or it just suggests that we like something.
If a user says that warp is cool, we don't necessarily need to say anything.
is very valuable when we like to tweet and show appreciation towards the user.
And then the feedback loop looks like this. So you can add these emoji reactions. And
what you can also do is add a note for buzz that it will pick up in this lecture itself.
So here you can see I gave some feedback to buzz that we shouldn't correct the user in
this case. They said something nice or they had a question about the product. We gave
them an answer of where they can find that feature. It makes zero sense to correct the
user about whatever context they asked this question in. And that feedback then gets picked
up by Buzz. It runs daily. It looks at all these emoji reaction differences. It looks
that all these threads and it draws these takeaways and it opens a pull request and
it sends a suspect message with that pull request. It explains what had changed in a
very brief way, it links to the PR. So we get all of this information pushed to us.
We don't pull anything. And so because it's also all in Slack, it's the same tool, it's
the same process that the team is already using, so it's extremely low friction. It
It makes it very easy for the team to interact with it.
And so, bots opens this pull request, links to it every morning.
We just click the link, it's like a 60-second PR review because it's just a few English
line changes and it makes a lot of, we have context about what it has done and we can
quickly see if that's a good change to make.
You can see the pull request here and it basically adjusted the instructions in the
relevant place.
just add a random rule at the end of a list. It looked at its own current instructions
and it adjusted them in the most appropriate way to not correct users when we shouldn't
correct users. And so then we can just take a look at this through this normal progress
review process and merge it in if everything looks good. I also actually personally really
like the feature where you can just make quick edits to the instructions so you have a little
bit of control over how exactly things are phrased. We also do this because we don't
want the agent to just change its instructions willy-nilly. We want to have some control
over it, not drifting into some weird direction that it keeps doubling down on. We want to
know what it's doing because it's important to us.
So what this looks like in practice in numbers is we have about a few thousand mentions a
month, 50% of those get skipped.
So as I mentioned, we save a ton of time by skipping these things.
And that's such a perfect task for bots or an agent to just take over, because we don't
end up spending any time on them.
It consists of 15 skills.
Some of this is triaging things.
Some of this is writing posts.
Some of this is about reporting, some of this is about analytics.
For example, on a daily basis, I get a DM from Buzz that shows me a bunch of graphs that
it generated about our distribution of the different actions that we take about who is
replying, how much, so that we have some health metrics on this part of what the team is responsible
for.
And all of this just really allows us to get more done as a small team.
We have a few thousand Cloud Agent runs per month, so all of this runs on its own in the
background in the Cloud.
We use OZ, which is Warp's orchestration platform for running Cloud Agents.
And it just runs on a schedule, runs on various triggers.
I think yesterday you all have seen routines on the Cloud Code, and it's a very similar
concept where API calls, web hooks, Chrome jobs, things like that trigger an agent to
just run in the cloud and it takes things off your plate so it gets triggered on its
own and you don't ever have to talk to it for it to get things done.
So all of these things are there to help you build agents that improve on their own.
What you basically need for this is to create and design the feedback loop that works for
you for your use case versus trying to nail the prompt from the get go.
And each of these pieces needs the others.
The principles are there so that the agent knows what it's doing.
Teaching it to learn is important so it can get better over time.
And then the feedback loop is important so it gets the input to be able to get better
over time and improve its instructions.
So if you remember one thing from this whole talk, if you recall just one thing that you
can take into into practice when you go back to your jobs and you build your own things,
is to focus on designing that feedback loop.
Think about how your agent will improve over time and not just how to nail that initial
prompt.
That initial prompt can be just good.
It doesn't need to be perfect, but you should try to figure out how you can create a feedback
loop that allows the agent to get better over time as your understanding of the problem
evolves, as new situations pop up, allow the agent to improve on its own versus you having
to go back and improve it manually.
So that's all.
Thank you all.
Go build great agents.
And thank you so much.
Thank you.
Thank you.
Thank you.
Thank you.
Thank you.
You