Hey, how's it going everyone?
I'm Dylan.
I work on the product team at AirOps.
And yeah, excited to walk you through how, you know, tell us how AirOps chases friction
with building AI products with Claude.
And I guess the main really big takeaway
I want you guys to come away with is building agents
and just making agents accessible
is honestly a really hard problem.
I guess like with developers, it's a bit easier.
People are used to kind of all these different concepts,
but when you try to make these accessible to, you know,
personas like marketers, there are a lot of friction points
and gonna talk through some of the friction points
that we have seen and battled with.
So, just to start off, quick intro of who we are at Aeropps.
We are a growth marketing platform for AI Search.
And AI Search for you guys is kind of like SEO, but for engines like Chashubit, Gemini,
Claude, you know, buyers are asking on Claude different, you know, questions of, hey, I
want to buy like these pairs of sunglasses, you know, how do you know, and how are you
making sure that you are showing up for these searches.
So we help brands see how they're appearing in search,
identify gaps, take action on those gaps,
whether that be creating content, refreshing content,
and then being able to measure the impact of whether or not
the actions that they're taking actually worked.
And so quick agenda, I'm just going
to walk through real quickly how we got here, kind of our
approach for agents.
A quick just run through of what we just launched last week
with AirOps Next.
And then two friction points that we really
focus and dial in on for our launch
when it comes to giving people the power of Claude
and making that super accessible.
And then closing with a couple of other friction points
that we're really looking at for our next ACT.
So how we got here.
AirOps used to be, or still is, but we were mainly focused
on orchestrating content through a traditional workflow
So node-based style, kind of like an NAN,
where you can drag and drop these different nodes.
You can orchestrate different variables
and how things flow through this elaborate workflow.
And with a workflow builder style,
and especially with our core customer
and audience being marketers,
you would hit this complexity ceiling
where you're trying to teach a content marketer
what liquid text is, what JSON is,
and all these different concepts.
And they also had this short shelf life,
which as new models came out,
Claude would release, Opus 4.6, 4.7, it keeps on going.
You would have to update these different steps.
It changes the way that you're building this workflow.
So customers are constantly having to go and update this workflow.
They just spent so much time in building.
And also just if they update a step in like step one,
they don't realize that variables and outputs
are referenced in step 20.
So there's a bunch of the complexity and it was really brutal.
And scaling enterprise use cases obviously then required someone technical to help really
guide through this workflow creating process.
So our goal was to really lower this barrier to entry for content marketers to build and
ship their ideas and the way they want to create content.
And also while still maintaining that quality bar, which is a very big focus for them because
of course brands don't want to just be pumping AI slop out and having governance around keeping
those enterprise standards.
So this is a quick, just preview of, not preview, but I look at what our Workflow Studio looked
like.
This use case was actually to create a content brief with internal links.
So as you can see, it gets kind of gnarly in there.
Just for something that does seem pretty simple, it just goes to show how much thought goes
into a lot of these workflows and the way that they want to create and orchestrate content.
So how can we take this kind of structure
and create an agent experience while still harnessing
the way that we use Cloud Code and these other agent tools?
So I think the first really breaking point
when we decided to invest heavier into agents
was really with the release of Opus 4.5.
I think that's when a lot of people
started to see really how smart the models were
and tool calling, being able to follow instructions,
in a way that they weren't breaking the standards
that they were setting for it.
And for us, we are at the same time
we're trying to make building easier.
And one of the ways that we do that,
and I'll dive more into it a bit later,
is through this document-based style builder.
And the first thing that we tried doing
was actually taking this Google Doc in a way,
and with an LLM, compiling a workflow in the background,
which was an interesting idea.
It was a bit brittle.
Like there's a lot of error points that can happen there,
trying to turn like this non-deterministic kind
of instructions and making this actual workflow.
Then one after that was then using a traditional agent
orchestration framework where you're kind of defining
these nodes, you're letting an LLM decide different decision
points.
And we got to a point where we were getting pretty good outputs.
But with those traditional frameworks,
they honestly are like pretty brutal.
If I ever want to change the way I want
to orchestrate different sub-agents,
I basically have to make code changes.
I actually have to go and change the way
I'm routing these different nodes and steps together.
And that's when we really decide to invest heavier
into the Cloud Agent SDK, where it's
obviously pretty awesome that you can orchestrate agents
just through markdown files and provide skills
in different contexts, really just
by manipulating this environment and harness
rather than actually doing that programmatically
through a traditional agent framework.
So that's kind of led to our launch,
which we just had last week with AirOps Next.
And briefly, just to run through,
what did we exactly launch?
The first one was Quill,
which was kind of our branding of our agent captain
for content marketers.
Quill had access to all the data that we provide to teams,
whether that be AI search data,
the brand context and brand kit, which houses
literally everything about a brand.
So Quill is following instructions
of how content should be created.
And just making it easier throughout the UI
to get people from insights they see in the dashboards
to actual actions, being able to take the findings and gaps
and get to that in the very next step.
The next one is Playbooks.
So Playbooks is basically our new building experience.
And for all you developers out there, it's pretty much
like a skill.
That's very much what we grounded it in,
is how can we make skills accessible?
And marketers are all used to the kind of document-based style.
But we just allowed for collaboration
on these playbooks slash skills and also governance
and versioning.
So you'll have people with 10 different versions
of this playbook of how do I want to create
this piece of content.
And real quick, just results that we've seen from customers.
We did case study with Parallel.
We helped them produce and create content.
And they saw 130% increase in citation rate, 42% increase
in share voice.
And they were able to go live in one week, which for us
is a huge accomplishment just because traditionally, especially
because we work with these enterprise customers,
it usually takes around a month, at least
with the workflow builder of constant feedback,
going back and forth, like, hey, this is not really
how I want to be speaking in my blog.
There's citations that are kind of being who we say or aren't
right.
These are other citations that I want
to use for this piece of content.
So it was really incredible to be able to get
to that acceptance criteria in such a short amount of time.
And other just like quick customer quotes, too.
Incredible to see how Agents has moved what really like LLMs
can do and how marketers are viewing using
these more objective workflows,
animals saying it felt more like
a mid-level strategist for their team.
And at Rippling, being able to focus on off-boarding
a lot of the tedious tasks,
and really focusing on where does my expertise come in,
and being able to add that value to the agent
while it's creating that content,
and feeding that unique context.
So we were able to really accomplish this
by focusing on two main friction points.
So when building and trying to bring high quality agents
to these more highly professional and enterprise use cases,
we really had to focus on how do we fit into the actual workflow
of these content marketers?
And how do we make sure that we ensure quality outputs
through the way that we're building our harness
and orchestrating our harness?
So the first one, first point I want to talk through,
is, I have it saying, endless use cases
forces intentionality.
And I think a lot of us have probably been there too
when we first started using Cloud Code or using an agent.
And we're like, this thing is really powerful.
I can do a ton of different things.
It's really easy to start sprawling into the spiral of,
yeah, there's so many different use cases.
And I think not only on the product engineering side,
but also for customers, how do we
be very intentional about what is the actual problem
that we're trying to solve here, and how do we really force it
and understand what is this workflow that we're trying
to solve for people.
So kind of like in my head now, I
have this mini Steve Ballmer head just going,
like, marketers, marketers, marketers, just really
trying to focus on who is my customer in this use case
that I want to tackle.
And just a quick glimpse into what
does this workflow look like for content marketer?
This is one example of a use case for content creation,
where they'll discover what do they want to cray on,
they'll research of this specific topic
that they want to make sure that they're ranking in AI search
or traditional search for,
they'll draft a brief during the article
and then add any sort of like internal linking
and best practices when it comes to SEO and AEO.
And throughout this whole process,
there are different human review points.
Human review comes up a ton
and human loop comes up a ton in content marketing,
especially just with when you're being cognizant
about the content that you're pushing out,
you really wanna make sure that it is adhering
to the way that your brand talks,
to the information that you're serving out on the internet,
and just giving you the best shot of also being surface
within kind of like generative AI search.
So that whole process kind of led us to focus on two parts
when it comes to the most,
I guess these are kind of like the two most important parts
we saw with the content marketing flow.
One is having a document-based kind of like IDE, which,
again, was like our playbook view.
We wanted, you know, marketers are super familiar
with documents.
They are used to Google Docs.
They've probably been working with Docs for years and ages,
even before technology, like you have this piece of paper
that you're kind of defining how you want to do something.
So making that familiar versus, you know,
this node-based workflow builder.
Transparency was also really important.
One thing that users actually really liked
about the workflow builder was that they could actually see
what tools were being used at each step.
So how do you still have a document?
But at the same time, when I read through,
I can understand, this is the exact tool
that I'm using at this instant.
This is the type of context I'm feeding at this point
and just being transparent around that.
And that kind of goes hand-in-hand with control as well.
How can I make sure that I'm still in control
of this seven instructions or workflow that I'm building,
especially now that it is really not as deterministic
because it is a document.
So those are three areas that we focused on.
And then again, on enforcing human review,
how can we bring governance, configurability,
accountability into an agentic workflow,
which I think is pretty unique for us.
And I don't think a lot of brands
that tackle that problem.
I think a lot of it is with agents, also with coding,
is kind of let it go.
It'll finish what it's done with.
And human review, in that instance, with coding,
is usually like with PR reviews, efforts
will literally finish the whole job.
It's rarely ever, while it's going through the actual coding
process, it'll kind of ask for feedback of taste
in different areas.
So that was another area that we really focused on.
And with that, I actually want to jump into a quick demo
of what this looks like within Aerobs.
So yeah, the first thing I want to cover is our playbook.
And again, it's kind of like this skill,
this natural language builder where I can go in
and either like type in like with a slash command
and define different inputs, different outputs,
and also all these different tools that
content marketers can use and that they're super used to using.
And when it comes to tools as well,
you can add any MCP.
So there's other outside connectors that you usually use.
You can use those and access them.
We also have the ability to schedule different triggers.
So this gives this kind of like always on skill
or playbook or agent that can do this certain action
at either a scheduled cadence based off web hooks.
We also have monitor, which we've kind of like partnered
with like parallel when it comes to just being able
to put a query and I like saying, you know, watch the internet in a way so when certain
things happen, it would trigger off this playbook to then run.
And then the last one is AO insights.
So whenever a metric drops, let's say like my citation rate dropped in the last week,
then it would trigger off one of these playbooks.
It can go and like do this research and come back to me of, hey, like this is the reason
why this happened.
And as I jump through into this kind of process of, you know, someone basically create this
SOP of how they want to create this blog, if I come to the outline section, we enforce
human review by adding the ability to assign different users at the end of each section.
So as, you know, the agent goes through this whole playbook, it will come to this section
and step and decide to, I think we have a tool
in the background that will then like fire off.
And basically, since I'm assigned,
I'm the only person that can actually unblock this agent.
Other people can still leave comments
on the outputs and artifacts and different feedback,
but I'm kind of like this gatekeeper now
that has to review this piece of content
and we're trying, we do that to ensure that governance.
And what that actually looks like
and how that surface to users,
one way is through our inbox.
So we have an inbox with an AirOps
where every single time human view happens
or different opportunities are surfaced,
users can actually come in here and see those directly.
So I can click on one of these items.
It will open up our, basically like agent runs
on the right side is basically this agent
like running through its whole process.
I can kind of see its thought traces on the left side,
all the different outputs and artifacts
that are one defined by me through that playbook.
And I can then either edit this existing document,
leave certain comments, and then kind of kick it off
and approve it.
The other way that we also surface this governance
in human review is through our grid.
So our grid is our way of orchestrating content,
or orchestrating basically content at scale.
So in this specific example, I'm just
kind of showing the way that you can actually collaborate
with these.
Augustine is also in this document with me.
I'm able to edit it, also leave human review.
If I close this out, I can see within the grid
all these different outputs that are running.
We're basically kind of like running skills at scale
where each one of these rows is a specific job
that I want to accomplish, and I'm running that playbook.
And I can also then see like human review at scale
and clicking these different cells to then leave
like my feedback and ensure that all this content's
talking in my right tone of voice.
Awesome.
The second friction point I want to talk about
was with agents, the biggest worry
is around consistency, especially when you're coming
from this workflow base.
So how did we tackle going about producing quality outputs?
How do we make sure that customers are also
seeing that and being aligned to that as well.
One of the ways that we visualized it,
actually, this graphic here was created and used
and referenced by our VP of Sales, which funny enough,
is actually one of the best explanations I feel
like I've seen of Harness Engineering.
On the product side, we've tried it a couple of times,
but it's just funny from the good market
side, so it was the best way that we've
been able to explain this, where we have this car, where you then
have an engine, the model obviously being
clod, opus, or sonnet, whatever engine
that you want to pick for that.
And everything else that goes around then,
everything else that you build on top of it,
is super important in terms of creating a super great agent.
So the two that I want to focus on,
and the one that we focused on the most,
was around tools and the way that we were orchestrating
context.
And the clod agent SDK and also clod managed agents API
have been super helpful in terms of being
able to iterate on this quickly and just making sure
that we're kind of programmatically setting
these different subagents.
And yeah, they've been super instrumental in terms
of us getting to a quality output.
So the first one I want to cover is around tools.
And basically, the background on tools
are you could give an agent a bunch of primitive tools.
So in this specific example, what we were always
trying to accomplish with Claw and our agent
was helping Claw to understand what is wrong with a page
on my website.
We have different tools, like access to traffic data,
citation data, scrapers, so I can find similar competitor pages.
And we started off, honestly, with a skill of, hey,
Here's the way of dissecting whether or not a page is losing,
what's wrong with a specific page,
are the schemas off, how to compare to your pages.
And it'll kind of go on like these safari trips,
and it's honestly a bit token inefficient.
So one thing that we want to focus on
is how can we create specialized tools of jobs
that Claw does that we know is going to do over and over again,
and how can we just make that a bit more deterministic,
where Claw can then put a URL and immediately spit out
everything about that page, as well as different structured
content gaps versus other similar pages in that space,
as well as what should be my target keywords
and my target prompts for it.
The second tool that we made in just a workflow
that we saw that was super common with content marketers
was our page versus tool, which was essentially
being able to benchmark my page against top ranking pages
in that space and finding what exactly is
behind those pages and how can I close those gaps?
So this is just a really simple way
of getting context efficiently.
It's kind of like a code mode in a way.
That's been something that's been popular nowadays
is being able to be more programmatic in terms
of how we're fetching context versus kind of looping
through these different tool calls.
It's like, can I actually just produce code that will fetch
exactly what I need in one loop?
The second one is through subagents.
Subagents have definitely been instrumental and crucial
in terms of getting to that quality of output.
In general, what we tell users to with Playbooks,
and honestly when you're first creating your agent harness,
they actually just start off with Claude itself
and just have it go through its own tool calls
and really not trying to make it too complex
in all the context that you're trying to give to it.
So that's where we start off with.
And we were reaching a couple of kind of like air spots
when it came to the quality of the outputs we're getting.
So what we did was decide to add on,
like over time, certain sub agents.
The first one was a compliance check.
So being able to just make sure that we're not like kind of
polluting that main context window,
just because context rock honestly is like still a huge
problem and will probably continue to be a problem when it
comes to like what are, you know, the tokens and text
that you're attending to.
So being able to spin off the sub agent that will go
go through and have everything that needs to know about my brand
and whether or not the content that I've produced
is following those rules.
They'll come back with a score, whether or not
it adhered to them, what were the things wrong with it,
and then the agent can then take that feedback
and then kind of make edits.
The second one we did was around writing.
We first, again, tried just using the regular Claude
harness to actually write the content.
But we found that it was better to spin off a sub-agent
with its own very focused context window
to focus solely on just writing that piece of content.
So it's not distracted by any of the research
that was created or any of maybe old compliance checks
and just being able to very focus in on that one job.
The next one is a brand kit subagent.
So again, like brand kit for us is basically
just like kind of like a knowledge base
or context layer of everything you need to know about a brand.
And this is something that we kick off actually
at the beginning of all of our runs
where we have a bracket sub agent that will go
and fetch all of the relevant context it needs
and then stores it as like an internal artifact.
So then throughout the whole process,
our main agent loop can then reference those artifacts
versus having to like use tools in our MCP
to refetch like that context.
Cause then what happens is in different sub agents,
you might actually have different brand contexts
that was fetched over time.
So just being able to fetch that upfront,
store as an artifact and then just redirect the agent
to always be referencing that same artifact.
And then finally, just still being able to add custom ones.
This is more for us like an internal tool
where some of our like solutions architect who
work for our customers can spin off different sub agents
when needed.
And just really helps with just maintaining that context.
I think that's definitely been the biggest learning,
is although like context windows do continue to grow.
You have like a million context window with Opus 47.
but just still being really cognizant that just because it is larger doesn't mean that
you should be using the whole thing.
You should still be very efficient of what sorts of contexts you're letting the model
attend to.
In terms of results, what we saw was actually a 8% decrease in fewer tokens consumption
around at least that specific tool.
So instead of, again, that tool spinning off different primitives and going over and over
Again, you actually just have this one tool call
that fetches a page and then all the results from it.
The second one, from that also, those specialized tools
will also speed.
Beforehand, while it was making those 20 different tool
calls to fetch all the concepts it needs,
you just have this one entry point that spits out
everything that you need.
And then when it comes to the quality side,
we had 10 enterprise customers, while we were running this beta,
actually start publishing content in under two weeks.
So they were able to self-serve and also get
to a quality of output through these agents, which
was something that used to be kind of like a hold-handing
experience.
It also had a really high kind of like ceiling.
So yeah, when it comes to building these agents
and with Cloud being able to execute and making it easier
to execute on things, it's really easy to think
that there are no more problems to solve.
But really every single time a problem is solved,
just that friction point always keeps moving.
So those are kind of the two that we're really chasing after
and we really want to tackle.
And there's a ton more that we also
want to continue chasing.
But I think just overall, it's a great thing
to just continue to chase friction,
because that really is how you create production agents
and make those more accessible to users
outside of more technical spaces.
So just to quickly close out, the next two kind of friction
points that we're looking into battle
and excited to share more learnings on
are around really self-improvement and feedback loops,
which there were awesome talks yesterday
around dreaming sequences and how,
I think the most interesting one is actually
how do you structure summaries of different traces?
What's the best way of collecting most relevant memories
and also forgetting is actually a feature,
like being able to forget certain types of memories.
And the last one is benchmarking content creation agents.
I think something really interesting about RSpace
is that it isn't law, it isn't coding,
it's not something that is really easy to say
whether or not it's something's correct.
There is a lot of taste that goes into a piece
of content that you create and there's a lot of opinions
of how I want certain contexts to be formatted and created.
So we're like the best ways that we are creating benchmarks
so that every single time we do change our harness,
we add a sub agent or we decide to add a skill to our harness,
how do we know that it's actually improving outputs
and not just kind of vibes that we're going after.
Yeah, I'm around all day,
would love to chat with you guys
and hear about what you guys are building,
but yeah, I hope this was helpful.
Thank you.
Thank you.
Thank you.
Thank you.
Thank you.
Thank you.
Thank you.
Thank you.
Thank you.
Thank you.