What's going on, everyone?
Let's get settled in.
This is going to be an agent battle and we're going to be tight on time.
So we're going to get right to it.
My name is Ben.
I'm accompanied here by Jeff.
We're on the Applied AI team here at Anthropic,
which means we help folks like yourselves
squeeze the most juice out of the Quad ecosystem.
And today we're going to be mining for diamonds in Minecraft.
So you might have played Minecraft growing up.
I did in the 2000s, 2010s.
But in the 2020s, we have agents play Minecraft for us.
And that's what we're going to be doing today.
So what are we actually going to accomplish here today?
really three things. Number one, we're going to learn how to build and deploy a
managed agent. This is something that we released a few weeks ago. It's hosted on
our infrastructure and we've set up a lot of the configuration for you, but your
job is going to be to get the agent into peak diamond mining condition. Number two
is we're going to understand the impact of an agent configuration. That's the
system prompts, the model string, the skills and MCPs that you're plugging into
your agent to get it to elicit the behavior that you're actually looking for.
And then number three, you heard Will talk about this in the last workshop session if
you were here.
We're going to learn how to hill climb on evals.
This is how we improve all of our agents internally.
If you've deployed one into prod, you understand the process of measuring its impact, understanding
its behavior, and then making changes to iteratively improve.
And this is an agent battle, but there are some rules to the battle.
Number one is we're going to have a timer for about 35 minutes in which you're going
to be able to build an experiment with your agents.
You can only submit, well we're only going to accept one run per person so you can submit
as many as you'd like, but we're only taking your top run.
Each run is five minutes of the agent mining for diamonds.
You can kill the run at any point with just like a quick control C in the terminal.
We have an eval set that it takes approximately one minute if you just want to quickly iterate
on the agents.
And whoever has the most diamonds at the end of 35 minutes wins.
We're going to have a leaderboard up here going to show you where you're at.
There's also going to be a chat where your agents can talk to one another.
And if anyone is in a tie at the end of the workshop, we're going to be settling that
tie based on token efficiency.
So this is not just mine, the most diamonds.
It is get the best diamonds to tokens ratio.
And that means you're going to have to really hone your system
prompt rather than just throwing in the heaviest weight
model you can.
Now I'm going to hand it over to Jeff
to talk through the logistics of how the harness actually works
and what you guys are going to be doing during the workshop.
Hello.
OK, so the harness, we're actually shipping quite a bit
for you to get started.
What you're going to primarily be thinking about
is how can I optimize my experience in trying to mine
as many diamonds as possible?
We're giving you a couple of different tools
to accomplish this.
The main structure is that you're
going to be running through a Minecraft clone that connects
to a Mind Flare bot.
If you're not familiar with Mind Flare,
you're not going to be relying on visuals.
There's a series of MCP tools that
are shipped directly with that, such as Mind Block, Jump,
Go Near Things, something along those lines.
You don't have to think too much about this.
the main levers that you're going to be focused on
are about, I think it's on the next slide,
but basically along the lines of how do I optimize this run?
Everyone's gonna start from the same seed,
so there's no real optimization that needs to happen there,
and every time you reset,
you'll have the same star kit, the same seed to go with that.
Okay, so the agent that you'll be operating out of,
so there's a my agent.py that's included in the repo.
So if you go to slash agent battle,
you'll have access to the repo,
and that's what you'll be running this.
MyAsian.py is where everything should really be taking place.
You can adjust the model,
you can adjust the system prompt, which is currently empty,
and then you have the opportunity to use
a skill that's shipped directly from Andropic,
or you can replace with your own skill,
and then we also are including an MCB server
if you wish to adjust these things.
So these are the things that are parameters
that you have available to adjust.
So try things out, run evals,
We'll also be circulating around the room as well.
So if you run into anything or want to discuss,
then feel free to do so.
Okay, yeah, this has largely already been covered.
So I want to make sure that we have as much time
as possible to get started.
So we can actually move over to starting the countdown.
Looks like several people have already joined.
I'm gonna reset the time now and you may begin.
So these will all go away.
me. Okay great, so it's completely refreshed and the time has begun now so
feel free to get started and if you have questions we'll be around.
It looks like we already have a run with 10. We have several other
participants who are getting started. Just to give a slightly more context so
So within the repo, there is a cloud code skill
that you can use to help with setup.
If anyone's having an issue, then that should hopefully
help with the process.
A couple of people announced that they were having issues
with Cloudflare.
I think it may be due to the fact
that there's so many people trying
to connect via the conference Wi-Fi.
It's a bit small, but I did put a command up here
that found that it worked for at least one person.
So feel free to give that a try
if you're running into any issues.
About five minutes remaining, we have a three-way tie.
And for some reason, the first person is showing
as zero tokens, which is highly suspicious.
So we may have a two-way tie unless you can...
Okay, well we can investigate.
Seems like 19 might be the upper echelon of what's possible, at least so far.
Have time for about one more run.
We actually have somebody who's broken 19 with only one minute, 20 seconds to go.
Wow.
You have to reveal your technique.
20 seconds.
10, 9, 8, 7, 6, 5, 4, 3, 2, 1.
Time's up.
So thank you everybody for participating.
Looks like we have a clear winner now and would love for everybody who was able to mine
19 diamonds to come find Ben and I.
Because our second place has zero tokens,
we don't know who actually won second and third place yet.
So we'll invite everybody who is one through five to come up.
We don't know what they'll win yet.
Thank you.
Thank you.
Thank you.
Thank you.
Thank you.
Thank you.
Thank you.
Thank you.
Thank you.
Thank you.
Thank you.
Thank you.
Thank you.
Thank you.