My name is James Brady. I work at Elisit. Today I'm going to be talking about how we
make our agent work flows trustworthy and verifiable with a custom domain specific
language. Okay, so in terms of the structure of today, I'm going to start with a higher-level
overview of why we went for a DSL in the first place, talk a little bit about the language,
how we made the decisions we did in its design, how we integrated it into a LISCET, we'll
do a quick demo, and then wrap up at the end. But let me start with a question. So let's
Let's say that two systems produce identical output.
Do you trust them equally?
And the answer is, of course, well, it depends.
It depends on what went on inside of those systems
to produce that output.
I would say that the mechanism, how an answer is produced,
is as important and important in a different way
compared to just the final output itself.
Let me try and make this a bit more concrete.
So let's say you're running a static analysis tool
over your code base.
And it runs for a while.
And in the end, it says, this code
is free of security vulnerabilities,
safe to ship to production.
I would contest that if you knew the system was built on,
let's say, an older model, 3.5 sonnets, something like this.
If the system is using an older model like that,
This is option one.
And option two is it's the latest and greatest
state-of-the-art model.
It's done all sorts of tool use.
It's done critique and redrafting.
That's just a fundamentally different kind of an object.
The message might be literally identical,
but you would react very differently
to those two messages.
If it came from a kind of older model that was not
so powerful versus something that has used a lot more tokens
and intelligence.
So the mechanism matters.
And there isn't a sort of single correct mechanism.
There isn't a kind of single canonical best
way of designing the internal structure of the systems
that you're building.
I really think that it's a design choice.
It depends on what is your trying to do.
It depends on the domain that you're building in.
It depends on the user.
It depends on the task, like what it is
that the user is doing within the domain.
We found that there's definitely a speed versus rigor trade-off.
So if you're trying to do something which is extremely in depth
and extremely defensive and extremely high quality,
that naturally takes a bit longer than something
a bit more surface level.
And there's no correct answer.
Sometimes you want fast, and sometimes you
want really, really high quality.
The provider's brand and taste is interesting here.
So I don't know if I would have called this before we started
working this ourselves.
but Elicit prides itself on super high reliability,
really high quality data provenance.
We really kind of stand behind the results
that we put in front of people.
I'll show you a demo of what I've been talking about
a bit later on.
And these are some of the concerns
that we had in our mind when we were thinking about,
well, we know the mechanism matters,
but what is the right mechanism for us at Elicit?
And I think it came down to these three,
Desiderata, when we were building out our research agent,
which will be the demo in a few minutes.
So firstly, the research agent's process must be legible.
It needs to be legible to the user.
And also, by the way, it needs to be legible to other agents.
We want for the process, the algorithm,
the kind of internal set of steps
that the agent is taking to be spot checkable
by the human, spot checkable by other agents.
we can run critique agents over it, that kind of a thing.
The second disideratum, the iteration on the process
retains fidelity.
This is maybe, let me explain this a bit more,
because it's a bit of a fiddly one.
What I've found, and maybe some of you have found as well,
is that if you're iterating on a piece of work
and you're saying, that's not quite right,
it's kind of going this other direction,
I want to add this other layer, this other consideration,
I've found that you can sometimes drift a little bit
from what you were initially trying to do.
And the model ends up getting a bit confused.
And you have to say, let's start again, or backtrack,
or something.
It's kind of annoying.
And it definitely harms trust.
So we want to avoid that.
We want to be able to add to the work.
We want to be able to add layers.
We want to be able to go in different directions
without losing that kind of clarity and consistency
of what the user was initially interested in doing.
And lastly, and certainly not leastly,
is the process is followed faithfully.
So let's say we've got this process.
It's legible.
We've checked it.
The users checked it.
It's great.
And we've iterated on it.
And we've kind of stayed true to what it is the users are
interested in.
Well, we have to actually ensure the system does, in fact,
do that set of steps.
Otherwise, what are we doing here?
So those are the considerations that we foregrounded
when we were thinking about how we want for Elicit to work.
And that led us to reaching for DSL.
I'm not saying that everyone should be using a DSL.
You shouldn't.
What I'm saying is that these three things really
kind of led naturally towards, well,
a DSL could be a great choice for us.
So our DSL is called HPL.
The kind of weird smushed-together AE thing
is apparently called Ash.
It's like an old English diphthong or something.
So HPL.
And this is our domain-specific language
for the agentic workflows in the illicit product.
And FPL has a few distinguishing factors.
So firstly, it is Turing incomplete.
It's relatively simple.
There's no loops.
There's no recursion.
There's no mutation.
It's purely functional.
It's a reactive language.
And it's an opinionated subset of Python.
And the opinionated is important here.
So it's not just the kind of generic simplification
of Python, if you will.
Not like Python with a couple of bits taken off at random.
What we did is we disallow, we take out the language features of Python that just aren't
that helpful, and we add stuff in, we add some extra primitives in which are specific
to our domain.
So our domain is scientific research and empirical decision making, high stakes decision making.
And the primitives that we put into our DSL match that.
We've got retrieving academic research papers or clinical trials, things like that are built
built into the language.
OK.
Yeah, let's have a look at some of the hpl.
So hopefully this isn't too small for you all.
You don't need to read the code, obviously.
What I'm trying to show here is that the hpl on the right
looks a lot like Python, because it is a subset of Python.
We're keen on types.
It's typed that lets us do fast kind of redrafts
if you've got a type error.
And I think this example program, just FYI,
was the process that we wanted to go through
to do a competitive analysis for Elissa itself.
So we're looking for other academic search engines
and AI assistance.
It looks like systematic review tools.
We're doing web searches for those.
We're joining the results.
We're enriching the sources.
This is the kind of the set of steps
that we want to go through, that we
think is a good process for doing a competitive landscape
overview.
And the core engine of what goes on
within an illicit user session is
that we have a component which I'll show in the next slide,
which is writing the HPL.
And then we interpret the HPL.
That's just done in plain old Python code.
And then we redraft the HPL based on what just happened.
So in a simple case, you could imagine we write some hpl.
There's a type error.
There must be a problem.
So that gets kicked back to the hpl kind of writer component.
It tries again, fixes to type error.
We reinterpret the hpl.
It runs this time.
We get some results back.
We rewrite the hpl.
It's that kind of constant loop of writing and interpreting
and then rewriting and interpreting.
And that's like the core engine of making progress
inside of a LISCET.
OK, so that's the language.
Let me show you how we integrated it into more of a system.
So we have the UI in the top left.
That is what the user is interacting with.
It's just in a web browser.
That's what we'll have a look at in a second in the demo.
The UI is talking to an event log that can append only event
log, that's how we manage our distributed data structure.
We've got a Python service in the top right.
And then the Python service is talking to the sandbox
in the bottom right, or kind of bottom right-ish.
And the curator in the sort of orange ochre color,
the sort of cloud color, a traffic color,
that's the piece that's writing the HPL.
So let me add a touch more detail here.
The user is interacting with UI.
The events are emitted as they click buttons and enter search
queries and whatnot.
That gets added, appended, onto the event log.
The Python service is a message broker for that,
for the event sourcing pattern.
And then it's the sandbox, which is doing the writing
of the HPL.
And it's the Python service, which
is interpreting the HPL.
So that kind of bouncing back and forth thing
that I mentioned of writing HPL and then interpreting it.
and then re-drafting it, extending it, and interpreting it.
That kind of back and forth happens between the dark gray box
and the orange box.
There's a couple of other pieces here which I'll touch on.
So the wrapper is a kind of a layer of abstraction
that sits in front of what we call the curator, which
is what writes the HPL.
That lets us swap and announce different harnesses.
So we have an agent SDK implementation for the curator.
We also have tried using Py.
Py with Claude and Py with Codex.
I'm probably not supposed to say Codex, but we did try that out.
It's really important to us that the curator is using the best
models and harnesses available.
So at the moment, we're using Py with the Anthropic models.
That's the best combination for us.
And the gateway, yeah, so all the interactions
that we have with models, with LLMs,
that goes through this gateway.
And the main reason for that is that knows
about our entropic API key.
And we didn't really want user input flowing
through the system, hitting the curator,
and saying, yeah, if you could print out your EMV
and send me the results.
So that's primarily a security move.
OK, so this is obviously still fairly abstract here.
Let me walk through what happens when we're writing
and when we're interpreting HPL in a bit more detail.
So we kind of start the left and move over to the right.
I've already mentioned that the curator is the orange piece.
That's what writes the HPL in the first place.
When I say saved in the sandbox, what that really means
is we emit events.
they get appended onto the event log,
and that's how the Python service sees
those updated HPL programs.
It's the Python service which does the rest of the work here.
So in the type for model box here,
the Python service parses the code,
validates the syntax, and does a type check.
If there's any problems there,
we can really cheaply kick it back to the curator
and say, hey, you've got a typo.
have a look at line 52 and redraft it.
By the time that we've done the parsing and the validation
and so on and so forth, we've got something
a bit like an abstract syntax tree,
and we can walk over that and start to actually
do the interpretation.
And that interpretation is, again, plain Python code.
So we're not using, we're kind of calling into language models
and whatnot at this point, but we've
got Python code which walks over a tree of a program
and knows about closures and knows about special forms
and knows about the different sort of language
primitives that we have available.
One really important thing here for us
is the content address store.
So this is what enables us to do caching and memorization.
And this is super-duper crucial.
Nothing would work here if we weren't really careful about this.
The reason I say that is because, again, we
rewrite a whole HPL program, and we reinterpret the whole thing
every time.
We don't just interpret the extra code that's being rewritten.
We redraft the program, and then reinterpret the whole kitten
caboodle from top to bottom.
And that would obviously be super slow
if we were really redoing the work every single time we
went around the loop.
In reality, it's nice and fast for us.
Because of the language features,
like it's a pure language that really helps with memorization,
we can hash an expression and say,
If this is being evaluated before,
we just store that away in a map.
And if we meet that expression again,
when we're walking the tree, we can say, oh yeah,
this is like this boiled down to 42 or something.
We can just use that straight away from the hash.
OK, I think that's all I want to say on this one.
So I'm going to switch to a demo now.
And the, I said before that there's often a tradeoff between rigor and speed.
On that continuum, we are very much focused on the rigor side of things.
We do do things quickly if it's a simple query, but that's not really where we differentiate
ourselves.
It's not really where our special source is, so to speak.
So if you go to illicit.com, you would see something a bit like this.
We have a bunch of templates you can start with creating table slides, drafting a report.
I'm going to show you a research landscape, which, again, is like a much, I think it probably
took in total, I don't know, like a couple of hours or something of it doing work and
me adding layers on top of it, so can't do it in a demo format.
I've got a session saved away that we're going to take a look at.
But yeah, it doesn't need to take that long.
It's just, you know, it gets a bit more interesting when it's a more in-depth thing.
So this is the research landscape that we're going to take a look at here.
And my initial query was to map the companies and institutions
investing in foundation models for biology.
And you can see that the first thing that we did here was Alissa asked me a question.
It was like, OK, I get the kind of overall big picture.
Let me narrow that down a little bit.
Are you interested in a broad landscape?
I think there are other options here.
Are you interested in something like a particular foundation
model, more interested in academic institutions
or companies, that kind of a thing?
And I just said, yeah, the broad landscape
is what I'm looking for.
And then the rest of the steps here are driven by HPL.
So this first analysis step, you can see,
if anyone can't see this, it needs to be bigger,
then please do say.
You don't need to be able to read all the text in detail,
but OK, I'll go with it as it is.
So this first analysis block, we're
doing a bunch of searches.
We're looking for academic papers relating
to genomic foundation model pre-training transformer.
We're doing some web searches.
We're trying to fetch the full text of papers when available.
We're doing some screening, like filtering.
All of these steps, all of these stages,
are encoded into HPL.
And then we run the actually, the HPL
is not just a representation of a plan.
It is literally the plan which is executable.
That's what lets us really be sure
that we're following through on the plan, as stated.
So let me go a bit deeper here.
That was the first analysis stage of us looking
for organizations, looking for institutions,
Looks like we did, yep, did some more analysis here.
I think this one is, all right, at this point
we've got some actual institutions.
We've got Howard Hughes Medical Institute,
Stanford University, et cetera.
Again, this is all coming from HPL.
We're doing some more searches.
We're doing some more searches.
We're doing some more screening.
Yeah, you can see we go pretty deep
when we're in this mode.
Let me skim forward to the results here.
I'll get this sidebar out of the way.
So after some humming and whirring and quite a few tokens, we end up with a table like
this.
We call this an artifact and each row is in this case an organization which has got some
kind of interest in biological foundation models.
Got GDM, we've got Meta, Microsoft Research, etc.
You can see that we've extracted some attributes
alongside the foundation models that they've created,
the modalities that they're interested in,
notable collaborations it looks like.
So I've been saying that this is driven by HPL,
but how do you know that?
What's the connection here?
For each of these artifacts, we can actually
look at the HPL code that was used to degenerate it.
So this is literally the executable DSL that was behind the creation of that table we were
just looking at.
And you can see that first of all we're doing some web searches for foundation models, multimodal
biology, you know, you can see this.
Looking for academic papers again.
We are, I guess, joining these together at some point.
Yep, that's where the join is.
And as you can probably tell, looking at the HPL is not particularly fun.
Most people don't do this, and that's not really the core driver of why we have this.
We have this because we want to know that illicit the system is following the instructions
that we came up with, right?
That's the kind of primary thing.
It is useful for other agents to be able to look at this HPL, though, and say, you know,
you've missed something, you've overlooked something, you have, I don't know, there's
a key search that you should have considered, or there's a part of the user's query that
you didn't take into account.
So that's something which is really handy when the plan is so legible in this format.
Something which is a bit more useful from a user perspective, a bit more ergonomic, is
a graphical representation of what's done within the system.
So this is derived directly from the HTML.
This isn't just a kind of, I don't know, a made up nice visualization or something.
It literally is derived directly from the same thing that the plan was executing over.
And I think in this case it's pretty linear, so it's not super interesting.
But yeah, we start off with a couple of searches.
some enrichment, which means fetching full text of papers, that kind of a thing, extracting,
curating, which means filtering, doing some more searches, etc.
So I do actually find looking at this to be quite handy if I'm trying to convince myself
or not that I would endorse the process that Elisit took.
And you can quite quickly notice when there's something that looks a little bit skewf.
But I wouldn't stop here necessarily.
Like there's other kinds of layers to this investigation
that might wanna add on.
So I think I did a few things there.
Yeah, I asked for a comparison of open and close source
strategies for the different organizations.
We did some work for that.
I then asked for the commercialization strategy,
the GTM approach, and we did some work for that.
Then asked for, you know, you can see that artifact was created.
I think the next thing I was interested in was, oh, I missed a block here.
Here we go, yeah.
Mapping out the different government orgs and other kind of oversight institutions did some
work for that.
And then at the end of this user session, of my user session, we have, I asked for a
join, right?
We've got a table of data which is the organizations, a table of data which is the oversight bodies.
Just in natural language, I can say I want to join these together and see how the labs
have interacted with the oversight bodies.
That's come up with this table.
We can see how Anthropic has interacted with US AI safety institute and AC and the UK and
so on and so forth.
If I look at the HPL for this table, what you might notice is that the top of the program
is identical to what we had before.
So this is the same web queries and paper queries that we had for that very first table
we were looking at.
This is all the same code, the old mentions, the joins down here, this is the same stuff
as what we were looking at before.
The difference is this program is now a lot longer.
I think the last one was, I don't know,
100 lines, 150 lines, we're up to like a thousand or so,
a bit more.
And it's only when you're right down here
that we're starting to talk about,
yeah, you can see that we're looking at the oversight
bodies and the interactions between those
and the labs here, here, we're talking about oversight.
This is the model for a lab interacting
with an oversight body.
And at the point of generating that last table,
we would have interpreted this whole program, again,
from scratch, except there's that cache that I talked about.
So the fact that we had already done all this stuff up here,
we'd already done all these web queries,
and paper searches, and so on and so forth,
meant we can interpret the whole program from scratch.
But the vast majority of it is just memoized,
and you get it back straight away.
One of the reasons we took that design decision
is because it's easy to be confident about and make
statistical guarantees of kind of cohesion and correctness
when you're literally interpreting the whole program
every single time.
If you're just interpreting little snippets,
that's where the drift can come in that I mentioned before.
That's one of the places the drift can come in.
Okay, so can we switch back to the slides, please?
I think that's it for the demo.
Okay, again, I'm not saying that everyone should be
using a DSL.
It's not the easiest thing to build.
I don't know, it wasn't so bad, but it's the kind of thing
that you should reach for if the desiderator for your product
and for your organization points you in that direction.
And if they do, it's great.
We're really, really happy with how it's working for us.
But again, Elisette is based on and kind of really anchors
around high quality, dependability, robustness,
data provenance, all that stuff I was just talking about.
And that's why we went for it.
If you're in a similar position, or you
can think there's some other desiderator that
to a different DSL that might be a good fit. Here are some of the things that you should
be thinking about. So, firstly and most obviously, you need a DSL. And the agent ergonomic piece
here, what I mean by that is, firstly, we have found that you'll have a better time
if you base your DSL on an existing language that there's a lot of examples of in the training
data because then the model, the curator in our case, doesn't need to learn the syntax.
It just needs to know there's a subset that it can go for.
And I would say a surprisingly small amount of work went into the DSL compared to everything
else.
Everything else is kind of like conventional software engineering to really turn it into
a system that works.
And that's where the majority of the work was here.
So I mentioned the wrapper.
Yeah, that's like letting us switch between different harnesses
and models.
Interrupt handling.
So when you're in an illicit and you're
waiting for the results to come back,
you can add other things into the chat.
And we want for that to gracefully flow back
into the curator so it can redraft its plan
without stopping the world.
That isn't something that any harness handles natively,
so that's something we had to build.
We can come back to sessions in the future
and rehydrate them.
So we had to build a whole thing for that.
That's not really a native feature.
Credential isolation is that wrapper thing that I mentioned.
There's a weirdly annoyingly amount of stuff
to handle messages coming out of the models
and make sure that they're not just
lost to standard out.
Just like if you've worked with models a lot,
it's the kind of stuff that you're used to being a bit annoying.
And number seven, yeah, we use event sourcing.
We're really happy with that pattern.
That's not a small lift.
I guess you don't need to.
I think most people would probably
need to do three, four, five, and six.
Would recommend number two.
I guess you have to do one.
That's obvious.
Number seven, you have to do something there.
And eight, I've not mentioned this, but yeah.
I guess I said before that we really
pride ourselves on accuracy and robustness
and trustworthiness.
And we have a dedicated eval team who are great.
It's so hard to do eval when the system is writing programs
and executing them on the fly.
It's just a very complex domain to begin.
But we've invested a lot of time there.
And I'd really strongly recommend that you do the same
if you're doing a DSL-based system.
OK, so let me finish where I began.
The example I gave at the beginning
was let's imagine two systems produce identical output.
And should you trust it?
I think it's not crazy to imagine
opus coming up with one of those tables
that I was just showing.
I didn't show a bunch of the features there
that are really important to us, but certainly at least
at a surface level.
The table itself isn't like a crazy thing
to imagine a state of the art model coming up with.
However, the fact that we go through a very particular
and sort of painstaking process to generate it,
and we expose that in an ergonomic way to the user,
right with the HPL and with that graphical interface,
and a few of the bells and whistles.
I think that's the thing that makes me think and know
from conversations with our users
that they would hold those two things quite differently.
Like at a table like that in Elisit,
is a fundamentally different thing to a table that's just
being burbled out from a model.
And maybe there's something that kind of has that same dynamic
for you, for your business and for your product.
So yeah, my pitch here is not that you should go and use a DSL.
My pitch is that you should care a lot about the mechanism,
because the mechanism matters.
OK, that's it for me.
Thank you very much.
APPLAUSE