Thank you to everyone for participating in the discussion and for your overall support! As I said, I didn't expect this. I'm always just an email or tweet away, so you know how to reach me. It was great talking to you all!
Oh, definitely. I recommend you go for contracts. I've used something similar for a contract that iteratively "stitched together" a broken ontology graph. Here's some of the data models for inspiration -- you could have something similar for your ops, and write the contract to solve for one op, then apply the op, etc.
---
class Merge(LLMDataModel):
indexes: list[int] = Field(description="The indices of the clusters that are being merged.")
relations: list[SubClassRelation] = Field(
description="A list of superclass-subclass relations chosen from the existing two clusters in such a way that they merge."
)
@field_validator("indexes")
@classmethod
def is_binary(cls, v):
if len(v) != 2:
raise ValueError(
f"Binary op error: Invalid number of clusters: {len(v)}. The merge operation requires exactly two clusters."
)
return v
class Bridge(LLMDataModel):
indexes: list[int] = Field(description="The indices of the clusters that are being bridged.")
relations: list[SubClassRelation] = Field(
description="A list of new superclass-subclass relations used to bridge the two clusters from the ontology."
)
@field_validator("indexes")
@classmethod
def is_binary(cls, v):
if len(v) != 2:
raise ValueError(
f"Binary op error: Invalid number of clusters: {len(v)}. The merge operation requires exactly two clusters."
)
return v
class Prune(LLMDataModel):
indexes: list[int] = Field(description="The indices of the clusters that are being pruned.")
classes: list[str] = Field(description="A list of classes that are being pruned from the ontology.")
@field_validator("indexes")
@classmethod
def is_unary(cls, v):
if len(v) > 1:
raise ValueError(
f"Unary op error: Invalid number of clusters: {len(v)}. The prune operation requires exactly one cluster."
)
return v
class Operation(LLMDataModel):
type: Merge | Bridge | Prune = Field(description="The type of operation to perform.")
I'd argue it's all of them. Contracts simply make better agents. I believe it also gives a very nice bias on how to talk about agents -- as apps obeying contracts. If you find time, please read this blog post; it gives the underlying motivation for using contracts in agent design: https://futurisold.github.io/2025-03-01-dbc/
It's subjected to randomness. But you're ultimately in control of the LLMs's hyperparams -- temperature, top_p, and seed -- so, you get deterministic outputs if that's what you need. However, there are downsides to this kind of LLM deterministic tweaks because of the inherent autoregressive nature of the LLM.
For instance, with temperature 1 there *could be* a path that satisfies your instruction which otherwise gets missed. There's interesting work here at the intersection of generative grammars and LLMs, where you can cast the problem as an FSM/PA automaton such that you only sample from that grammar with the LLM (you use something like logits_bias to turn off unwanted tokens and keep only those that define the grammar). You can define grammars with libs like lark or parsimonious, and this was how people solved JSON format with LLMs -- JSON is a formal grammar.
Contracts alleviate some of this through post validation, *as long as* you find a way to semantically encode your deterministic constraint.
Yes, that's correct. If using say openai, then every semantic ops are API calls to openai. If you're hosting a local LLM via llama.cpp, then obviously there's no inference cost other than that of hosting the model.
We hear you. We might end up renaming it. In the paper we have a footnote about the name choice -- it's meant to credit the foundational work of Newell and Simon that inspired this project.
I'd appreciate it! It's cool and I wish you success. Just hope that when someone says "We're using Symbolic AI" a year from now, it won't be even more ambiguous than today :D
