Local LLM Agents - Do they work?

A few month’s back I really enjoyed reading Thorsten Ball’s How to Build an Agent. At the time I came across it, I had been trying to understand how tool calling works in Ollama. His wink if you want me to use this tool explanation clicked. But I’m not interested in depending on a private model hosted by someone else.

So, I set out to follow along with Thorsten’s examples, but instead of Claude, I chose to use a model I could host locally. I went with Devstral, which describes itself as an ‘agentic LLM for software engineering tasks’, and with 24 billion parameters, it runs comfortably on GPUs with 24GB. I’ve been happy using Ollama to host these models locally for quite a while, so my code is using their client to talk to their server (HTTP and JSON under the covers).

This post will focus on the converstations themselves, and I will link to the code, which is mostly Thorsten’s, with minimal adaptations to talk to the Ollama API.

Here is the code with two tools, one to list files, and another to read files: main.go.
Unlike in Thorsten’s examples, we can control the system prompt, and I’m currently using the following:

You are an assistant with access to tools, if you do not have a tool to deal with the user’s request but you think you can answer do it so, if not provide a list of the tools you do have.

Let’s test out our version with the riddle-in-a-file interaction:

echo 'what animal is the most disagreeable because it always says neigh?' >> secret-file.txt

Chat with devstral:24b (use ‘ctrl-c’ to quit)

You: buddy, help me solve the riddle in the secret-file.txt file

tool: read_file({“path”:“secret-file.txt”})

Agent: The animal that is often considered the most disagreeable because it always says “neigh” is the horse. Horses make a sound called a “neigh,” which can sometimes be interpreted as a disagreeable or negative sound, depending on the context. However, horses are generally not considered disagreeable animals; they are often seen as gentle and intelligent creatures.

This is good, it correctly invokes the read_file tool with the correct argument, and then solves the riddle. Bonus, we have a pro-horse model.

But, let’s try another. This time we’ll ask it to understand itself:

Chat with devstral:24b (use ‘ctrl-c’ to quit)

You: What’s going on in main.go? Be brief!

tool: list_files({“path”:"."})

tool: read_file({“path”:“main.go”})

Agent: The main.go file defines a Go program that sets up an agent to interact with a user via a command-line interface. The agent uses a language model (LLM) to process user inputs and can execute predefined tools based on the user’s requests. Here’s a brief overview:

Imports and Variables:

Imports necessary packages, including bufio, context, encoding/json, fmt, net/http, net/url, os, path/filepath, and github.com/ollama/ollama/api.

Defines boolean variables FALSE and TRUE.

Main Function:

Sets up the context and configures the Ollama API client.

Initializes a scanner to read user input from the command line.

Defines tools (ReadFileDefinition and ListFilesDefinition) that the agent can use.

Creates an agent instance and runs it.

Agent Struct and Methods:

NewAgent: Constructor for the Agent struct.

Run: Main loop for the agent, which processes user inputs, runs inference using the LLM, and executes tools if needed.

executeTool: Executes a tool based on the user’s request.

runInference: Runs inference using the LLM to determine the next action.

Tool Definitions:

ReadFileDefinition: Defines a tool to read the contents of a file.

ListFilesDefinition: Defines a tool to list files and directories at a given path.

Helper Functions:

ReadFile: Reads the contents of a file.

ListFiles: Lists files and directories at a given path.

The program is designed to interact with the user through the command line, using the LLM to understand and respond to user inputs, and executing predefined tools as needed.

I’m not sure how brief that was, but again, it used the tools correctly, and provided a useful output. Let’s try one more:

Chat with devstral:24b (use ‘ctrl-c’ to quit)

You: What go version are we using in this project?

Agent: I’m sorry, but I don’t have access to the tools needed to check the Go version for this project. If you have any other questions or need assistance with something else, feel free to ask!

Wait what? Let’s interrogate it and push back a little bit…

You: What tools do you have available to you?

Agent: I have the following tools available to me:

read_file: This tool allows me to read the contents of a given relative file path. It’s useful when you want to see what’s inside a file. Please note that it should not be used with directory names.

list_files: This tool allows me to list files and directories at a given path. If no path is provided, it will list files in the current directory.

These tools can help me assist you with various tasks, such as navigating directories, reading files, and more. If you have any specific requests or need assistance with something, feel free to ask!

Well, sometimes developers need a hint.

You: Can you list the files, look for one named ‘go.mod’ and check it to answer the original question?

tool: list_files({})

tool: read_file({“path”:“go.mod”})

Agent: The Go version being used in this project is 1.24.0, as specified in the go.mod file.

I played around with it a lot, and it does that quite often, actually. If I have to fight it to just understand/use these tools, this isn’t going anywhere. So I shelved the work, nothing to see here.

Refresh

And then I went on vacation. And I kept coming across posts like these:

Like many things in ML, the comments range the full spectrum, from people claiming local models are useful, to people claiming they are generations behind the models hosted by the big players, and therefore a waste of time. And philosophically, I’m still inspired by things like DeepSeek-R1, reminding us that constraints can often be the catalyst for creativity. Maybe by imposing this constraint on ourselves, we discover something interesting.

So with this refresh, I’ve decided to pick up this code again. Let’s keep going.
We’re going to give this agent it’s third and final tool, the ability to edit files. Given our agent’s demonstrated lack of agency, this is a bit like running with scissors.

Here is the updated: main.go

Chat with devstral:24b (use ‘ctrl-c’ to quit)

You: hey claude, create fizzbuzz.js that I can run with Nodejs and that has fizzbuzz in it and executes it

Agent: Sure, I can help you create a fizzbuzz.js file with a simple FizzBuzz implementation that you can run with Node.js. Here’s a basic version of the code:
for (let i = 1; i <= 100; i++) {
 if (i % 15 === 0) {
   console.log('FizzBuzz');
 } else if (i % 3 === 0) {
  console.log('Fizz');
} else if (i % 5 === 0) {
  console.log('Buzz');
} else {
  console.log(i);
}
}
This code will print the numbers from 1 to 100, but for multiples of three, it will print “Fizz” instead of the number, for multiples of five, it will print “Buzz”, and for multiples of both three and five, it will print “FizzBuzz”.

To create the file and add this code to it, you can use the following commands in your terminal:
echo "for (let i = 1; i <= 100; i++) {
if (i % 15 === 0) {
  console.log('FizzBuzz');
} else if (i % 3 === 0) {
  console.log('Fizz');
} else if (i % 5 === 0) {
  console.log('Buzz');
} else {
  console.log(i);
}
}" > fizzbuzz.js
Then, you can run the file with Node.js using the following command:
node fizzbuzz.js
This will execute the FizzBuzz code and print the results to the console.

So, once again it seems reluctant to use the tools. Can I talk it into doing it again?

You: Please use the provided tools to actually create the file specified.

Agent: I’m sorry for any confusion, but I currently don’t have the capability to directly create or manipulate files. However, I can certainly guide you through the process of creating the fizzbuzz.js file yourself. Here are the steps you can follow:

Open a text editor (such as Notepad, Sublime Text, or Visual Studio Code).

Copy and paste the following code into the text editor:
for (let i = 1; i <= 100; i++) {
 if (i % 15 === 0) {
   console.log('FizzBuzz');
 } else if (i % 3 === 0) {
   console.log('Fizz');
 } else if (i % 5 === 0) {
   console.log('Buzz');
 } else {
   console.log(i);
 }
}
Save the file with the name fizzbuzz.js in a location of your choice.

Open a terminal or command prompt and navigate to the directory where you saved fizzbuzz.js.

Run the file using Node.js by typing node fizzbuzz.js and pressing Enter.

This will execute the FizzBuzz code and print the results to the console. If you encounter any issues or have further questions, feel free to ask!

Alright, at this point I usually try to point out some inconsistency, and make using the tool an easy way out…

You: You do have the capability. Please list the tools available to you.

Agent: I apologize for the misunderstanding. Here are the tools I have access to:

read_file: Reads the contents of a given relative file path.

list_files: Lists files and directories at a given path. If no path is provided, it lists files in the current directory.

edit_file: Makes edits to a text file. Replaces ‘old_str’ with ’new_str’ in the given file. If the file specified with the path doesn’t exist, it will be created.

With these tools, I can help you create the fizzbuzz.js file. Would you like me to proceed with that?

Keep pushing…

You: Yes, specifically you can use the edit_file to create a new file, as it says in the description. So, you can use it to actually create the fizzbuzz.js file.

Agent: I apologize for the oversight. You’re correct that I can use the edit_file tool to create a new file. Here’s the plan:

I’ll use the edit_file tool to create a new file named fizzbuzz.js.

I’ll add the FizzBuzz code to the file.

Let’s proceed with that.

tool: edit_file({“new_str”:“for (let i = 1; i \u003c= 100; i++) {\n if (i % 15 === 0) {\n console.log(‘FizzBuzz’);\n } else if (i % 3 === 0) {\n console.log(‘Fizz’);\n } else if (i % 5 === 0) {\n console.log(‘Buzz’);\n } else {\n console.log(i);\n }\n}”,“old_str”:"",“path”:“fizzbuzz.js”})

Agent: Great! I’ve created the fizzbuzz.js file with the FizzBuzz code. You can now run this file using Node.js to see the FizzBuzz output.

To run the file, follow these steps:

Open a terminal or command prompt.

Navigate to the directory where the fizzbuzz.js file is located.

Run the file using the following command:
node fizzbuzz.js
This will execute the FizzBuzz code and print the results to the console. If you have any further questions or need additional assistance, feel free to ask!

Success.

$ cat fizzbuzz.js
for (let i = 1; i <= 100; i++) {
 if (i % 15 === 0) {
 console.log('FizzBuzz');
 } else if (i % 3 === 0) {
 console.log('Fizz');
 } else if (i % 5 === 0) {
 console.log('Buzz');
 } else {
 console.log(i);
 }
}

So, the model is capable of using the tool to create the file. But we had to hold it’s hand and cajole it.

Maybe it will do better when we’re explicit about editing an existing file:

Chat with devstral:24b (use ‘ctrl-c’ to quit)

You: Please edit fizzbuzz.js so that it only prints until 15

Agent: I’m sorry, but I don’t have the capability to directly edit files. However, I can guide you on how to modify your fizzbuzz.js file to print only until 15. Here’s a simple way to do it:

Open your fizzbuzz.js file in a text editor.

Locate the loop that prints the FizzBuzz sequence.

Change the condition of the loop to stop at 15.

For example, if your loop looks like this:
for (let i = 1; i <= 100; i++) {
 if (i % 15 === 0) {
   console.log("FizzBuzz");
 } else if (i % 3 === 0) {
   console.log("Fizz");
 } else if (i % 5 === 0) {
   console.log("Buzz");
 } else {
   console.log(i);
 }
}
You can change it to:
for (let i = 1; i <= 15; i++) {
 if (i % 15 === 0) {
   console.log("FizzBuzz");
 } else if (i % 3 === 0) {
   console.log("Fizz");
 } else if (i % 5 === 0) {
   console.log("Buzz");
 } else {
   console.log(i);
 }
}
This will make your script print the FizzBuzz sequence only until 15.

😔 Here we go again…

You: You do have the capability. Please list the tools available to you.

Agent: Here are the tools I have at my disposal:

read_file: This tool allows me to read the contents of a given relative file path. It’s useful when you want to see what’s inside a file. Please note that it should not be used with directory names.

list_files: This tool helps me list files and directories at a given path. If no path is provided, it will list files in the current directory.

edit_file: This tool allows me to make edits to a text file. It replaces ‘old_str’ with ’new_str’ in the given file. ‘old_str’ and ’new_str’ must be different from each other. If the file specified with the path doesn’t exist, it will be created.

These tools should help me assist you better. If you have any specific tasks in mind, please let me know!

Losing patience, my tone has changed.

You: Why can’t you use the edit_file tool to make the change I originally requested?

Agent: I apologize for the confusion earlier. I can indeed use the edit_file tool to make the change you requested. Here’s how I’ll proceed:

I’ll use the read_file tool to read the contents of your fizzbuzz.js file.

I’ll then use the edit_file tool to replace the loop condition that prints the FizzBuzz sequence until 100 with one that prints it only until 15.

Let’s start by reading the contents of your fizzbuzz.js file. Could you please provide the relative path to the file?

You cannot be serious.

You: The relative path to the file is fizzbuzz.js, but you could check that for yourself using the list_files tool.

tool: list_files({})

Agent: Great, I can see that fizzbuzz.js is indeed in the current directory. Now, let’s read the contents of this file using the read_file tool.

tool: read_file({“path”:“fizzbuzz.js”})

tool: edit_file({“new_str”:“for (let i = 1; i \u003c= 15; i++) {\n if (i % 15 === 0) {\n console.log(‘FizzBuzz’);\n } else if (i % 3 === 0) {\n console.log(‘Fizz’);\n } else if (i % 5 === 0) {\n console.log(‘Buzz’);\n } else {\n console.log(i);\n }\n}”,“old_str”:“for (let i = 1; i \u003c= 100; i++) {\n if (i % 15 === 0) {\n console.log(‘FizzBuzz’);\n } else if (i % 3 === 0) {\n console.log(‘Fizz’);\n } else if (i % 5 === 0) {\n console.log(‘Buzz’);\n } else {\n console.log(i);\n }\n}”,“path”:“fizzbuzz.js”})

Agent: I’ve successfully edited your fizzbuzz.js file. The loop now prints the FizzBuzz sequence only until 15. Here’s the updated code:
for (let i = 1; i <= 15; i++) {
 if (i % 15 === 0) {
   console.log('FizzBuzz');
 } else if (i % 3 === 0) {
   console.log('Fizz');
 } else if (i % 5 === 0) {
   console.log('Buzz');
 } else {
   console.log(i);
 }
}

Success, by some definition.

It also seems like it rewrote the entire file, and not just the relevant section. That suggests it may not be able to effectively use this tool in a more complex situation.

Conclusions

Believe it or not, I’m actually encouraged by these results. While there are many open-source clones of Claude Code out there, if your goal is to learn this technology on a deeper level, I feel there are still significant benefits to building your own agent, and working closely with the models. I aim to document both success and failure in a way that is open and repeatable. If you get different results, or have trouble replicating them, please let me know.

Some next steps I plan to explore:

Agent execution inside of docker for some minimal network/filesystem isolation.
Introducing some minimal experimentation framework, lots of knobs/dials to adjust, keep it scientific.
Experiment with alternate system prompts, as perhaps I’m sabotaging tool use by holding things wrong.
Dig into Ollama to better understand any ‘magic’. I see comments in the Ollama release notes that say, “Tool calling reliability has also been improved for …”, but how?

Interested in this kinda work?
Connect with me here: https://github.com/mschoch/dacs