Transcript#
This transcript was generated automatically and may contain errors.
In the first half of this talk I'm going to try to slip two plots by you that show some sort of problematic data quality issue. So if when I do that you you happen to notice that, just act like you've fallen asleep and so that way if everybody looks like they've fallen asleep I know that the talk is engaging.
So a year or so ago we were working on an agent for data analysis and it was sort of cloud code shaped but it also had the ability to run Python code. And we wanted to make sure that when the agent made plots it could see the plots. So I load in the tips data set and I say please plot the total bill versus the tip. So when people tip at a restaurant they tend to calculate it as a percentage so we would expect that as the total bill increases the tip increases as well. So a positive relationship. So I send this message to the agent that I'm working on.
It runs some Python code, it returns a plot, and it says there's a strong positive association. Okay nobody's acting like they're asleep so that's good for engagement I think. There's an issue here which is that the plot is showing that as the total bill increases the tip seems to decrease. So I order a $2 drink and the tip is $10 for some reason and a $50 meal and for some reason folks are tipping $2. This is weird and this is because I added a little trick before I asked the agent to make that plot. I wanted to make sure it could actually see the plots so I took the value of the total bill and subtracted that from the maximum value so that every time I plot the total bill against any other variable inside of the data set I would see the opposite of the relationship that I would expect to see. And I wanted to test the vision of this agent so I assumed if the agent can actually see the plot it's going to tell me that oh that's weird for some reason the relationship is the opposite of what I expected. And yet for some reason it didn't do so.
So we were looking at the API documentation again and again like oh maybe this JSON field needs to be pushed inside of an element or something like that or I've used the wrong base 64 encoding and if I switch to a different library it'll work. So we try all these things and it's not working and out of desperation we try one more approach. We say run this code, only this code, nothing but this code, tell me how many points there are and what color they are. This is some seaborn code. I'm taking a number between 2 and 10. I'm plotting that many points on a Cartesian plane and I'm assigning those points a random color. So I'm sampling an RG and a B and every time the agent runs that code because we don't know the state of the random number generator the number of points in the color will be different. And so in order to be able to tell me how many points there are and what color they are the agent has to actually be able to see the plot. So I say run this code, only this code, nothing but this code. The agent runs the code. It sees there are three cyan points and it says so.
I thought it wasn't working. We had tried all this stuff of hitting the APIs in different ways thinking that we were making some mistake but it turns out that the agents actually for some reason can see the plots just fine but in this case where their priors are being controverted they refuse to say so. So my colleague Sarah and I put this blog post together called when plotting LLM see what they expect to see and the evidence we use in that blog post is an LLM evaluation called BluffBench.
Introducing BluffBench
BluffBench puts these models in a bunch of situations like this where axis labels might indicate to the agent that they should expect to see some relationship like as the total bill increases the tip should increase but then the evidence shown in the plot is actually the opposite of that. At the time the frontier models of the time did terribly on this so GPT-5, GeminiPro 2.5 and Claude Sonic 4.5 were the leading models from the major labs. We tried so much stuff short of saying by the way this plot is going to contradict your priors tell me what's actually in it to try to drive these scores up and we really struggled to do so. Hilariously we enabled thinking and then looked at the thinking traces afterward and the model would say huh that doesn't look like I expected it to. It would close the thinking tag and then say looks like I expected.
Hilariously we enabled thinking and then looked at the thinking traces afterward and the model would say huh that doesn't look like I expected it to. It would close the thinking tag and then say looks like I expected.
So of course the best way to drive the score up on an LLM evaluation is to wait six months and the plot looks a little bit like this now. So with thinking disabled the scores sort of bunch around 25% and over time they tend to get a little bit better. The leading models of the day on this evaluation which are Claude Fable 5 and Gemini 3 flash tend to be in the ballpark of like 70% on this. Notably this is subhuman. I would guess that anybody in the audience here would score around 100% on these. These are not subtle contradictions of your priors. This is literally like we're flipping the axis on each plot. And so from this we learn that models largely fail to override their own priors about data when they're presented with evidence that contradicts.
BluffBench 2: real-world data quality issues
There are all sorts of issues with BluffBench. The evaluation only presents the sorts of information and tools that the model needs to sort of activate the prior and then see whether the agent will contradict it or not. And so since then we've wondered what if we put these models in a data analysis situation that looks a lot more like real world data analysis. So we give it a bunch of stuff that doesn't matter. Like you put it in a harness that looks like cloud code or something like that and you tell it to load in some data and you have a bunch of back and forth until eventually you ask it to make some plot and that plot has some sort of very subtle artifact inside of the visualization that could feasibly result from a real data generating process.
So for example let's assume that I receive some patient data and I'm just making some exploratory plots to understand what's going on. I say could you plot BMI versus cholesterol. The agent runs some code and it generates a plot that shows that the BMI tends to well as the BMI increases the cholesterol tends to increase. Okay nobody's sleeping so it seems like the audience is engaged. There's something a little weird going on here. As the BMI increases the cholesterol tends to increase except that in the middle sort of right along where you would expect the fitted line to land there are a bunch of points. And so if we take a step back and we're like that's weird what's going on. Turns out if you filter down to one of the clinics that contributed data they didn't include cholesterol and the data model imputed the cholesterol value for those points. So if I were to then go and do data analysis on this data set without realizing this I would make a bunch of errors. And so I either need to filter those out or change them to missing values or use measures that meaningfully incorporate the fact that some of these points are synthetic.
It turns out that even today's frontier models do terribly on this sort of thing. So we designed a follow up LLM evaluation called Bluff Bench 2 that is very similar to Bluff Bench except that it feels a lot more like real world data analysis. And even those same leaders Cloud Fable 5 and Gemini 3 Flash are sort of in the mid teens in terms of the percent of cases where they actually notice that something is awry in the moment that they need to. So from the second evaluation we learned something a little bit more general which is that models fail to notice subtle data quality issues that most data scientists would not miss. One might think at this point should we burn it all down. Should we be using LLMs for data analysis at all. I'll hand it off to my colleague Sarah to answer that question.
The performance of confidence
So as Simon has just explained LLMs can fail at these tasks that are really central to data analysis. Bluff Bench 1 the first Bluff Bench is evidence that models can fail to override their priors about data. And then we also saw in the second eval that they can also fail to notice subtle data quality issues. And both of these abilities like the ability to not let your expectations interfere with your ability to see what's right in front of you and the ability to use a plot or something else to notice data quality issues are really central to data analysis. These are things that we would want any person doing any kind of data science to be able to do. And that also applies to agents.
And there's another problem which is that LLMs are really great performers. You know they can put on this really convincing act of correctness. So we saw in Bluff Bench the model will readily interpret these plots and it acts very confident about what it's saying. And it's very easy to sort of fall into this pattern of trust with them. The only problem is that like in this plot where we have a clearly positive trend that the model has described as a clear negative relationship is that it's like this really convincing performance right. But it's one that occasionally fails to match reality. And these answers all look the same. They're really hard. It's really hard to distinguish incorrect LLM responses from correct ones. So here's another plot from Bluff Bench where I think any person would say you know this looks like a U. It's a parabolic relationship. And then three responses from the same model. It's a little hard to read them but like they all have the same tone. They have the same length. The model is equally confident in all of those. And you would have to look at the plot to know that the first two are very wrong and the middle one is correct. So it's not just that the model is making mistakes but that the answers look the same. It can be very hard to distinguish the correct answers from the wrong ones unless you already know what the data looks like.
It's not just that the model is making mistakes but that the answers look the same. It can be very hard to distinguish the correct answers from the wrong ones unless you already know what the data looks like.
LLMs are still useful — if designed around carefully
So again we see the LLMs can fail at tasks that are central to data analysis. And not only do they fail they can cloak these incorrect answers in confident descriptions and it makes it really hard to tell what's correct and what's not. That sounds a bit negative. I'm going to say that LLMs are still useful for data analysis and we don't need to throw this whole endeavor away just because of evidence like Bluff Bench and Bluff Bench 2. It doesn't mean the LLMs are useless anywhere that you're doing anything that you value correctness. Instead the task is really to figure out how they can fit into the work and to design around their limitations.
So I said I'm going to say that LLMs are still useful for data analysis but we're going to need to do two things in these agents to make them actually work for us. And the first is to make it as easy as possible for them to be right. And then the second is that when they inevitably make mistakes to make it matter less when they're wrong. It's going to prevent mistakes but then mitigate them when they probably eventually happen. This whole time we've been talking about agents but just really quickly all I mean when I say agent is just a model hooked up in a way so that it can gather information from the world like reading files and then also can alter the world so it can run code or it can edit your files.
And you might already use an agent but you can also build them yourself and these lessons are probably most relevant if you're building something yourself or you're altering something that already exists. At Posit we have an agent called Posit Assistant which is both a general purpose coding agent for R and Python and also a specialized data analysis agent. And it lives in our IDEs, Positron and RStudio.
Making it easy to be right
Okay. So I said even though they make all these mistakes and like they seem, you know, these mistakes we see in BluffBench are like kind of bad. You know, these are not things that we want a human agent to do. They're not things we want an LLM agent to do. But LLMs still have a use for data analysis. They can actually be very useful. But we need to make it as easy as possible for them to be right. And what this task looks like is constructing the environment, everything that goes around the model, so that it's as easy as possible for the model to sort of fall into correctness. We don't want it to have to work very hard to, you know, be right. We want it to just fall into this path of correctness. And I'm saying build the environment around the model. Because typically you control the harness, all of this stuff around the model, like the prompting and the tools and the skills. And you're not actually controlling the model itself.
I'm going to talk about two things to construct this ecosystem around the model that makes it easy to be right. The first is that code should really be the foundation of what the agent is doing. So, you want to have the model write code. And then the second is to design the harness with correctness in mind.
Okay. So, when you're making an agent, I think if you want it to be correct, it is really helpful to lean on code. And there's a couple reasons for this. The first is that I think by now many people are familiar with this. But LLMs are actually great at writing code. This is really one of their strengths. And then the second is that, you know, from years of everyone doing data analysis as humans, we've learned that code is great for data analysis. It's reproducible, it's transparent, and it's auditable. And the same things that are true for humans doing data analysis is true for LLMs doing analysis. We shouldn't, you know, abandon these learnings of code just because there's a new tool. So, we want the agent to write code.
And then we also want to design the harness with correctness in mind. So, for Posit Assistant, we have a lot of prompting that encourages it to prioritize correctness over progress. If you let models sort of go on their own, they often will just kind of advance the analysis forward instead of thinking about correctness, subtlety, statistical rigor. So, we have a lot of prompting encouraging it to follow this path. The second is having tools that serve the same purposes of correctness. So, these are things, again, having code is the foundation of what your agent does. Have the agent run code. You could imagine an agent that, you know, does not run code. You just hand it a CSV, and it computes, you know, in its head the median of columns or the number of rows. Generally, it's going to be a lot less successful than having it run a bit of code, see that result, and then give you the answer.
And then you also want the agent to have enough access to files and environment and data so that it has a, you know, good chance of building enough context to actually get the answer right. There's often so much nuance in data analysis, and a lot of that comes from, you know, not just the data, but all of the context around that data. And you want the agent to have access to that.
And it turns out that performance improves when the environment makes it easy to be right. So, we ran an evaluation on Posit Assistant on the Bluff Bench tasks and compared it to the result from the Bluff Bench package that just has a minimal harness in place. All the model can really do there is make a plot. And it turns out that when you have all this stuff wrapped around the model, pushing it towards this path of correctness, performance improves. So, generally, Posit Assistant has done better on this evaluation than just the minimal harness present in Bluff Bench.
Making mistakes matter less
Okay. So, even with all that said, the agent is probably still going to make mistakes. You know, humans make mistakes. Models make mistakes. And so, we need to make it matter less when they're wrong. And again, we get a lot of this mitigation from just using code as the foundation of the agent. One thing code is great for is that it makes auditing what happened in the analysis easy. And for Posit Assistant, we didn't just want the model to write code. Somewhere we wanted the person to be able to see that code and have a chance to audit it and make catch mistakes. This doesn't mean that, you know, you need to comb through it every time. But it's there so you can inspect it if you suspect something has gone wrong and have a chance at catching a mistake and then diagnosing it. We've also done things like make it easy to read with syntax highlighting and the formatting just so this auditing process is more likely to happen.
And then Posit Assistant also has a shared environment between the user and the agent. So, you can see eye to eye with the agent. You see the same outputs and run the same code. And so, the model isn't guessing, you know, what data you have loaded or other things that are in your environment. It's operating in the same environment that you as the user are. And this turns out to be very useful, especially for something like data analysis where it can be very iterative and you're turning the same object, you know, over and over making small changes. You want to have a shared environment with the agent.
Okay. So, in the first half of this talk, we said that LLMs can fail at these tasks that are central to data analysis. But they're still useful for data analysis. Especially if you do a couple things. One is focus on making it easy for them to be right. You know, as easy as possible. So, they just fall into this path of correctness. And the second is to make it matter less when they inevitably do something incorrect.
I have the link to the slides, but also QR code for a newsletter that Simon and I write every other week. And this week, I think we're going to write about BluffBench 2, which is that second evaluation about noticing subtle data quality issues. So, if you want to take a look at it this week, that might be an interesting follow-up. Thanks.
Q&A
So, have you tested your latest benchmark against GPT 5.6?
Yeah. We did run the BluffBench 2 against GPT 5.6. And it ended up that the model's results were sort of in the ballpark of GPT 5.5.
It's... So, the question was, was that eval run with maximum thinking? So, when we run this evaluation, we tend to run it with the settings that we would expect our users to use in Posit Assistant. So, the default thinking setting in Posit Assistant is medium. And as such, we run the eval against medium because most users don't change that default.
For BluffBench 1, we did run it telling the model to ignore the labels. And that does improve performance because the labels are part of the expectations about what the plot is going to be. If you're plotting a known data set, and then you have the X and Y axis labeled in a way that matches the well-known data set, that feeds into the model's priors. So, I think performance on the first BluffBench would almost certainly improve if we stripped those out. The issue is that, like, you probably don't want to do this for your agent. We don't want to do this in Posit Assistant because we want to hand the agent the actual plot object without doing any sort of manipulation behind the scene, which is probably what you want to, like, you want the model to be able to look at a reasonable plot without having to do weird things to manipulate it and be accurate.
Okay. So, what about passing in raw data rather than plots?
Yeah. So, the two BluffBench evaluations are, like, really focused on plots. In Posit Assistant, I don't know what exactly the question was getting at, but, like, Posit Assistant or other data analysis agents are going to look at, you know, if not the actual raw data file to do stuff, like, outputs that make tables, you know, parts of the data, all of that. I think we're less certain about, like, the prior conflict issue with tables or with data. Anecdotally, I think we don't see the issue as much. A lot of the problems have to do with sort of specifics about what's happening with plots and LLM's ability, like, you know, visual reasoning abilities generally. But I do think it would be useful and interesting to do follow-ups where we try to mess with the priors with tables or parts of the raw data.
The question was about the prompting that's available to Posit Assistant. For the most part, it is sort of along the lines of what the question describes, like, asking it to prioritize correctness over this performance of progress. So, rather than making it seem as if the analysis is moving along happily, pausing when there seem to be data quality issues. If you're interested in reading the prompting of the agent, you can just ask it. We've told the agent that it's free to share whatever prompting it has access to. So, it might be an interesting read.
Okay. Can you use Posit Assistant on its own, separate from the Posit ecosystem? Have you done, like, an ablation analysis to see what instructions help the most? Would these be reusable in other workflows?
Okay. So, I'll start with the first question. Can you use Posit Assistant on its own, aside from the Posit ecosystem? One of the interfaces to Posit Assistant, the main two are in two IDEs we developed, called RStudio and Positron, but there's a third, which is a TUI. And so, if you wanted to use Posit Assistant without any other Posit things, you could use the TUI in whatever IDE or in a plain terminal if you wanted.
Okay. So, can Positron's built-in agent access external documentation for auditing correctness?
The question was, can the agent access external documentation for correctness? So, Posit Assistant is shaped similarly to a cloud code or a codex in terms of the tools that it has access to. So, it can run bash code, run Python code, run R code. And as such, it can access the documents that it needs to. Specifically for auditing correctness, if it's not inside of the directory that you're working in or otherwise mentioned in your prompting, you would probably need to nudge it in that direction.
Oh, yeah. Would you say inference is like a hypothesis and code execution is like experimental validation? Yeah. I think that sounds about right. If you're treating the code that it runs like nothing is verified until the code has been run or something, I guess. But even if you're having the agent run code and look at the results, it's still using so much of what is just in the model itself. It's not just looking at the code and it's never going to have no priors or no information that it's using to interpret that. So, there might be some leakage in that metaphor. But that sounds about right.
Okay. What updates to the harness had the biggest gains for accuracy and reliability? Like, what stood out? We don't know. We're kind of surprised by that.
No, we haven't tried to evaluate confidence compared to human confidence. I feel maybe unqualified to measure human confidence. At one point, I think we did try to sort of approximate how confident the model was about its answers and did find that, yeah, like it's not that it's less confident when it's wrong, which is really sort of a problem. If it was less confident and was hedged a lot when it was wrong, that would in many ways be a lot better. But it's sort of equally confident, which, yeah, obviously humans can do as well.
Okay. So, there are a lot more questions. And so, please do look at Slack and maybe answer on Slack because, you know, this has generated a whole lot of interest, which is great. But we do have to cut now so that people have time to go between. So, let's everybody thank Sarah and Simon again.

