ok. what are the best studies of AI coding that actually *measure* it and don't just ask the devs or their managers how they *feel* about it?

davidgerard@circumstances.run

not whether the dev thinks they're faster. but whether they clearly *measure* faster, by some reasonable methodology.

frankly, the best study i know of so far is the METR study. that's limited and provides all its own caveats in an extremely honest manner.

AI bros pooh pooh the METR study, but they conspicuously don't do it again in a way that would solve their objections.

instead, the AI bros just don't seem to measure shit.

but surely someone's done a study as good or better than the METR study, right?

dpnash@c.im

@davidgerard

I also want to see a study that doesn’t just measure the time saved (or not) on tasks each person does with generative “AI”, but also the time spent (definitely not saved) by each person cleaning up other people’s “AI”-caused messes. There have been reports along the lines of “X% of workers in Industry Y report having to clean up AI slop”, but I haven’t seen anything that’s detailed enough to say much more than the headline stats. What *is* clear is that just measuring what “AI” saves on specific tasks is not a sufficient measure of its effect on productivity overall.

Abspeckgeflüster – Forum für Menschen mit Gewicht(ung)

ok. what are the best studies of AI coding that actually measure it and don't just ask the devs or their managers how they feel about it?