AI is making us write more code.
-
AI is making us write more code. That's the problem.
I analyzed research papers on AI-generated code quality. The findings:
→ 1.7x more issues than human-written code
→ 30-41% increase in technical debt
→ 39% increase in cognitive complexity
→ Initial speed gains disappear within a few monthsWe're building the wrong thing faster and calling it productivity.
That's my experience too.
Dave Farley's. MSE channe, which I usually respect,l recently claimed the opposite though, based on a study they took part in.
-
AI is making us write more code. That's the problem.
I analyzed research papers on AI-generated code quality. The findings:
→ 1.7x more issues than human-written code
→ 30-41% increase in technical debt
→ 39% increase in cognitive complexity
→ Initial speed gains disappear within a few monthsWe're building the wrong thing faster and calling it productivity.
@mlevison Goodhart's law in action. Actually, I'm not sure it even is an example of Goodhart's law. Raw quantity of code output would never have correlated strongly with quality. What do they think they're doing‽
-
AI is making us write more code. That's the problem.
I analyzed research papers on AI-generated code quality. The findings:
→ 1.7x more issues than human-written code
→ 30-41% increase in technical debt
→ 39% increase in cognitive complexity
→ Initial speed gains disappear within a few monthsWe're building the wrong thing faster and calling it productivity.
@mlevison I use LLMs to help me with basic code writing tasks, generating the structural frameworks, saving me a lot of typing time. However, I never rely on that code out of the box, I always review it thoroughly and often just snip and prune. I would never attempt to give an LLM a complicated set of instructions, it's going to fail every time.
-
@mlevison I use LLMs to help me with basic code writing tasks, generating the structural frameworks, saving me a lot of typing time. However, I never rely on that code out of the box, I always review it thoroughly and often just snip and prune. I would never attempt to give an LLM a complicated set of instructions, it's going to fail every time.
@mlevison Intellisense, pretti, etc. are all just tools for a smart developer.
-
AI is making us write more code. That's the problem.
I analyzed research papers on AI-generated code quality. The findings:
→ 1.7x more issues than human-written code
→ 30-41% increase in technical debt
→ 39% increase in cognitive complexity
→ Initial speed gains disappear within a few monthsWe're building the wrong thing faster and calling it productivity.
@mlevison I hear you. I am seeing it first hand. And I am being hammered for pointing this out and encouraging caution, while others are being rewarded for shipping
“because AI”.Just call me ‘Cassandra’
️ -
@mlevison This talk on AI productivity from a Stanford researcher goes into detail on this. Really interesting and gels with what you're saying: the actual productivity gains tended to be with simple (low-complexity, greenfield) tasks rather than anything bigger or more complex. https://youtu.be/tbDDYKRFjhk?si=AM5DPcJGeg_3ignp
-
@mlevison Intellisense, pretti, etc. are all just tools for a smart developer.
@crackhappy @mlevison Jetbrains vanilla Intellisense was pretty good even before the latest epidemic of AI psychosis.
-
@crackhappy @mlevison Jetbrains vanilla Intellisense was pretty good even before the latest epidemic of AI psychosis.
@thirstybear @mlevison I refuse to call what we currently have "Artificial Intelligence" because it is not. It's a fundamentally clever implementation of Markov chains with way way too much power applied.
-
AI is making us write more code. That's the problem.
I analyzed research papers on AI-generated code quality. The findings:
→ 1.7x more issues than human-written code
→ 30-41% increase in technical debt
→ 39% increase in cognitive complexity
→ Initial speed gains disappear within a few monthsWe're building the wrong thing faster and calling it productivity.
@mlevison
l suspect the line about the volume of code increase is conservative. -
AI is making us write more code. That's the problem.
I analyzed research papers on AI-generated code quality. The findings:
→ 1.7x more issues than human-written code
→ 30-41% increase in technical debt
→ 39% increase in cognitive complexity
→ Initial speed gains disappear within a few monthsWe're building the wrong thing faster and calling it productivity.
@ivanmorgillo What's your experience about this?
-
AI is making us write more code. That's the problem.
I analyzed research papers on AI-generated code quality. The findings:
→ 1.7x more issues than human-written code
→ 30-41% increase in technical debt
→ 39% increase in cognitive complexity
→ Initial speed gains disappear within a few monthsWe're building the wrong thing faster and calling it productivity.
@mlevison Question : would have moonlanding been also a success if AI would have had control ?
-
@mlevison Question : would have moonlanding been also a success if AI would have had control ?
@mlevison We may remember the iconic foto of dear Margaret Hamilton stood beside a pile of papers of machine code for the AGC computer . What if AI would have programmed it ? A late competition would be enlightened about mankind’s biggest challenge last century
-
AI is making us write more code. That's the problem.
I analyzed research papers on AI-generated code quality. The findings:
→ 1.7x more issues than human-written code
→ 30-41% increase in technical debt
→ 39% increase in cognitive complexity
→ Initial speed gains disappear within a few monthsWe're building the wrong thing faster and calling it productivity.
@mlevison I don't think any AI can write code worse than Windows, though. Just saying.
-
AI is making us write more code. That's the problem.
I analyzed research papers on AI-generated code quality. The findings:
→ 1.7x more issues than human-written code
→ 30-41% increase in technical debt
→ 39% increase in cognitive complexity
→ Initial speed gains disappear within a few monthsWe're building the wrong thing faster and calling it productivity.
@mlevison @Em0nM4stodon
→ 30-41% increase in technical debt
-
AI is making us write more code. That's the problem.
I analyzed research papers on AI-generated code quality. The findings:
→ 1.7x more issues than human-written code
→ 30-41% increase in technical debt
→ 39% increase in cognitive complexity
→ Initial speed gains disappear within a few monthsWe're building the wrong thing faster and calling it productivity.
@mlevison Echoes of the mountains of boilerplate crap churned out towards the height of the .com bubble
Pay by the SLOC, get more SLOC. Whether it does anything useful or not.
-
@mlevison I use LLMs to help me with basic code writing tasks, generating the structural frameworks, saving me a lot of typing time. However, I never rely on that code out of the box, I always review it thoroughly and often just snip and prune. I would never attempt to give an LLM a complicated set of instructions, it's going to fail every time.
Couldn't human made deterministic tools (or changes to programming languages) help with boilerplate work instead of indeterministic intransparent generative AI?
-
Couldn't human made deterministic tools (or changes to programming languages) help with boilerplate work instead of indeterministic intransparent generative AI?
IIRC for some languages there also have been deterministic refactoring tools too that take over the tedious parts of refactorings (like "rename method" which exactly identified callers to adapt them).
-
The bottleneck was never writing code. It's understanding what to build.
If you're using AI coding tools, focus on:
• Smaller features (if it's 1000 lines, it's too big to review)
• Clear acceptance criteria before you prompt
• Tests first, AI-generated code second
• Security audits (AI can't do this)More code isn't the goal. Solving real problems is.
@mlevison I had a discussion with some developers about this last week. If we expect developers to take responsibility for the code they generate, the amount of work is capped by their ability to review code. And if we expect more (which is absolutely necessary, if we want the promised productivity gains), we can't blame them, if it goes wrong.
If we celebrate huge productivity gains, we give up the right to complain, if AI fails. And it will fail.
-
AI is making us write more code. That's the problem.
I analyzed research papers on AI-generated code quality. The findings:
→ 1.7x more issues than human-written code
→ 30-41% increase in technical debt
→ 39% increase in cognitive complexity
→ Initial speed gains disappear within a few monthsWe're building the wrong thing faster and calling it productivity.
@mlevison Overhyped and underdelivering, just like the 4GL hype of the late eighties which promised to make programmers redundant within 10 years.
https://en.wikipedia.org/wiki/Fourth-generation_programming_language
-
N necrosis@chaos.social shared this topic
