Wanted: Advice from CS teachers

@futurebird

treat it like a video game. each error is a life but you have to burn all your lifes to get assistance. start with 5 lives?

@maco

And if you're paying for it, there is an implied warranty that you'll get what you paid for.

Oh well, disputes w will be settled using lawyers with LLMs.

Which will further normalise the occupation of society by these corporate spokesbots.

@aredridel @futurebird @david_chisnall

@futurebird
I teach intro courses at a university, which could be a little different but probably not completely. We often do "live coding" in class, with either me or a student at the keyboard while we solve a problem. Regardless of whether it's me or a student "driving", there are always lots of errors to fix, so it's an opportunity to model error-fixing as a normal, expected, creative activity. I always thank students for pointing out my boo-boos, which are plentiful.

@maco @futurebird @EricLawton @david_chisnall That's complex and ever-changing, as business tuning is wont to do.

You usually pay for input and output tokens both, and thinking is part of that. But most people are using plans that give them some sort of semi-metered time-based access — five hours of time with the model and a token limit within that. It's a strange system.

Tokens are roughly in the lex/yacc sense, but they're a new thing, for LLM models. They're not precise parser tokens with parts of speech, but they are roughly "words”. Not exactly, since language is morphologically complex, and programming languages carry semantics in other granules, but the idea that they're words is not wrongheaded.

Others are going flat-fee (F/e, something like z.ai hosted GLM-4.7 is a flat fee per month, and quite low.)

(Also that one is interesting because cost to operate it figures are quite public. The model is public, the hardware requirements are about $15000, so you can do the math on it pretty easily to see what capital costs would be. Also environmental! Like that's 3 high end server GPUs, so a lot of heat, but also to humanize it, "less than heating a house" amounts of energy by far.)

@maco @aredridel @EricLawton @david_chisnall

I strongly suspect they are vastly undercharging and counting on building dependency and jacking up the prices later.

The free workshops seem to be all about that and didn't impress me much. But, I did get to play with the tech so I could better understand it which was worth it despite all of the sales pitch infused through the process.

@futurebird @aredridel @EricLawton @david_chisnall oh yes, I had the impression there was some of that going on. Some of the services did jack their prices up some time a year or two ago; I remember there being sticker shock. I think people expected what was free at first to go to like $20/mo and it actually went much higher.

But I have no details. I haven’t used any of it.

@futurebird @maco @EricLawton @david_chisnall Oof. A sales pitch embedded in it sounds miiiiiserable.

As far as pricing ... man it's hard to tell. The training of models is very expensive, and energy-consuming. That has to be amortized somehow. But the actual running takes only a little more than 'home computer' level. (and cruddier models do run on home computer scale things)

@futurebird not a cs teacher, so feel free to disregard, but maybe you could split lectures up with students just taking notes during some examples, and following along with others? My favorite coding professor also often intentionally put in common errors to the examples he was doing, then asked the class what needed to be done to fix them

@futurebird i remember to have read the current prices are only 1% of what it would need to cover the cost.
And thats not yet including the increased energy prices caused by the co2 tax increses now coming.(not that any of the big companies would pay this)

@futurebird@sauropods.win Yes, I would say so. Functions in math are different from functions in code. Mathematical functions look more like lookup tables or dictionaries. One sticking point is the flow of control: a function has a block of instructions that are not executed at the point where they're written in the source code. This is really confusing for some people, especially if they've just been taught that computers go through a list of instructions one by one, executing each in sequence.

Add in functions that have side effects, functions that don't return a value (procedures), functions that trap the rest of the execution (continuations), etc., and you're well outside of what most people understand mathematical functions to be like. The mathematical sine function can't make a network connection or write to a file or...

You can sometimes suss this out by comparing a function to a dictionary (or similar lookup type data structure). Those don't involve changes in the flow of control, and students tend to grasp what they're doing much faster. Students who grasp dictionaries sometimes cannot transfer that understanding to functions because of the flow of control issue, I think, so it can be helpful to probe whether they understand one but not the other and try to figure out why.

@Catfish_Man @futurebird

"If debugging is the process of removing bugs, then programming must be the process of putting them in." - Dijkstra

@futurebird Not much to add you haven't already thought of, but I agree with a lot of what you said and feel your frustration. If I need to keep going with the instruction like in your example (turning into a function next), I would tell the students with errors to shift to copying the new stuff down as notes, so they don't compound multiple errors if they continue to try coding along with me, ...

@futurebird ...then during the walk-around time, have them "ask three before me" so they can practice fixing each other's errors. I also like your idea (and should do more of it myself) of giving them code with errors as a warm up, and asking them to think about how to fix.

@maco @aredridel @futurebird @EricLawton @david_chisnall In general, they charge for both input tokens and output tokens, at different rates. For example, Claude Opus 4.5 charges $5/million input tokens, and $25/million output tokens.

In order for an LLM to keep track of the context of the conversation/coding session, you need to feed the whole conversation in as input again each time, so you end up paying the input token rate many times over.

However, there's also caching. Since you're going to be putting the same conversation prefix in over and over again, it can cache the results of processing that in its attention system. Some providers just do caching automatically and roll that all into their pricing structure, some let you explicitly control caching by paying for certain conversations to be cached for 5 minutes or an hour. So then, you pay once for the input and once for the caching, and then you can keep using that prefix and appending to it.

If you're paying like this by the token (which you do if you're just using it as an API user), then yeah, if it gets it wrong, you have to pay all over again for the tokens to correct it.

However, the LLM companies generally offer special plans for their coding tools, where you pay a fixed rate between $20 and $200/month, where you have a certain guaranteed quota but can use more than it if there's spare capacity, which can allow you to use more tokens for a lower price than if you just paid by the token. But of course it's not guaranteed; you can run out of quota and need to wait in line if the servers are busy.

And their tools handle all of that caching, selecting different models for different kinds of tasks, running external tools for deterministic results, etc.

@futurebird @quixoticgeek if you can put a student's screen on the projector, maybe walk each one in turn through "read the error aloud, ask what it might mean, see which line it's complaining about, read that line aloud," and so on?

Could help reinforce that these problems can be worked through. Also changes the social calculus from "I cast Summon Help From Authority Figure!" That part could go either way, but handled well, it could build camaraderie ("see, we all make mistakes")...

@futurebird I will venture an answer - I haven't read all the replies to your post so I may be saying things that are already said.

I'm a UK-based secondary (aka high school) maths teacher who also teaches CS, including introducing students to programming (usually python).

Here's some thoughts of things to try.

1. Practise finding errors. Give them code with errors and ask them to find them. Set problems to generate particular error messages. eg Can they write code where the mistake is on line 3 but the error message says line 33?

2. Have a list of general prompts that you will say. If the error is on a different line, just say "Take a look at line N" then walk away to the next student. If they've named a variable badly, say "variable names can't have spaces". Make them do some work here.

3. Clearly delimit demonstration time and coding time. "Fingers off keyboards and mice" is a common phrase in my classrooms (and I will stop the entire demo if I hear a clicking).

4. Make them keep notes. I use Google Colab so that they can interweave notes with code snippets (at least in the early days) to encourage this.

5. Partner up. It's often easier to find the error in someone else's code than in your own. Could even have a "walk around time" when all the students go and look at others' screens to both get ideas and see if they can spot errors.

That's what springs to mind on reading your thread. I hope some of it's useful! As with all advice on the internet - keep what helps and ignore what doesn't.

@EricLawton Oh my goodness yes this 100.

Stolen from somewhere: there are 2 hard problems in software, (1) human communication and (2) convincing people that human communication is important

@futurebird @david_chisnall

@wakame @Catfish_Man @futurebird The stat I heard was that you introduce two bugs for every line of code you write.

@apontious @wakame @Catfish_Man

@unlambda

Once there are few enough expert human programmers left, the price will go up.

And, if I read you correctly, they don't guarantee output accuracy with respect to input tokens but charge extra to try again.

And if they charge per output token, that is incentive to generate filler, certainly not to optimize.

@maco @aredridel @futurebird @david_chisnall