This Friday I will be in a panel at the SEN symposium (which I was participating in 2 years ago too, talking about autograders)

The organizers told me that the “discussion will revolve around opportunities, challenges, and potential solutions” and I was allowed to discuss one statement. Since they only gave me 7 minutes (and no slides :() I thought it’d be fun to elaborate a bit more here.

My chosen statement is:

Before we even discuss if or how to use LLMs in CS education, we should talk about why and why not.

Felienne Hermans – SEN symposium June 7th 2024

Why

What has been bugging me in the discourse around LLMs in CSed a lot, is that the core sentiment seems to be: let’s use them, because we can (in a great contemporary Dutch saying that can’t really be translated apty: “niet omdat het moet, maar omdat het kan”). We don’t discuss specific problems, just solutions, we see that in the description of the panel too 1¹.

But what do we even think LLMs can offer? Let’s look at the reasons people give in favour or the use of LLMs.

Why 1. Professionals will use LLMs, so students will need to (learn how to) use them too

I actually don’t hate this argument, I think it is a pretty good point: we need to prepare our students for professional careers. However… LLMs are quite new, and we don’t really know how professionals use them yet (although of course academics are now performing one study after another to better understand their use in practice), and we do not know whether in a few years, when our current students graduate, the situation will be the same.

Here is what we do know:
Programmers don’t use LateX, they use Word or maybe Markdown.
Programmers use GitHub.
Programmers refactor code.
Programmers use debugging tools.
Programmers perform code reviews, programmers use CI/CD tools, programmers deploy code on the Cloud, etc, etc.

So if we are shaping our curriculum by the future usage of professionals (which again, I do not object to at all) why don’t we teach all the above tools, and why don’t we stop immediately with teaching and promoting LateX, a system proven to be worse than Word and not used in industry. If we do the one but not many of the others, what does that tell us about our real motivations? Could it maybe be the case that we think LLMs are very very cool (like LateX) and let that cloud our judgements? Seeing how little actual tools and practices from industry are being taught, I think this is a reasonable hypothesis.

Why 2. LLMs can replace teachers, so we must explore how to use them

The next underlying reason that people rush to use LLMs in education, is that they, fully or partly, believe that LLMs can teach students, rather than professors. Let me firstly throw in this amazing comic by Doug Savage.

I like teaching! I like grading, and making powerpoint slides, not only because they are simply part of my job, but also because grading gives me unprecedented insight into my own teaching, and preparing slides helps me outline my thinking. I don’t want to automate those inherent parts of teaching, and neither should you. If you want to automate teaching, why are you a teacher??

However, me saying that “grading is teaching” and “preparing is teaching” creates interesting discussion. What is teaching? Saying that AI can automate some parts of teaching leads to deep philosophical questions about education which I feel most CS professors are not equipped to properly discuss, because of the lack of philosophical and educational theoretical grounding.

Because in order to discuss what LLMs might mean for “education” we need to define education, and that is harder than you think.

To connect this to point 1, what even is the goal of CS education? Is it to train programmers? Is it do train future scientists? To train programming “though leaders”? ²And should education (any education, or CS in particular) be about teaching skills, or “ways of thinking”, or about giving people the vocabulary and mental tools (like math) to deal with all sorts of issues.

It has been possible, for a very long time, to learn most content of a CS undergrad program by yourself, when I was a kid in the 90s I learned a lot from books from the library, then with YouTube, MOOCs and now (people are saying) with LLMS. Yet, registration numbers for CS programs are soaring! So we must offer (at least in the views of 18 year olds and their parents) a value. What is that value? I would say the value is context, camaraderie with fellow students, connections with teachers, and learning what you did not know you did not know. None of these are automatable with a machine, so what do we even think the LLMs are doing?

And more important thing is not only what the LLMs are doing, but also for whom. In the school of education, where I teach pre-service CS teachers (lerarenopleiding, in Dutch), we tell our students that there is no such thing as a good intervention. Any decision you make while teaching is good for some students, but worse for others. More explanation is good for kids with lower prior knowledge. More group work is good for students in a dominant group.

So who are we designing CSed programs for? For kids that already know they like programming and what to become programmers, or for people that don’t know what programming means? We have to constantly make trade-off.s For example, about equity. If in a given class of 50 students, 10 people are confused and 10 people are bored, who do we care about most? Who do we address and help first? Because we don’t talk about these type of high level design goals, many people design their teaching for people “like them” (likely to be excited about all sorts of technology, including programming and LLMs)

Answering questions like these (which is hard and messy and imprecise) needs to come before deploying any kind of tool. We are already seeing results (very much in line with what learning theory would predict) that students with more prior knowledge are helped most by LLM use, so who are we helping and who are we disengaging (even more)? If you answer with “for all students” you lack an understanding of teaching theory.

Why 3. Students will use LLMs to do their homework!

As I said two years ago at the SEN symposium, I am excited to change introductory programming to be less about programming, and to get rid of autograders. When LLMs came along, I had some level of optimism that this would be the end of programming exercises involving a lot of syntax, but the opposite has been true. We are now leaning into the fatalistic notion that LLMs in CS education are inevitable, and refraining from further reflection upon our teaching.

Let’s dive in a little bit more about what exercises for programming courses usually look like. I think more or less like this:

We give students a prompt, and they have to make a plan how to solve the exercise, choose the right concepts and combine them together in a running and working program. These individual steps, I believe but I base this belief in cognitive science, represent different types of activities. Steps 1 and 2 mainly use the Working memory, fitting a solution in your head and weighting different options. Step 3 mainly uses your long term memory, you will have to remember the right syntax to correctly implement your idea.³

Despite the fact that these are really different things, we grade only the final product, which students struggle a lot to to arrive at, sometimes because their plan was flawed, something because they don’t command the syntax well enough.

The reasons I think are threefold: 1) This is how we all learned, so we think it works, 2) because most CS professors lack the educational theoretical vocabulary to distinguish the steps, and 3) because we can check code easily, so we must (“niet omdat het moet maar omdat het kan”). And not only do we only grade syntax, we also only explain syntax. We never explain to students how they should make a plan, evaluate a plan, choose concepts etc.

In my opinion the deepest issue that we currently have in programming education, is that we do not split these steps into separate exercises and assignments. It would be trivial to do, give students 20 prompts and just have them write down a high-level plan, so they practice this skill, and then transform the plans into lists of concepts, and then to syntax. Only after practicing those skills in isolation, they should be combined. Everything we know about learning supports this way of teaching (in math, we practice many concepts and skills in isolation before combine them; in language learning we do the alphabet, vocabulary and grammar separately) and yet we do not do this.

Because doing this creates the question of how would we grade the natural language plans and lists of concepts? My answer would be that multiple choice is actually a fine strategy, but maybe there even LLMs would make sense, because we have a problem in which natural language needs to be processed! (There are different reasons to not use LLMs though, but at least this is a LLM like problem).

Yes instead of adapting our teaching methods and finally stopping the tyranny of autograders… we keep doing the things we do, but students have access to LLMs and now they are learning even less.

As I also wrote about last week, when I was a young assistant professor, and I was complaining about cheating (pre-GPT) a wise old professor once told me: “When students cheat, *you* are being unreasonable.” I of course thought that was a silly thing to say, the students were just being lazy!! But, in retrospect, this colleague made a great point, we are asking unreasonable amounts of learning in an introductory programming course, so of course students are going to use help, their peers, the internet and now LLMs. If we would make our education more reasonable, there would be less “cheating”.

Do you think students would immediately go to LLms if we’d simple ask them to explain in words how to find the longest alphabetical substring, after they had seen a few similar examples in the lecture? I don’t think so.

—

1. Not to pick on the organizers! This pattern is common everywhere. ↩︎
2. I have talked before (in Dutch) about the opaque goals of CS in higher education: https://www.agconnect.nl/carriere/arbeidsmarkt/waar-is-een-universitaire-informaticaopleiding-voor ↩︎
3. In reality of course there is no linearity in this process, people go back and forth between steps. In this example a student might only realize at the implementation stage that a for loop does not give you easy access to the next letter, needed for this assignment and then switch to a while loop. ↩︎