Victor C.M. is scraping a layer of lime mortar from a brick that has survived 141 winters. His movements are rhythmic, almost geological. He doesn’t look at the blueprint because the blueprint is a suggestion, whereas the physical weight of the masonry is an absolute truth. I watched him for 21 minutes this morning, thinking about the way we measure expertise. He knows exactly how much pressure to apply to avoid cracking the nineteenth-century face of the stone, yet if you handed him a written exam on the chemical composition of hydraulic lime, he might fail. We are obsessed with the proxy-the score, the certificate, the standardized output-and in our obsession, we have forgotten how to look at the work itself.
Operational Reality
Proxy Measure
Yesterday, I pushed a door that clearly said pull. I stood there for a split second, chest colliding with the handle, feeling that specific, hot flush of stupidity. It wasn’t that I couldn’t read. It was that my brain had anticipated a different mechanic of the world. This is exactly what happens in high-stakes language testing. We build doors that say ‘English Proficiency,’ but the mechanics we use to open them have nothing to do with speaking a language. We use multiple-choice logic and rehearsed monologues, then act surprised when the candidate hits the door face-first in a real-world stickpit or boardroom.
A few months ago, a language assessment researcher sat in a dim office in Zurich, comparing FCL.051 results with actual operational communication samples from the same group of pilots. He was looking for a bridge between the test score and the reality of a pilot handling a fuel emergency in heavy rain. What he found was a correlation of 0.31. In the world of statistics, that is the equivalent of saying that the test is about as good at predicting operational success as a coin flip would be at predicting the weather in a month. The test wasn’t measuring English; it was measuring the ability to navigate the specific, idiosyncratic architecture of the test itself.
The Validity Crisis
Consider the way we assess listening. In a standard exam, you hear a recording of two people discussing a shopping list. You are asked to identify how many apples they bought. In reality, listening is a messy, collaborative act of negotiation. You listen for intent, for stress, for the silence between words that indicates a speaker is lying or exhausted. By stripping the context down to a binary choice of ‘4’ or ‘5’ apples, we aren’t testing listening. We are testing the ability to isolate acoustic data while ignoring the human element. This is the validity crisis: when the test becomes the end, it ceases to be a means. We train people to be test-takers, not communicators. We produce 101 candidates who can recite a grammar rule but cannot negotiate a simple misunderstanding with a frustrated air traffic controller.
Victor C.M. finally looks up from his wall. His hands are caked in a grey-white dust that seems to have become part of his skin. He tells me that a wall doesn’t care about the mason’s intentions. It only cares about gravity. Language is much the same. The sky doesn’t care if you know the difference between the present perfect and the past simple if you cannot convey that your left engine is trailing smoke. Yet, we spend billions of dollars on assessments that prioritize the former. We have created a parallel reality where ‘English’ is a set of 51 specific tasks performed in a sterile room, rather than a living tool used to move ideas from one brain to another.
Language as a Tool
Sterile Room Tasks
Operational Reality
I’ve spent 11 years looking at these gaps. I’ve seen students who could score in the 91st percentile on a grammar diagnostic but couldn’t order a coffee without an emotional breakdown. Why? Because the test removed the ‘noise’ of human interaction. But the noise is the language. The stuttering, the backtracking, the ‘uh-huhs’ and the ‘you knows’ are the grease that keeps the gears of communication from seizing up. When we remove those to make a test easier to grade, we are essentially testing a car’s engine while it’s on a block with no wheels. It might rev beautifully, but it’s not going anywhere.
The Language of Life and Death
In the aviation sector, this disconnect is literally a matter of life and death. The ICAO Language Proficiency Requirements were meant to ensure safety, but in many regions, they have devolved into a game of rote memorization. Candidates learn ‘test-wise’ strategies. They learn how to sound like they are fluent without actually being able to process unexpected information. They learn the rhythm of the examiner’s questions. They become experts in the ritual, not the reality. Understanding the nuances of Level 6 Aviation helps illustrate how varied these hurdles can be, even when they all ostensibly aim for the same standard of safety.
I remember an 41-year-old engineer I once worked with. He was brilliant, capable of explaining complex hydraulic systems with a pencil and a napkin. But under the fluorescent lights of a testing center, his English evaporated. He became obsessed with not making a mistake. The fear of the rubric paralyzed his tongue. He wasn’t failing because his English was poor; he was failing because the test demanded a performance of perfection that doesn’t exist in nature. Language is inherently imperfect. It is a series of corrections. If you push the door and it doesn’t open, you pull. The test, however, marks you down for pushing the door the first time.
Recipe vs. Ingredients
We need to stop treating language as a fixed list of ingredients and start treating it as a recipe that changes based on who is at the table. A mason like Victor knows that the mortar mix changes based on the humidity and the temperature. He adjusts. A pilot adjusts to the accent of a controller in Marseille or Mumbai. Standardized tests, by their very definition, do not allow for adjustment. They are rigid. They are the door that says pull when your instinct says push, and they don’t care if you have a valid reason for your instinct.
There is a strange comfort in a score. It’s a number. It’s 81 out of 101. It looks objective. It looks like truth. But it is a curated truth. It is a snapshot of a person in an unnatural state of stress, performing a task they will never perform again in their actual life. We have sacrificed the ‘operational’ on the altar of the ‘assessable.’ It is easier to grade a multiple-choice question than it is to evaluate how well a human being can make themselves understood during a crisis. We chose the easy path, and we are paying for it with a global workforce that is certified but not necessarily capable.
Victor C.M. puts his tools away as the sun starts to dip. He hasn’t finished the wall, but the part he did is solid. It will stand for another 71 years, regardless of whether anyone ever tests his knowledge of mortar. He works in the real world, where the stakes are physical. Our language tests exist in a vacuum, where the stakes are purely bureaucratic. Until we bridge that gap-until we start measuring the ability to get the job done rather than the ability to pass the exam-we will continue to be surprised when the high-scorers fail to communicate.
“The test is the map, but the map is not the territory.”
I’m still thinking about that door. The one I pushed. I was looking at the sign, but my body was moving based on my experience of every other door in that building. We are training people to look at the signs and ignore their experience. We are teaching them to prioritize the ‘correct’ answer over the ‘functional’ outcome. If I push a door and it doesn’t open, I haven’t failed at ‘dooring.’ I have simply encountered a design that didn’t align with my operational reality. Our tests are poorly designed doors. They are obstacles that claim to be gateways. And as long as we keep measuring the push instead of the passage, we’ll keep hitting our heads against the glass.
Functional Outcome
Poorly Designed Doors
Measuring the Push
Maybe the solution is to bring the mason into the testing room. Or the pilot. Or the engineer. Not as subjects, but as the standard. We need to ask: ‘Can this person do the thing?’ not ‘Can this person describe the thing using the third conditional?’ Because at the end of the day, when the wind is at 31 knots and the visibility is dropping, no one cares about your grammar. They care if you can hear the warning in the voice on the other end of the radio. They care if you can find the words to stay alive. Everything else is just lime dust on a historic brick.
Describing the thing
Doing the thing
How much of what you call ‘skill’ is just your ability to survive the systems we built to measure you?