On coding, it’s clear Western frontier labs are still significantly ahead in enterprise engineering work where you’re dealing with huge code bases & simple iterative prompts (since people rarely write sophisticated technical specs to communicate with LLMs, much more often you’ll just see “fix this code for me it’s not working.”) This is exactly the setting this benchmark tries to capture.
But strong as they are, the costs of hyper scaling are also starting to pile up. True, they are “cost effective” relative to open models in situations where the user puts in minimal effort (since the open models will just repeatedly fail in those situations), but they still increase the cost of building software dramatically. The extra productivity they provide isn’t justified by the product value due to there being a finite market & only so much money to go around. Nobody is going to spend more on software services than rent, food, gas, etc.
So companies are ironically caught in a catch 22 with AI that is only really solvable via lay offs and scaling down the number of engineers. This is however a vicious cycle for capitalist economies where supply & demand are intertwined. The higher productivity from AI is not being absorbed by the market because consumers are being eliminated by the lay offs. Longer term there will be a ceiling at which the costs of using AI will exceed the market value & once we get there (and it’s probably soon), there will be opportunities for Chinese labs to catch up & overtake via value since it’ll become a race to the bottom on price.
Last edited:
