Substrate Engineering
Writing code is much, much easier than reading code later and understanding it, still less spotting bugs in it. Spotting a bug is a challenge even when you know there is a bug: that’s what debugging is, and it’s hard. And when you prompt Copilot or Cody, and use the code it provides, code review is exactly what we are doing.
The better LLMS get -
the more they boost velocity
by generating working code -
the harder it will be to notice when
they get things wrong
Our software foundations need to get much better, across the board, or else widespread use of LLMs is going to make our software quality much worse. Our only chance is to double down on defense in depth: in making all of our software foundations better.
Goes on to talk about designing configuration languages that are more robust to LLM's imaginations and mentions Dhall and Starlark
if we take this space seriously — and we should! — I think there is room to build a language that has a lot of what Starlark has going for it in terms of familiarity and accessibility, but which also pulls in the good ideas from Dhall’s type system. And then we could see: does it work well? Where does it fall down? And does it in fact help with providing useful guardrails for using LLMs? I don’t know: it’s a hypothesis. But we should try it!