apenwarr on insight, engineering, and how an LLM might relate to them
Particularly enjoyed this quote, remembered from an instructor:
Nobody in the world knows how to build a paperclip that will never break. We could build one that bends a thousand times, or a million times, but not one that can bend forever. And nobody builds a paperclip that can bend a thousand times, because it would be more expensive than a regular paperclip and nobody needs it.
Engineering isn't about building a paperclip that will never break, it's about building a paperclip that will bend enough times to get the job done, at a reasonable price, in sufficient quantities, out of attainable materials, on schedule.
Engineering is knowing that no matter how hard you try, some fraction of your paperclips will snap after only one bend, and that's not your fault, that's how reality works, and it's your job to accept that and know exactly what fraction that is and design around it, because if you do engineering wrong, people are going to die. But what's worse, even if you do engineering right, sometimes people might die. As an engineer you are absolutely going to make tradeoffs in which you make things cheaper in exchange for a higher probability that people will die, because the only alternative is not making things at all.