ChatGPT, the snake oil salesman
One would think that there is no longer reason to write anything about ChatGPT, that the capabilities are better understood now and its usage is mostly limited to producing inconsequential content for amusement. I wrote about it in January, comparing it to a circus monkey which entertains but definitely doesn't understand the content which it is producing. Later, I wrote about the dangers of taking the output at face value, especially given the fact that ChatGPT completely hides how the answer is derived and what source was used to train the system.
One can find it amusing when two lawyers used ChatGPT for a court filing while citing court cases which didn't exist. Let's just say that that judge didn't find it funny. The lawyer tried to explain himself with 'I did not comprehend that ChatGPT could fabricate cases'. Fortunately, the other party in the lawsuit used a human-based due diligence process and discovered the error.
In this example, the error was easy to spot since the information was written up and experts were able to review it. However, it shouldn't be dismissed as a harmless mishap. The legal system is based on truth and facts. Only then does the judge have the chance to pronounce the most informed judgment.
But then there is another line of business where accuracy, explainability and traceability is a must - software development. Books were written about how important it is to write good code and how to properly test it. We rely on this accuracy all the time. There is no moment in your life where the accuracy of computer systems is not critical to your well being. And yet software developers are trying to shortcut the process and use ChatGPT (and similar systems) to produce code.
To become a software developer is like any profession. To become really good at it, you have to spend years and years to perfect your trade. You can use various tools to help you with mundane tasks. You can automate repetitive steps. However, you are always responsible for the work you produce. Every single line of code is always traceable to you.
Yes, there are times when you get into a dead end and you need help. There are many places on the Internet where developers can ask for help or see how others have solved a similar problem.
One of the places is Stack Overflow, a website where questions get asked and answers provided, argued and rated. A true community where developers contributed to better code. It is also a website, which got used for its content during the ChatGPT training process.
If you wonder how good the answers to your coding questions are, you are not alone. Researchers from Purdue University had the same question. They ran a study 'Who Answers It Better? An In-Depth Analysis of ChatGPT and Stack Overflow Answers to Software Engineering Questions'.
The results were not good.
52% of ChatGPT-generated answers were incorrect.
As well, about 78% of the answers suffered from different degrees of inconsistency to human answers.
The worst finding from the study? 'However, participants still preferred ChatGPT-generated answers 39% of the time. When asked why they preferred ChatGPT answers even when they were incorrect, participants suggested the comprehensiveness and articulated language structures of the answers to be some reason for their preference.'
Several notable findings from the study:
Many answers are incorrect due to ChatGPT’s incapability to understand the underlying context of the question being asked. ChatGPT makes fewer factual errors compared to conceptual errors.
ChatGPT rarely makes syntax errors for code answers. The majority of the code errors are due to applying wrong logic or implementing non-existing or wrong API, Library, or Functions.
Users overlook incorrect information in ChatGPT answers due to the comprehensive, well-articulated, and humanoid insights in ChatGPT answers.
Similar to the snake oil salesmen from the past, ChatGPT is able to convince people with authoritative answers about the correctness of its answers. Using it during software development you just create a hidden, ticking bomb.
What started as an amusement, an interesting research project is quickly becoming a sharp knife in the hands of children. Every time you read about the dangers of AI taking over, remember, it is nonsense. What we should be afraid of is people with no comprehension about the impact of their actions. That's the recurrent pattern.