This is exactly the scenario I encountered. I have a teammate who is using ChatGPT to ask questions as opposed to searching documentation and it gave incorrect info all around on all the following:
1. Where the property would be found in the event schema
2. What structure the value would be
3. The fact that it's not included at all for AWS AppSync specifically in question, even if the previous two had been answered correctly regarding API Gateway event schema.
All of this was available in the first doc returned in the search results of both API Gateway and AppSync respectively.
The fact it convincingly tells you otherwise causes additional friction since they're pointing at their screen telling me I'm wrong cause look at the response from ChatGPT.
You can turn the temperature down on the OpenAI API and it is much more accurate. They also have a code-specific model which I don't think ChatGPT is using.
But, you have to admit that if ChatGPT had a process whereby it'd internally list the sites most likely relevant to your question (essentially Google the keywords) then included those sites as context it'd likely ace your question?
Basically, if Google owned ChatGPT the result would be very different.
This is exactly the scenario I encountered. I have a teammate who is using ChatGPT to ask questions as opposed to searching documentation and it gave incorrect info all around on all the following:
1. Where the property would be found in the event schema 2. What structure the value would be 3. The fact that it's not included at all for AWS AppSync specifically in question, even if the previous two had been answered correctly regarding API Gateway event schema.
All of this was available in the first doc returned in the search results of both API Gateway and AppSync respectively.
The fact it convincingly tells you otherwise causes additional friction since they're pointing at their screen telling me I'm wrong cause look at the response from ChatGPT.