When GPT Sounds Confident But Is Totally Wrong
So I was rebuilding my notion-to-Google-Calendar automation (for the third time this month đ) and needed GPT to generate a regex for a date format with optional times. The actual prompt I used was:
> Write a regular expression for matching dates like “April 4 2024” and optionally also allow times like “April 4 2024 at 9am”
To my surprise, GPT gave me a long regex that **looked** legit. It had month names, optional groups, anchors. At a glance, it passed the vibe check. But when I pasted it into regex101 to test it… nada. Not even close. It didnât match anything. Turns out it was suffering from a classic case of overconfidence and under-parsing: it assumed whitespace handling that didnât match real world inputs.
The most common issue I see (and re-see) with prompts like this? Missing edge case phrasing. GPT often sees “April 4 2024” and assumes the delimiter options never change. If someone writes “April 4, 2024” with extra spaces or a comma, it silently fails. Even worse, it wonât warn you.
If youâre trying to avoid this, you need to add precision during generation **and** verify the results with real input. Here’s how I fixed mine by tweaking the original prompt:
> Create a regular expression that matches month names, optional commas, variable spaces, and the word “at” followed by a 12-hour time with optional minutes.
Suddenly everything aligned, mostly because I named what *looked optional* but wasnât â like commas and whitespace.
Add Clarifying Fail Conditions To Your Prompt
This is something I always forget, then later kick myself about during debugging. You ask GPT to give you a bash script that renames files in a folder, and it does⌠kinda. But then it starts renaming files that you didnât want included â like hidden system files. Because you didnât specify what NOT to do.
Letâs take this simple prompt:
> Write a bash script that renames all files in a folder to match lowercase snake_case format
Fine, sure. Itâll reply with something using `for file in *` loop and probably some sed magic. But those asterisks include everything â sometimes `.` files, sometimes folders. On my Mac, I ran a version of this and it renamed `.DS_Store` to `ds_store`. Not a huge deal, but still: not what I meant.
Hereâs how you can make prompts safer with failure boundaries:
> Do not include hidden files, folders, or files that are already in snake_case. Skip files with dashes, unless they contain uppercase.
Being overly literal helps GPT draw an edge around success. It wonât magically infer your rules.
Youâd think GPT could guess the meaning from your goal â but it usually doesnât. It optimizes for typical examples, not weird ones. And letâs be real, all real-world file sets are weird.
Force Output Format With Explicit Tokens
Hereâs the thing that still trips me up: when I want structured data (like JSON), I assume GPT will understand that from me just saying âoutput as JSON.â Nope. Itâll sometimes mix in commentary lines or drop trailing commas. Both will break your parser.
I had a case where I needed GPT to respond with a JSON array of image captions based on descriptions. I told it:
> Return a JSON array of photo captions given these image topics
It responded with this:
“`json
// List of captions
[
“Dog running on the beach”,
“Sunset behind the mountains”,
…
]
“`
Wait, whatâs this line at the top? `// List of captions`?! Didnât ask for that.
Hereâs how I fixed it:
> Output only valid JSON. Do not include any extra commentary, explanations, or formatting outside the brackets. Your output must be parsable without modification.
This version worked because it gave GPT no wiggle room to âbe helpful.â Any time I want to pass a GPT result into something downstream (like a Node parser or webhook body), I now surround the prompt with hard boundaries:
> Begin output with an open bracket. Do not say anything else. Do not explain or describe.
That wording matters more than I expected. And yet yes, half the time I test it once, then start trusting it, and it breaks the moment I need it live đ
Test With Obnoxious Edge Cases
Hereâs what I do now and shouldâve done forever ago: once GPT gives a solution, I test it with inputs that shouldnât work. Like deliberately wrong formats, mixed cases, missing data.
For example, if GPT gives me an Airtable formula like:
“`airtable
IF({Status} = “Complete”, “Done”, “Pending”)
“`
I try things like lowercase complete, null status, or âcompletedâ instead. Because in the real Airtable base I use, half the statuses were copy-pasted with invisible trailing spaces. So nothing matched.
What fixed it?
I updated the prompt to:
> Airtable formula to check if status field contains the word complete, case-insensitive, ignoring whitespace around it
And got back:
“`airtable
IF(FIND(“complete”, LOWER(TRIM({Status}))), “Done”, “Pending”)
“`
Much better. Sure, still brittle, but at least it doesnât stop working the moment someone pastes in ” Complete ” from Slack.
Point is, assume the environment is filthy. Assume users will screw things up. Assume data is missing. Then prompt *as if* you’re dealing with a hostile input box đ
No Mention Of The Upstream Bug
Sometimes you get a broken answer not because the prompt is bad, but because GPTâs memory or pretrained assumptions are wrong. Like the time I asked it:
> How do I get current userâs email in a Google Workspace Google Apps Script
Its answer confidently used `Session.getActiveUser().getEmail()` â which returns blank unless the script is deployed with the right permissions. That fair-sounding answer wasted an hour of testing because GPT didnât warn me about the null.
If youâre doing anything tied to authentication (Apps Script, Airtable scripts, Postman pre-requests), GPT often âforgetsâ important scaffolding like:
– You must publish script as web app with correct auth scopes
– You must have advanced services enabled
– That variable only works when deployed, not in the editor
To avoid this kind of missing-context bug, I go back and specify exactly what state Iâm in. For example:
> Iâm using Google Apps Script in the script editor, not deployed. I am not using a web app. The script is bound to a spreadsheet. I want the current userâs email, not the script owner. Whatâs the correct API call (if any)?
GPT will *still* hedge a little, but at least now it sometimes admits:
> Without deployment as a web app or proper permissions, Session.getActiveUser() may return blank.
That tiny bit of honesty is what you need to short-circuit bad assumptions before you start tearing apart all the other parts of your app.
Add Dummy Payloads For Webhooks Or APIs
Getting GPT to generate webhook responses is a hit-or-miss game until you start including a dummy payload. Otherwise itâll invent keys that donât exist.
I had this webhook from Pabbly that looked like this (simplified):
“`json
{
“user_email”: “test@example.com”,
“purchase_status”: “Completed”
}
“`
But when I prompted:
> Write a zap that triggers when a purchase is completed and emails the user
GPT used `email` as the field name and checked for status as `success` đ totally guesswork.
Once I updated my prompt like this:
> Use this test data: {“user_email”: “…”, “purchase_status”: “Completed”}. The zap should email user_email if purchase_status equals Completed.
GPT nailed it. Gave proper filters, correct field names, and no made-up fields.
Always paste in sample webhook data when asking for automation logic. Otherwise GPT hallucinates generic keys that sound real but wonât ever trigger.
Repeat The Goal Out Loud MidPrompt
This is a weird trick but it works surprisingly often. During long prompts where I ask GPT to write a multistep automation or script, I try adding short reminders to itself about what the end goal is.
Something like:
> Our final output should be a working Zapier javascript code step that extracts a userâs first name from full name, makes it lowercase, and updates a Google Sheet. DO NOT include full name again. Only first name in lowercase.
Even though I already said that earlier in the prompt, repeating it *after* the explanation gives GPT better cueing to restrain itself. It especially helps with things like:
– Avoiding extra variables
– Not re-initializing things you just parsed
– Avoiding extra steps that sound smart but arenât actually needed
So now I literally sandwich prompts like this:
> Step 1: Explain logic
Step 2: Write code
Reminder: Final answer should return only first name in lowercase, nothing else
Itâs like doing your own linters, but with words đ