You are viewing limited content. For full access, please sign in.

Question

Question

Feature Request - Display AI Confidence

asked on May 11

Hi All,

 

Would it be possible to embed an AI confidence ranking into smart chat and smart fields? I.e. if the confidence of the field collection was low, you could have a workflow decision route this for a human review etc.

 

This has come up in a few tenders now as competing products offer this functionality. 

 

Cheers!

Chris

2 0

Answer

SELECTED ANSWER
replied on May 13

Which is the correct and honest answer. LLMs, broadly speaking, cannot provide confidence levels for anything in the way statistical models can. Asking for a confidence level from the model here is a "vibe check" at best. If a vibe check is still useful to you as a fuzzy, non-deterministic confidence level, you could try adding something like "If you don't have over X% confidence in the identified field value, append '[Low Confidence]' to the return value.". Gives you something to search / trigger workflows on.

You can get into approaches involving "LLM-as-judge" which have a second model score responses from the first based on pre-defined grading rubrics and such, but that's slow and expensive and you need good grading rubrics for every scenario. Doesn't really make sense for something general purpose by design like Smart Fields.

0 0

Replies

replied on May 11

It is on their list of features to look into. I asked them about it at Empower 2026.

2 0
replied on May 11

Nice, hopefully we can get something in writing her that it's on the radar.

0 0
replied on May 11

I also asked them at Empower and was given the response of "confidence level is effectively irrelevant because the AI makes it up".

0 0
SELECTED ANSWER
replied on May 13

Which is the correct and honest answer. LLMs, broadly speaking, cannot provide confidence levels for anything in the way statistical models can. Asking for a confidence level from the model here is a "vibe check" at best. If a vibe check is still useful to you as a fuzzy, non-deterministic confidence level, you could try adding something like "If you don't have over X% confidence in the identified field value, append '[Low Confidence]' to the return value.". Gives you something to search / trigger workflows on.

You can get into approaches involving "LLM-as-judge" which have a second model score responses from the first based on pre-defined grading rubrics and such, but that's slow and expensive and you need good grading rubrics for every scenario. Doesn't really make sense for something general purpose by design like Smart Fields.

0 0
You are not allowed to follow up in this post.

Sign in to reply to this post.