Target Audience - Annotator
Adding Phrases to Skills (Two Methods):
- Open Skill Details and add phrases. Can see all details about the skill using this method
- Open Conversation Details and use the Add to Existing Intent feature to convert the conversation into a training phrase for any skill
Best Practices
- Phrases should be short and specific
- Variables should be used if possible
- Unresolved Conversations should drive improvements
- Test your annotations a lot
- Get feedback
1. Phrases should be short and specific
- Try to write a bare-bones phrase that captures the essence of a skill, but is specific enough that it uniquely identifies this skill
- Add variations of this bare-bones phrase
- Change up the verb
- Change up the object
- Add meaningful modifiers, qualifiers
- Repeat this process
- Think of a new bare-bones phrase which can relate to the skill
Ex: "Password Reset Skill" bare-bones phrases:-
- reset password
- forgot password
- cannot log in
- locked out of account
-
- Think of a new bare-bones phrase which can relate to the skill
- Add variations of this bare-bones phrase
- Some caveats
- Try to avoid 1-word utterances. 1-word utterances can result in lowered behavior, so the models have a safeguard and can only recognize a 1-word utterance as an exact match
- Try to avoid 2-word utterances as well. In general, 2-word utterances are not specific enough and can result in (a) false positives and (b) overlap with phrases of other skills
2. Variables should be used if possible
- Aisera has an entity ontology. The ontology has entity categories and entities. The entity categories are organized in a tree structure.
Ex. Google could have the following path: IT > Applications > Search Engines > Google - Ex. "Troubleshoot Search Engine Skills
-
- Good Variables:
Name: searchEngine
! Variable names should be descriptive and not contain any spaces.
Entity Category: Search Engines
! Annotators should familiarize themselves with every entity & entity synonym when adding a variable. - Good Phrases:
troubleshoot ${searchEngine}
! the structure for a phrase if a variable is introduced: ${variableName} - Bad Phrases:
troubleshoot Google
troubleshoot Bing
troubleshoot DuckDuckGo
- Good Variables:
-
- Why is this important?
-
- Models will return more accurate, explainable results
- Cleaner, fewer phrases
- More maintainable
-
3. Unresolved Conversations should drive improvements
There are two types of unresolved conversations:
-
- Unhelpful conversations
- Knowledge Gaps (No Answer)
- Unhelpful conversations
-
- These are cases where the model returned a skill, but the end user gave feedback that this was not helpful
- If the skill should not have been returned, phrases need to be modified or removed
- If the skill should have been returned, either no change is needed or the fulfillment needs to be improved
-
-
Knowledge Gaps (No Answer)
-
-
These are cases where the model returns nothing
-
If a skill should have been returned, phrases need to be added to the appropriate skill (possibly a new skill)
-
If no skill should have been returned, no change is needed
-
-
4. Test your annotations a lot
Retrain and test regularly in AI Workbench > AI Lens. Publish when ready
5 Get feedback
- The most effective annotators get feedback
- Feedback from constant testing in AI Lens
- Collaboration with other annotators
- Review unresolved conversations and annotate skills by adding similar phrases
- It is important not to create phrases that could overlap between skills, focus on the essence of the skill and what makes it unique.
- Consider different approaches to the skill. For example, for a skill “password reset”, you may want to have phrases like “I can’t log in” , “how to reset my password”, which have different approaches: problem & solution.
- Try to avoid using 1-word phrases or using only general words. Aim for more complete phrases “need help with billing”. Single-word phrases can only be understood by an exact match.
- Avoid words like “please, asap”, other adverbs that aren’t unique to the skill
- More isn’t always better, but a good rule of thumb is to have at least 15 phrases per intent.
- Use Variables.
Variables should be used whenever possible. Instead of writing out a training phrase of “I need Adobe application”, “I need Slack”, it’s better to add a variable (varApplicationName) for the appropriate entity category (e.g., Applications) and write out a training phrase like: “need ${varApplicationName}”.
Additional Notes:
All phrases entered by users go through a spell-check model and would be understood by a bot. That said, training phrases should not have misspellings
Comments
0 comments
Please sign in to leave a comment.