Target Audience - Annotator

Adding Phrases to Skills (Two Methods):

1. Open Skill Details and add phrases. Can see all details about the skill using this method

2. Open Conversation Details and use the Add to Existing Intent feature to convert the conversation into a training phrase for any skill

Best Practices

1. Phrases should be short and specific
2. Variables should be used if possible
3. Unresolved Conversations should drive improvements
4. Test your annotations a lot
5. Get feedback

1. Phrases should be short and specific

1. Try to write a bare-bones phrase that captures the essence of a skill, but is specific enough that it uniquely identifies this skill
   • Add variations of this bare-bones phrase
     • Change up the verb
     • Change up the object
     • Add meaningful modifiers, qualifiers
       
       
   • Repeat this process
     • Think of a new bare-bones phrase which can relate to the skill
       Ex: "Password Reset Skill" bare-bones phrases:
       • • reset password
         • forgot password
         • cannot log in
         • locked out of account
           
           
2. Some caveats
   • Try to avoid 1-word utterances. 1-word utterances can result in lowered behavior, so the models have a safeguard and can only recognize a 1-word utterance as an exact match
   • Try to avoid 2-word utterances as well. In general, 2-word utterances are not specific enough and can result in (a) false positives and (b) overlap with phrases of other skills

2. Variables should be used if possible

1. Aisera has an entity ontology. The ontology has entity categories and entities. The entity categories are organized in a tree structure.
   Ex. Google could have the following path: IT > Applications > Search Engines > Google
   
   
2. Ex. "Troubleshoot Search Engine Skills
   • • Good Variables:
       Name: searchEngine
       ! Variable names should be descriptive and not contain any spaces.
       
       Entity Category: Search Engines
       ! Annotators should familiarize themselves with every entity & entity synonym when adding a variable.
       
       
     • Good Phrases:
       troubleshoot ${searchEngine}
       ! the structure for a phrase if a variable is introduced: ${variableName}
       
       
     • Bad Phrases:
       troubleshoot Google
       troubleshoot Bing
       troubleshoot DuckDuckGo
       
       
3. Why is this important?
   1. • Models will return more accurate, explainable results
      • Cleaner, fewer phrases
      • More maintainable

3. Unresolved Conversations should drive improvements

There are two types of unresolved conversations:

• • Unhelpful conversations
  • Knowledge Gaps (No Answer)
    
    

     

1. Unhelpful conversations
   • • These are cases where the model returned a skill, but the end user gave feedback that this was not helpful
     • If the skill should not have been returned, phrases need to be modified or removed
     • If the skill should have been returned, either no change is needed or the fulfillment needs to be improved
       
       

2. Knowledge Gaps (No Answer)

   • • These are cases where the model returns nothing

     • If a skill should have been returned, phrases need to be added to the appropriate skill (possibly a new skill)

     • If no skill should have been returned, no change is needed

4. Test your annotations a lot

Retrain and test regularly in AI Workbench > AI Lens. Publish when ready

5 Get feedback

• The most effective annotators get feedback
• Feedback from constant testing in AI Lens
• Collaboration with other annotators
  
  
  
• Review unresolved conversations and annotate skills by adding similar phrases
• It is important not to create phrases that could overlap between skills, focus on the essence of the skill and what makes it unique.
  
• Consider different approaches to the skill. For example, for a skill “password reset”, you may want to have phrases like  “I can’t log in” , “how to reset my password”, which have different approaches: problem & solution.
  
• Try to avoid using 1-word phrases or using only general words. Aim for more complete phrases “need help with billing”. Single-word phrases can only be understood by an exact match.
  
• Avoid words like “please, asap”, other adverbs that aren’t unique to the skill
  
• More isn’t always better, but a good rule of thumb is to have at least 15 phrases per intent.
• Use Variables.
  Variables should be used whenever possible. Instead of writing out a training phrase of “I need Adobe application”, “I need Slack”,  it’s better to add a variable (varApplicationName) for the appropriate entity category (e.g., Applications) and write out a training phrase like: “need ${varApplicationName}”.

Additional Notes:

All phrases entered by users go through a spell-check model and would be understood by a bot. That said, training phrases should not have misspellings