December 2016

Volume 31 Number 13

[Microsoft Bot Framework]

Use Bot Framework for Anytime, Anywhere Access to Application Data

By Srikantan Sankaran

In the last issue I introduced the Microsoft Bot Framework, and discussed how you can take advantage of it to meet a challenge that many organizations face today, which is to give customers interactive access to their services, via voice and text (msdn.com/magazine/mt788623). In the article I described a business scenario that involved consolidating information from different line-of-business applications deployed at an insurance provider organization, using technologies like Microsoft Flow and Azure Logic Apps. The content, comprising both structured and unstructured data, is indexed using Azure Search, and then made available for client applications to consume.

Figure 1, which also appeared in the previous article, depicts the architecture of the solution that implements the scenario.

Figure 1 Architecture of the Solution

In this article, I’m going to build and deploy a bot application that can consume the information gathered via Azure Search. The bot application will be deployed as an Azure Web app, and published through the Microsoft Bot Framework Connector Service. Through the Connector Service, the Bot application will be accessed from the Skype channel, which will give consumers access to the status of their insurance policy requests and let them download the issued policy documents, schedule site inspection visits and so on. Consumers will be able to identify themselves to the bot using the Microsoft account they registered with when applying for an insurance policy, and can use either messaging on the Skype client or voice commands to interact with the bot. The bot application integrates with the LUIS service, one of several services available in Azure Cognitive Services, to interpret conversations from the consumer in order to execute queries on Azure Search.

The steps required to build the solution are explained in the following sequence. The snippets of code are taken from the Visual Studio 2015 solution, which can be downloaded with this article. Here’s what you should do if you’re following along:

• Create and publish a LUIS model that captures the different conversations that need to be supported in the bot, using the UI provided by the LUIS service.
• Use the Visual Studio 2015 template to create a new bot application. You can then either add the code blocks from the solution provided with this article, or follow the rest of the article referring to the downloaded solution file. The MessageController.cs file is the entry point into the bot application, which is a special type of a .NET MVC application.
• Embed the LUIS functionality into the bot, using the LUIS dialog available in the Bot Framework SDK. This is implemented in the PolicyInfoDialog.cs file in the solution.
• Using the Azure Search SDK, implement code that invokes the Azure Search APIs and execute the search requests interpreted by the LUIS service. This is implemented in the SearchService.cs file in the solution.
• Implement a FormFlow dialog to capture the user’s Microsoft account in the Skype channel, in an interactive way, when a user request is made the first time. This is implemented in the SigninForm.cs file in the accompanying solution.
• Use the Azure Search SDK to retrieve the user profile information based on the Microsoft account captured when the user first accesses the bot. This is also implemented in SearchService.cs
• Use the Bot State Service to retain the User Profile information. The bot user’s address and contact information, which is stored in the Bot State Service, is used when scheduling a site inspection visit. This is also implemented in the PolicyInfoDialog.cs file.
• Test the bot application by running it locally as a service on your development machine, and use the Bot Framework Channel Emulator to assess user interaction with the service.
• After testing the bot application locally, deploy it to a Web app in Azure right from Visual Studio.
• Register the bot application with the Microsoft Bot Framework Connector Service and configure in it the URL of the bot application deployed to Azure.
• Add the bot application name, app ID and secret information obtained during registration of the bot into the Visual Studio solution and republish it to Azure.
• Enable the bot application for access from channels like Skype and Slack, from the bot developer Portal.

You can download the Visual Studio 2015 solution for the bot application, and a file export of the LUIS model used in this application from the GitHub repository at bit.ly/2cOfANh.

Creating and Training the LUIS Model for Different Use-Case Scenarios

Figure 2 shows how I mapped out how a user could potentially converse (via a LUIS utterance) with the bot and how this should be interpreted as a specific LUIS intent or action to be performed via Azure Search, and how to extract keywords (LUIS entities) from the LUIS utterance to be used as query parameters when performing Search.

Figure 2 Mapping Utterances and Intents

The LUIS service provides a Web application at luis.ai in which a LUIS application model can be created visually for the problem at hand. Figure 3 displays the LUIS model configured for this scenario.

Figure 3 The LUIS Model for My Scenario

 The intents, entities and utterances identified in the previous step can all be created in the LUIS application. Use the Intents menu to add the intents and the Entities menu to create the entities. Note that there are two special kinds of entities:

• Hierarchical entities: In this case, the “Insurance Policy” is created as a hierarchical entity, the parent, with the various types, Life, Vehicle, Home and so on, created as the children.
• Pre-built entities: I used a pre-built, LUIS-provided entity called Datetime. Input parameters such as the year a policy document was generated, or the date when a site inspection visit is scheduled, will be mapped by the LUIS engine to these pre-built entity types.

The entity with the name “Keyword” will be used to perform a full text search in the policy documents index in Azure Search.

Now, select the New utterances tab and create utterances for each of the conversations identified previously, and label them.

For each utterance, identify or label the intent and the associated entities. For example, In Figure 3, I mapped the utterance, “get me the status of my life insurance policy request,” to the intent “GetPolicyRequestStatusByType,” and mapped the text “Life Insurance” to the hierarchical entity “Insurance:life.”

Repeat this step to add other variations, such as when the policy type is Vehicle, Home or General. Also, consider variations where the same request could be made in a different way. However, you don’t have to capture every permutation and combination of ways the same message might be conveyed. For example, when I add a new utterance with the phrase, “show me the status ...,” as shown in Figure 4, LUIS could rightly interpret that as a variant of the previous one, which used “get me the status ….” LUIS made the right suggestions on the intent and the associated entities, even though the question was asked in a different way.

Figure 4 Labeling LUIS Utterances and Handling Variations

After each utterance is captured, the LUIS application provides the option to Train the Model. This results in the AI engine in LUIS generating optimized models that can then be used by client applications. This step should be repeated whenever new utterances are labeled or changes are made to existing ones.

Publishing the LUIS model allows client-side applications to invoke it, using a REST API. The publish dialog in the application also lets you test the model to ensure it works as expected. Figure 5 shows how to quickly test the LUIS model by passing in a conversation phrase and viewing the output in the browser. (For convenience, Figure 5 shows the snapshots from three different invocation of utterances and the corresponding results side by side.)

Figure 5 Testing the LUIS Model

Notice that the intent recommended by LUIS has the maximum probability value assigned to it, and appears first in the list. The entities that were extracted from the phrase are also returned in the response. If there are multiple entities identified in the utterance, they’re returned, as well. You can see how the LUIS model interprets terms like “this year” in the utterance phrase, maps it to a pre-built entity “Datetime” and returns the value “2016,” which the client application can directly use.

Creating the Bot Application

Now that the LUIS model is ready, it can be used in a bot application. Download the Visual Studio 2015 Template from aka.ms/bf-bc-vstemplate to create the application. (Visit bit.ly/2dYrwyW to learn the basics of building a bot application and how to use a bot emulator to build and test the bot locally.)

The project created when using this template is a variant of an MVC project. The entry point to the project is MessageController.cs, which processes incoming messages from channels like Skype or Slack, and sends back the responses.

The LUISDialog for the Bot Framework is used to implicitly pass messages from the bot to the LUIS model. For each of the intents implemented in the LUIS model, a corresponding method is implemented that can be called directly when the bot application is running. This obviates the need for custom code to call the LUIS REST APIs to interpret the conversation and to identify the intent and entities.

The LUIS application Id and the secret values are used to decorate the class-level attributes, namely LuisModel. The LuisIntent attribute is used on the method that provides the implementation when that intent is encountered in the user request. This is all that’s required to get the LUIS integration going in the bot. Figure 6 depicts how this is implemented in code.

Figure 6 Implementing LUIS Integration

[LuisModel("xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", 
  "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")]
[Serializable]
  public class PolicyInfoDialogs : LuisDialog<object>
  {
    [LuisIntent("")]
    public async Task None(IDialogContext context, LuisResult result)
    {
      // Called when the incoming utterance is not recognized (default bin)
    }
    [LuisIntent("GetPolicyRequests")]
    public async Task GetPolicyRequests(IDialogContext context, LuisResult result)
    {
      //
    }
        ----------------------
      }

At this point, the implementation in each of the methods can be simple; I can simply return a message identifying which LuisIntent implementation was invoked.

In MessageController.cs, add the following snippet to capture the incoming message from the bot user and hand it off to the LuisDialog:

if (activity.Type == ActivityTypes.Message)
  {
    await Conversation.SendAsync(activity, () => new PolicyInfoDialog());
  }

Run the bot application, launch the Bot Framework Channel Emulator, which can be downloaded from aka.ms/bf-bc-emulator. Type in the utterances you trained the LUIS Model for earlier, and ensure that the response message indicates that the correct method mapped to the LUIS intent was invoked.

Search Integration

Now I’ll add the Azure Search SDK for .NET to the Visual Studio solution using the NuGet Package Manager. The SearchService.cs file in the solution uses this SDK and encapsulates all integration with the Azure Search Service. It implements methods that execute queries on Azure Search, return the UserProfile of the customer and retrieve Policy­Requests and PolicyDocuments.

Search commands for the different scenarios are built using the Lucene query syntax, which is implemented in the PolicyInfoDialog.cs file. Here are some examples: 

Retrieve all policy requests for the bot user in the conversation:

command += "CustomerId:(" + msftid + ")";
Retrieve all policy requests for the bot user that haven’t yet been approved:
command += "CustomerId:(" + msftid + ") -PolicyStatus:(Approved)";
Query policy requests for the bot user based on the policy request Id:
public async Task GetPolicyRequestDetails(IDialogContext context, LuisResult result)
  {
    ----------------------
      string  command += "CustomerId:(" + msftid + ") AND PolicyRequestId:(" +
        result.Entities[0].Entity + ")";
    ----------------------
  }

The preceding snippet shows how the PolicyRequestId Entity (the input parameters to execute the search queries) is passed by the LUIS engine through LuisResult (result).

Figure 7 shows how to implement both property-based search and full-text search for documents.

Note that you may want to refer to my previous article in order to relate to the property names and values in the Figure 7 code. Also, you can download the schema of all the tables used in the database accompanying this article using the links provided in later sections. 

Figure 7 Implementing Property-Based Search and Full-Text Search for Documents

foreach (EntityRecommendation curEntity in request.Entities)
  {
    switch (curEntity.Type)
      {
        case "PolicyNumber": // Look for documents with this filename
          {
            command += " filename:(" + request.Entities[0].Entity + ") ";
            break;
          }
        case "keyword": // Search within documents for this keyword expression
          {
            command += " " + request.Entities[0].Entity + " ";
            break;
          }
        case "builtin.datetime.date":
        // Retrieve based on the year a document was generated
          {
            char[] delimiterChars = { '.', ':' };
        List<string> allvalues = curEntity.Resolution.Values.ToList<string>();
             string val = allvalues[0];
             string[] vals = val.Split(delimiterChars);
             command += " lastmoddate:(" + vals[0] + ") ";
             break;
          }
        default:
          {
            break;
          }
        }
    }

Formatting the Response to the User

You can format the text in the response message returned by the bot for style, to add hyperlinks or media content and so forth. This helps make the bot application more user-friendly. Here are a couple of examples …

To set the font style to bold, only for the dynamic data returned from Azure Search, wrap it inside of double asterisks (**):

foreach (SearchResult<PolicyRequest> result in eachGroup)
  {
    searchResponse += (counter) + ". **" + result.Document.PolicyRequestId + "**" +
      " on the insured :**" + result.Document.InsuredIdentifier + "**
      for an amount of :**" +
    result.Document.Currency + " " + result.Document.AssessedValue + "**
      has status: **" +
    result.Document.PolicyStatus + "**\n\n";
    counter++;
  }

To add a hyperlink to a policy document returned by Azure Search, add the Link attribute to the URL of the document:

foreach (SearchResult<PolicyDocument> result in results)
 {
   searchResponse += counter + ". Link: [" + result.Document.filename + "](" +
     result.Document.fileurl + ") \n";
   counter++;
 }

Identifying the Bot User

The Bot Framework provides the user’s channel Id and the display name from the conversation context, but doesn’t provide the user­name used to log into the channel, such as Skype. Hence, this information should be explicitly obtained when a user starts to interact with the bot. The FormFlow dialog described earlier is used to prompt the user for this information. The user profile information is retrieved from Azure Search based on the account name.

To ensure that the user interacting with the bot is the owner of the Microsoft account name shared in the FormDialog, you can inte­grate an interactive login to the Microsoft Account Sign-in page, and provide access to the bot only after authentication and validation. This is out of scope of the article. Instead, you simply prompt the user for the account name using the SigninForm dialog and assume the user is who he claims to be:

public static IForm<SigninForm> BuildForm()
  {
    return new FormBuilder<SigninForm>()
      .Message("I would need some information before we get started.")
      .Field(nameof(MicrosoftAccount))
      .Build();
  }

Storing the User Profile in the Bot State

Once the user is identified during the commencement of a conversation in the bot, a user profile object is retrieved from Azure Search. This information is stored in the bot state, to be used, for example, when the user requests that a site inspection visit be scheduled. The address and contact number of the user is verified during the confirmation of the inspection visit.

The Bot Framework provides an API to store this information in the privateConversationData context, per user and conversation, as shown in Figure 8.

Figure 8 Storing the User Profile

private async Task SignUpComplete(IDialogContext context, IAwaitable<SigninForm> result)
  {
----------------------------------------------
----------------------------------------------
  UserProfile profile =
    await LoadBotUserState(profileIdentityForm.MicrosoftAccount);
  if (profile == null)
  {
  message = $"Sorry, I could not locate your profile in our system.
    Please retry with the right Microsoft Account";
  }
  else
  {
    context.PrivateConversationData.SetValue<bool>("ProfileComplete", true);
    context.PrivateConversationData.SetValue<string>(
      "FirstName", profile.FirstName);
    context.PrivateConversationData.SetValue<string>("LastName", profile.LastName);
    context.PrivateConversationData.SetValue<string>("Address", profile.Address);
    context.PrivateConversationData.SetValue<string>("Phone", profile.Phone);
    context.PrivateConversationData.SetValue<string>(
      "CustomerId", profile.CustomerId);
    message = $"Thanks {profile.LastName},{profile.FirstName}
      for identifying yourself! \n\n
      Let me know how I could assist you.";
  }
    await context.PostAsync(message);
  }
    context.Wait(MessageReceived);
  }

For every request, the bot application checks with the bot state to ensure that the identity of the user issuing the request is verified and the user profile information is available.

Using the Bot Channel Emulator Locally

The bot application requires the Azure Search namespace, access key and the index name to be used to execute the requests from the user. These parameters are stored in the SearchParameters.cs file, and their values set from the web.config file when the bot application is run. Now you can run the Visual Studio solution.

Launch the Bot Channel Emulator, and enter the conversations you trained the LUIS model for. A typical user interaction scenario is captured in Figure 9. Once the bot application runs successfully in the local environment, it can be published to an Azure Web app.

Figure 9 Running the Bot from an Emulator

Deploying the Bot to Azure

Create a Web app in Azure from the Azure Portal, then, in the Application settings, add the appSetting keys and values from the web.config file in the Visual Studio solution.

Before publishing the bot application to an Azure Web app, you need to register this bot at dev.botframework.com/bots/new. This process generates a Bot ID, a Microsoft App ID and a Secret. Add these values to the bot application’s web.config in Visual Studio 2015, and republish this app to Azure.

During the registration process, for the redirect URL parameter, use the SSL-enabled URL of the bot application in Azure and suffix that with “/api/messages.”

Enabling the Bot Across Channels

In the Bot registry at dev.botframework.com/bots, you can enable the bot on other channels. Figure 10 shows the bot application enabled across the Skype and Slack channels. Enabling the bot application for Slack requires some additional steps. A wizard is launched when you choose to add the bot to this channel, which takes you through the appropriate steps.

Figure 10 Registering the Bot on Multiple Channels

Figure 11 shows the bot application when accessed from Skype.

Figure 11 Accessing the Bot from Skype

Wrapping Up

The Microsoft Bot Framework and other supporting technologies like LUIS and Azure Search provide the right foundation to build intuitive bot applications that let customers interact with your line-of-business applications. The way each of these services fit into the overall architecture of the solution ensures that the individual components can be matured or improved independent of the others. For example, the LUIS model can be tweaked to support more utterances that users could use to interact with the bot, while entailing no changes on the core bot application or the content in Azure Search. Once deployed, a bot application can, without any additional code changes, be enabled for access from various channels, and more channels can be added over time, to increase the reach and use of the bot application.

Srikantan Sankaran is a technical evangelist from the DX team in India, based out of Bangalore. He works with numerous ISVs in India and helps them architect and deploy their solutions on Microsoft Azure. Reach him at sansri@microsoft.com.

Thanks to the following Microsoft technical experts for reviewing this article: Sandeep Alur and Hemanth Kathuria
Sandeep Alur is the Lead Evangelist for DX India and is based in Bangalore.

Hemanth Kathuria is a Senior Consultant in the India Services Team

Discuss this article in the MSDN Magazine forum