Semantic Kernel: Integrating Conversational AI into Enterprise Apps using DotNet, Python, and Java
Are you a DotNet, Python, or Java developer? Are you looking for a tool to help you incorporate conversational AI into your apps? Have you heard of Semantic Kernel?
In this post, you will learn how to use Semantic Kernel. Along the way, you will get to know how ChatGPT’s internals work in a 10,000-foot view using terms like generative pre-trained transformer (GPT), transformer architecture, self-attention mechanism, large language models (LLM), and natural language processing (NLP).
Afterward, we will move to the features of Semantic Kernel, like plugins, semantic functions, native functions, and planners. I have read the 180-page documentation of Microsoft Semantic Kernel, so you don’t have to.
This article is written by Devlin Duldulao, a senior consultant at the IT consultancy company Inmeta.
Devlin Duldulao is an experienced full-stack developer at Inmeta, recognized as a Microsoft MVP, Certified Trainer, and Azure Developer Associate. With over 10 years of experience, he assists businesses in modernizing and innovating digital services, focusing on Enterprise Cloud, web and mobile development, as well as system design and cloud services. Devlin shares his expertise through courses and talks at national and international conferences. He has also authored three books, including “Spring Boot and Angular,” published in 2023.
Thanks to Contributors:
Alexander Vaagan, Chief Data Scientist at Inmeta
Aurora Walberg, Consultant at Inmeta
… So, let’s get started!
GPT 10,000-foot view
When ChatGPT came on the 30th of November, 2022, it was mind-blowing. It functions similarly to a search engine but with human-like responses. OpenAI, the organization behind ChatGPT, used 175 billion parameters to train ChatGPT version 3.
ChatGPT holds the world record for the fastest application to reach a million users — in just five days. Meanwhile, it took Instagram 2 1/2 months and Spotify 5 months.
Then GPT-4 came on the 14th of March, 2023, with higher accuracy than GPT-3 using approximately 100 trillion parameters. A week later, OpenAI released ChatGPT plugins, which let the AI interpret programming language code and do an internet search before responding to the users. It also has a marketplace allowing businesses to integrate their custom apps.
Let’s talk about the deep learning architecture behind the GPT called transformer. A transformer is an encoder that scales well. The transformer has a parallel multi-head mechanism called attention. Unlike Recurrent Neural Network (RNN) architecture, which cannot be parallelized, the attention mechanism can parallelize the encoding or the processing of input sequences.
The image above is the start of the 2017 paper ‘Attention Is All You Need’ by Ashish Vaswani et al., Google Brain team, proposing the transformer architecture.
The scalability nature of transformer architecture makes it the fundamental building block of all Large Language Models (LLM is a machine learning model designed to generate human-like text based on the input it receives) due to the massive amounts of data the transformer can process.
Consequently, any LLMs that have learned a massive amount of data create outstanding human-like responses due to NLP working on models with billions or trillions of parameters. Like when you ask questions to ChatGPT or send prompts to LLMs.
Value Proposition to Companies and Businesses
So, what’s in it for you and businesses? How will this help B2C or B2B transactions? How do we use NLP and LLMs to bring value to companies and their users?
Aside from ChatGPT plugins, you can build applications like chatbots for customer assistance, content generation, language translation, sentiment analysis, market research analysis, personalization through virtual assistants, interactive voice assistants, e-learning platforms, and more across different industries.
AI Services and LLM Platforms
Now, let’s look at AI services and LLM platforms.
There are several services and platforms out there. We can use OpenAI, Azure OpenAI service, Google Vertex AI, Hugging Face, AWS Bedrock, and more.
But why do we need an AI orchestration SDK or framework to integrate our applications to create solutions instead of using AI services alone?
– With it, you can integrate multiple popular AI services into your apps.
– Learning different AI services’ APIs can be demanding.
– Direct API use doesn’t grant instant access to AI research advancements. For instance, there’s no in-built planning or AI memories in most APIs.
– Open-source solutions like LangChain aim to ease AI app creation.
Semantic Kernel is Microsoft’s offering to facilitate AI integration for enterprise app developers.
It supports DotNet, Python, and Java. While LangChain supports Python and TypeScript. Both are excellent toolkits for developing applications that deal with LLMs.
Getting Started with Semantic Kernel
How do you start with Semantic Kernel? You will need an API key from Azure OpenAI or OpenAI. You must use paid accounts to get these API keys.
Let me show it to you by starting with OpenAI, followed by Azure OpenAI. Have a look at the image below.
The above image shows you the three sub-domains of OpenAI. The ChatGPT, which you might have, the DALL-E for generating images from prompts, and the API for sending prompts to LLMs.
Next, go to OpenAI settings to get your API keys, as shown in the image below.
Click the “Create new secret key” button and copy it for your Semantic Kernel configuration.
Next is for the Azure OpenAI. See image below.
The first step to get the Azure OpenAI keys is to create an Azure OpenAI resource. Then click the “Model Deployments” menu under the resource management section.
Then, in “Models,” choose the gpt-35-turbo for your model deployment, which will create an endpoint for sending HTTP requests. Yes, you are correct that this is not a requirement in OpenAI because they have a default endpoint for all users with OpenAI API accounts. Also, note that the model’s name is gpt 35 and not gpt 3.5.
The image above shows a deployed model. Remember the “deployment name,” in my case, devlinchat, because you will need this later in your Azure OpenAI secret keys. You can go to the playground in the sidebar to check your endpoint and key.
You can then copy your endpoint, which is usually like this: https://whatever-name.openai.azure.com, and the key for later use.
Now you have a key for either OpenAI or Azure OpenAI; it’s time to check out what tools you might need to help you write your app.
Tools for Development
I recommend VS Code when writing your AI application because of some VS Code extensions. These VS Code extensions are:
– Jupyter — can run Python on a basic notebook
– Polyglot Notebooks — can run C# code in Jupyter notebooks
– Semantic Kernel Tools
– Where you can create and run semantic functions boilerplate
– Where you can see different AI service endpoints and switch between them
– Where you can easily switch to other models and see which one works best for you
These VS Code extensions can help you write POCs and plugins, specifically semantic functions, which I will discuss later.
Basic User Prompt Flow
We are almost ready to code, but before that, try to understand how the LLMs' flow takes the users’ inputs and sends back outputs. The flow will help you understand the process behind the semantic kernel, which is similar to other AI frameworks.
Okay. The user sends texts in the input from a frontend application. The backend receives the inputs and passes them as arguments to a function. So, there are two types of functions: a native function and a semantic function. A semantic function is a prompt template that can take input and send it to the AI service. A native function is a function written in a programming language you are using. Why would you need a native function? You will use it if AI models are not fit to answer your prompt, for example, if it includes math. You will write a math function to help language models because they are not good at math.
The input will be converted into a prompt, which the semantic kernel will use to send prompts to large language models of OpenAI, Azure OpenAI, etc. The AI service uses one or more LLM base models, like gpt-3.5 (text-completion-model), embeddings (text-embedding-model), gpt-4 (chat-completion-model), whisper (speech-to-text mode), DALL.E, moderation model, and others.
Lastly, the output from the LLMs will travel as a response to the backend calling them. Then, to the client-side application. Here, the users will read it and probably continue sending requests.
Basic App Development Flow
First, I want to give you an overview of the steps needed to build your AI application. The flow below is very basic, and there are many ways to build an AI application.
A vector database is used to find the most similar or relevant data based on their semantic or contextual meaning. If you have yet to learn what a vector database is, that’s cool because it has a built-in search engine. Vector databases itself is a big topic. I am considering writing a separate article solely for vector databases, including things like embedding and vectors.
Now, let’s move to the next section; semantic function.
Semantic Function
It’s time for the coding part. The GitHub link below includes both DotNet and Python code samples:
https://github.com/webmasterdevlin/semantic-kernel-inmeta-demo
We will create a DotNet core console app to simplify everything, but you should use a backend service to hide our secret keys when our application sends HTTP requests to LLM services. You can use any frontend framework and then use Fast API for Python, ASP.NET Core for DotNet, Spring Boot for Java, or a serverless function with, for example, Azure Functions to process the inputs of the client-side application. Then, attach the API keys before sending them to the OpenAI or Azure OpenAI service. Again, we will simplify this by using a DotNet console app and a Python app.
Create a DotNet Core console app and add the following packages:
· Microsoft.Extensions.Configuration
· Microsoft.Extensions.Configuration.UserSecrets
· Microsoft.Extensions.Logging.Console
· Microsoft.Extensions.Logging.Debug
· Microsoft.SemanticKernel
After installing the above packages, let’s create a helper tool for loading our env vars for sensitive information like API keys. Name the folder config, name the file Env.cs, and then write the code below.
Using Microsoft.Extensions.Configuration;
internal sealed class Env
{
internal static string? Var(string name)
{
var configuration = new ConfigurationBuilder()
.AddUserSecrets<Env>()
.Build();
var value = configuration[name];
if (!string.IsNullOrEmpty(value))
{
return value;
}
value = Environment.GetEnvironmentVariable(name);
return value;
}
}After writing a C# class for loading environment variables, create another helper tool. This helper tool, which will be a kernel builder extension, automatically chooses between the text or chat completion service. It will also choose between OpenAI or Azure OpenAI service. We can do this by writing a C# file inside the config folder. Name the file KernelBuilderExtension.cs and add the code below.
Using Microsoft.SemanticKernel;
internal static class KernelBuilderExtensions
{
internal static KernelBuilder WithCompletionService(this KernelBuilder kernelBuilder)
{
switch (Env.Var("Global:LlmService")!)
{
case "AzureOpenAI":
if (Env.Var("AzureOpenAI:DeploymentType")! == "text-completion")
{
kernelBuilder.WithAzureTextCompletionService(deploymentName: Env.Var("AzureOpenAI:TextCompletionDeploymentName")!, endpoint: Env.Var("AzureOpenAI:Endpoint")!, apiKey: Env.Var("AzureOpenAI:ApiKey")!);
}
else if (Env.Var("AzureOpenAI:DeploymentType")! == "chat-completion")
{
kernelBuilder.WithAzureChatCompletionService(deploymentName: Env.Var("AzureOpenAI:ChatCompletionDeploymentName")!, endpoint: Env.Var("AzureOpenAI:Endpoint")!, apiKey: Env.Var("AzureOpenAI:ApiKey")!);
}
break;
case "OpenAI":
if (Env.Var("OpenAI:ModelType")! == "text-completion")
{
kernelBuilder.WithOpenAITextCompletionService(modelId: Env.Var("OpenAI:TextCompletionModelId")!, apiKey: Env.Var("OpenAI:ApiKey")!, orgId: Env.Var("OpenAI:OrgId"));
}
else if (Env.Var("OpenAI:ModelType")! == "chat-completion")
{
kernelBuilder.WithOpenAIChatCompletionService(modelId: Env.Var("OpenAI:ChatCompletionModelId")!, apiKey: Env.Var("OpenAI:ApiKey")!, orgId: Env.Var("OpenAI:OrgId"));
}
break;
default:
throw new ArgumentException($"Invalid service type value: {Env.Var("OpenAI:ModelType")}");
}
return kernelBuilder;
}
}Now that we have these helpers, we can move to writing or generating semantic functions. As I said earlier, the semantic kernel has two types of functions. One is a “semantic function,” a templated prompt with or without parameters; the other is called a “native function.”
Let’s make our first semantic function. Open the DotNet console app project you created in VS Code, then click the Semantic Kernel Tools extension. You must first provide an OpenAI key by clicking the button shown in the image below.
The SK extension’s AI endpoints can help you quickly switch between endpoints and models. They are beneficial for knowing which model works for your application.
After entering your OpenAI key or Azure OpenAI key, you can create a semantic function by clicking “Get Started”, as shown in the image below.
Click the “Create your first semantic function”, create a folder, and then name the folder plugins. This folder will contain all the different plugins in your project. A plugin is a function in the semantic kernel. It used to be called Skill, but the SK team changed it to make it the same as the OpenAI plugin because they have the same concept.
Create another folder inside the plugins folder and name it according to the purpose of your function. Let’s say the folder for our first semantic function will have FoodPlugin name because we will create multiple functions that are related to food. Select this folder, then name your function DishRecipe and give a description like Show the recipe of the dish that is being asked. It should look like the image below after you have generated the semantic function.
The generated function has a config.json file for controlling the behavior of the function and a skprompt.txt file, which is a function written in a natural language. The content of the file will be a template for your prompt. The JSON object inside the config.json file looks like this:
Now let’s edit the skprompt.txt file with the code below that asks the recipe of a dish.
Give me the recipe of the {{$input}} dish elaborately.
[Format]
How to make {{$input}} dish
Ingredients:
Steps:
_______________________________________
Say you are not familiar with the dish if you don’t know the dish.
In the above code, you will find an enclosed double-curly brace that signals that this is a parameter. It says to give a recipe in an elaborate way, with the format, and tells the model to respond, “I am not familiar with the dish,” if the model does not know the dish. You can try it by pressing the play button shown in the image below.
The VS Code editor will prompt you to input the name of a dish and send it to the AI service. The response includes information such as AI provider, model, chat system, which function was executed, parameters, prompt, tokens, and duration of the prompt. This way, you can test prompts without using any programming language and talk to an API endpoint programmatically. The SK extension does everything for you.
Before we move to another section, try rerunning the DishRecipe and enter a made-up dish like abcd to see if the model will come up with something it doesn’t know, resulting in misleading or incorrect information. This is also known as hallucination.
We can now write code in the DotNet core console app and use the semantic function we generated earlier. Go to the Program.cs file and add the code below:
using Microsoft.SemanticKernel;
var kernel = new KernelBuilder().WithCompletionService().Build();
var pluginsDirectory = Path.Combine(Directory.GetCurrentDirectory(), "plugins");
var foodPlugin = kernel.ImportSemanticSkillFromDirectory(pluginsDirectory, "FoodPlugin");
var query = "adobo";
var recipe = await kernel.RunAsync(query, foodPlugin["DishRecipe"]);
Console.WriteLine("Recipe helper results:");
Console.WriteLine(recipe)In the code above, we are importing the Semantic Kernel Nuget package. We then instantiate a new kernel instance using the KernelBuilder and add the environment variables using the KernelBuilder extension. After that, we import the folder FoodPlugin using the ImportSemanticSkillFromDirectory.
Then we write our query, “adobo”, a popular Filipino dish, and send it to the AI service. Take note there that we are getting the semantic function by doing an index accessor or bracket notation.
Lastly, before running the application, let’s add our secrets to the environment variables by running the terminal commands below.
For Azure OpenAI users:
dotnet user-secrets set “Global:LlmService” “AzureOpenAI”
dotnet user-secrets set “AzureOpenAI:DeploymentType” “chat-completion”
dotnet user-secrets set “AzureOpenAI:ChatCompletionDeploymentName” “your-own-deployment-name”
dotnet user-secrets set “AzureOpenAI:Endpoint” “… your Azure OpenAI endpoint …”
dotnet user-secrets set “AzureOpenAI:ApiKey” “… your Azure OpenAI key …”
For OpenAI users:
dotnet user-secrets set “Global:LlmService” “OpenAI”
dotnet user-secrets set “OpenAI:ModelType” “chat-completion”
dotnet user-secrets set “OpenAI:ChatCompletionModelId” “gpt-3.5-turbo”
dotnet user-secrets set “OpenAI:ApiKey” “… your OpenAI key …”
dotnet user-secrets set “OpenAI:OrgId” “… your ord ID …”
After adding all the secrets, we can try the application by running dotnet run in the terminal. You should see the result of the AI service response in the terminal.
Now for the Python version of the code, it looks like this:
import config.add_completion_service
import semantic_kernel as sk
async def main():
kernel = sk.Kernel()
kernel.add_completion_service()
plugins_directory = "./plugins"
food_plugin = kernel.import_semantic_skill_from_directory(plugins_directory, "FoodPlugin")
query = "bulalo"
recipe = await kernel.run_async(food_plugin["DishRecipe"], input_str=query)
print("Recipe helper results:")
print(recipe)The only difference in the above code is the “adobo” query to the “bulalo” query, another popular dish in the Philippines. Now, let’s move to writing a native function.
Native Function
We did a lot when creating our first semantic function; the good news is that we can write our native function with less setup since we did some of it in the previous section of this post.
Now go to your plugins directory, create a folder, and name it MathPlugin. Create a C# file and name it MathPlugin.cs with the code below:
using System.ComponentModel;
using System.Globalization;
using Microsoft.SemanticKernel.Orchestration;
using Microsoft.SemanticKernel.SkillDefinition;
namespace SemanticKernelDemo.plugins.MathPlugin;
public class Math
{
[SKFunction, Description("Add two numbers")]
[SKParameter("input", "The first number to add")]
[SKParameter("number2", "The second number to add")]
public string Add(SKContext context)
{
return (
Convert.ToDouble(context.Variables["input"], CultureInfo.InvariantCulture) +
Convert.ToDouble(context.Variables["number2"], CultureInfo.InvariantCulture)
).ToString(CultureInfo.InvariantCulture);
}
[SKFunction, Description("Subtract two numbers")]
[SKParameter("input", "The first number to subtract from")]
[SKParameter("number2", "The second number to subtract away")]
public string Subtract(SKContext context)
{
return (
Convert.ToDouble(context.Variables["input"], CultureInfo.InvariantCulture) -
Convert.ToDouble(context.Variables["number2"], CultureInfo.InvariantCulture)
).ToString(CultureInfo.InvariantCulture);
}
[SKFunction, Description("Multiply two numbers. When increasing by a percentage, don't forget to add 1 to the percentage.")]
[SKParameter("input", "The first number to multiply")]
[SKParameter("number2", "The second number to multiply")]
public string Multiply(SKContext context)
{
return (
Convert.ToDouble(context.Variables["input"], CultureInfo.InvariantCulture) *
Convert.ToDouble(context.Variables["number2"], CultureInfo.InvariantCulture)
).ToString(CultureInfo.InvariantCulture);
}
[SKFunction, Description("Divide two numbers")]
[SKParameter("input", "The first number to divide from")]
[SKParameter("number2", "The second number to divide by")]
public string Divide(SKContext context)
{
return (
Convert.ToDouble(context.Variables["input"], CultureInfo.InvariantCulture) /
Convert.ToDouble(context.Variables["number2"], CultureInfo.InvariantCulture)
).ToString(CultureInfo.InvariantCulture);
}
}In the class in the above code, you will notice four mathematical operators. Each operator is a native function with attributes that provide information about it, such as name, description, and parameters.
Why are we writing native functions for the AI service to solve mathematical problems for us? So, you see, we are using an AI service, but our model is a large language model that has natural language processing capability. LLMs are not by default good at math but at languages, which is why we are creating functions that execute mathematical operators. Every method needs to have a description of what it can do and some details about the parameters.
Now, let’s update the Program.cs, where we can use the mathematical operators. We are also going to use a feature in Semantic Kernel that can take a prompt and create a plan on how to accomplish the request by combining different plugins or functions together. The feature that we are going to use next is called Planner.
using System.Text.Json;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Planning;
using Math = SemanticKernelDemo.plugins.MathPlugin.Math;
var kernel = new KernelBuilder().WithCompletionService().Build();
kernel.ImportSkill(new Math(), "MathPlugin");
var planner = new SequentialPlanner(kernel);
var ask = "My monthly gross salary is 9000 USD and the tax is 35 percent, how much would I have every month if I have a monthly food cost of 1300 USD and 900 USD for rent?";
var plan = await planner.CreatePlanAsync(ask);
Console.WriteLine("Plan object:\n");
Console.WriteLine(JsonSerializer.Serialize(plan, new JsonSerializerOptions { WriteIndented = true }));
var result = (await kernel.RunAsync(plan)).Result;
Console.WriteLine("Plan results:");
Console.WriteLine(result.Trim());We are now adding two more Nuget packages, namely System.Text.Json and Microsoft.SemanticKernel.Planning. We then import the MathPlugin and create an instance of a Planner.
As you can see in the code above, after importing the plugin, we can write a math problem that uses addition, subtraction, multiplication, and division.
Next is to log the plan of the planner so we can see how it arranges the sequence of different native functions by itself.
Run the plan using the kernel to log the result. You should see the plan object in the console showing that the AI’s plan is to multiply 9000 x 0.35 first. Then do a subtraction of 9000–3150. Then add 1300 to 900. Then again, subtract 5850–2200. And the result will be 3650.
The Planner works, and the model can understand how to solve the math question of multiple steps by using operators in sequence.
And here is the Python version of the Planner code:
import config.add_completion_service
import semantic_kernel as sk
from plugins.MathPlugin.Math import Math
from semantic_kernel.planning.basic_planner import BasicPlanner
async def main():
kernel = sk.Kernel()
kernel.add_completion_service()
kernel.import_skill(Math(), "MathPlugin")
ask = "My monthly gross salary is 9000 USD and the tax is 35 percent, how much would I have every month if I have a monthly food cost of 1300 USD and 900 USD for rent?";
planner = BasicPlanner()
plan = await planner.create_plan_async(ask, kernel)
result = await planner.execute_plan_async(plan, kernel)
print("Plan results:")
print(result)You will notice that the Planner code is not complicated to write in Python, and the interfaces are almost identical.
Now, let’s move to the next section.
Extras
There are more features that we didn’t go through here, like memories, connectors, external ChatGPT plugins, and more. I plan to do all the other features in the future posts related to Semantic Kernel.
Summary
Here, I will quickly list what we learned in this post.
We have learned that a transformer is a very efficient architecture for training models. We also have gotten to know that LLMs or large language models have a massive number of parameters, up to trillions of parameters depending on the models. We understood that NLP, or natural language processing, lets computers chat with us with human-like responses.
In the coding part, we discovered that Semantic Kernel is an SDK that can orchestrate prompts and send them to AI services with LLMs. And lastly, we have learned that you can use C#, Python, and Java programming languages in Semantic Kernel to build AI-first applications.
References
The links below helped me write this post. A big shout out also to the large Semantic Kernel community on Discord, which is very helpful to everyone working or playing with Semantic Kernel.
• linkedin.com/learning/building-skills-with-semantic
