Azure Computer Vision and Open AI

Imagine snapping a photo of your grocery haul and instantly receiving creative meal ideas tailored to what’s in the bag. In this post, we’ll walk through how to build a .NET Core API does exactly that — using Azure Vision’s Dense Captions to detect grocery items from an image, and Azure Open AI’s GPT-4.1 to generate meal suggestions.

The Vision + Language Pipeline

This API combines two powerful Azure services:
• Azure Vision (Image Analysis v4.0): Extracts descriptive captions from an image using Dense Captions.
• Azure OpenAI (GPT-4.1): Generates meal ideas based on the detected items.

Step 1: Azure Setup: 

Before writing a single line of code, you’ll need to provision two Azure resources:

1. Cognitive Services — Computer Vision

Steps:
• Go to Azure Portal
• Click Create a resource
• Search for Computer Vision
• Select the result titled Computer Vision (Microsoft)
• Click Create
• Fill in:
• Resource name:
• Region: Must be one that supports Image Analysis v4.0 (e.g., East US, West Europe)
• Pricing tier: Standard S0 or F0 (free tier for testing)
• Click Review + Create, then Create
Once deployed:
• Go to the resource
• Click Keys and Endpoint
• Copy your Key and Endpoint

2. Azure OpenAI

This resource enables access to GPT models.
Steps:
• Go to Azure Portal
• Click Create a resource
• Search for Azure OpenAI
• Select Azure OpenAI (Microsoft)
• Click Create
• Fill in:
• Resource name: e.g., 
• Region: Must be one that supports GPT-4.1 (e.g., East US, South Central US)
• After deployment, go to the resource
• Click Keys and Endpoint
• Copy your Key and Endpoint
• Deploy a model (e.g., GPT-4.1) under Model Deployments

Step 2: Detect Grocery Items from an Image

The endpoint uses Azure Vision to analyze a grocery image:

ImageAnalysisResult result = await client.AnalyzeAsync(imageURL, VisualFeatures.DenseCaptions, options);

Instead of using object detection or tags, this implementation leverages Dense Captions, which provide rich, localized descriptions of image regions. These captions are then distilled into a list of unique grocery-related phrases:

var groceryItems = result.DenseCaptions.Values
    .Select(obj => obj.Text)
    .Distinct()
    .ToList();

This approach is especially useful for cluttered grocery images where traditional object detection might miss context.

Step3: Generate Meal Suggestions with Azure OpenAI

Once the grocery items are extracted, they’re passed to the method, which uses Azure OpenAI to generate meal ideas:

var messages = new List<ChatMessage>
{
    new SystemChatMessage("You are a culinary assistant that suggests creative meals based on grocery items."),
    new UserChatMessage("suggest meals based on grocery items: " + itemsString),
};

The GPT-4.1 model is prompted with a conversational context .
The result is parsed and formatted for clean display:

string formattedText = firstContent.Text
    .Replace("\r\n", Environment.NewLine)
    .Replace("\n", Environment.NewLine);

Below is the full code snippet:

[HttpGet("analyzeImageMealsSuggestion")]
public async Task<IActionResult> AnalyzeImageForMealsSuggestion(string image)
{
    string endpoint = "Your endpoint";
    string key = "Your Key";

    var client = new ImageAnalysisClient(new Uri(endpoint), new AzureKeyCredential(key));

    // Use image from URL or uploaded file
    Uri imageURL = new Uri(image);
    //"https://media.istockphoto.com/id/171302954/photo/groceries.jpg?s=2048x2048&w=is&k=20&c=IC6Zb7kobTrpffJS6eQ2YAOcXmV4OItV3afi_3OSaEU="

    var options = new ImageAnalysisOptions
    {
        Language = "en",
        GenderNeutralCaption = true
    };

    // Analyze image for objects
    ImageAnalysisResult result = await client.AnalyzeAsync(imageURL, VisualFeatures.DenseCaptions, options);

    var groceryItems = result.DenseCaptions.Values
        .Select(obj => obj.Text)
        .Distinct()
        .ToList();            

    // Optional: Pass to OpenAI for meal suggestions
     var meals = await _mealService.SuggestMealsAsync(groceryItems);

    return Ok(new
    {
        items = groceryItems,
         mealsfinal = meals
    });
}


    public async Task<string> SuggestMealsAsync(List<string> items)
    {
        // Retrieve the OpenAI endpoint from environment variables
        var endpoint = AzureAIConstants.AzureOpenAIEndpoint;


        var key = AzureAIConstants.AzureOpenAIApiKey;

        string itemsString= string.Join(", ", items);

        AzureKeyCredential credential = new AzureKeyCredential(key);

        // Initialize the AzureOpenAIClient
        AzureOpenAIClient azureClient = new(new Uri(endpoint), credential);

        // Initialize the ChatClient with the specified deployment name
        ChatClient chatClient = azureClient.GetChatClient("gpt-4.1");

        // Create a list of chat messages
        var messages = new List<ChatMessage>
        {
         new SystemChatMessage(@"You are a culinary assistant that suggests creative meals based on grocery items."),
          new UserChatMessage(@"suggest meals based on grocery items:"+itemsString),
        };


        // Create chat completion options

        var options = new ChatCompletionOptions
        {
            Temperature = (float)1,
            MaxOutputTokenCount = 800,

            TopP = (float)1,
            FrequencyPenalty = (float)0,
            PresencePenalty = (float)0
        };

        try
        {
            // Create the chat completion request
            ChatCompletion completion = await chatClient.CompleteChatAsync(messages, options);

            // Print the response
            if (completion != null)
            {
                //return System.Text.Json.JsonSerializer.Serialize(completion.Content.FirstOrDefault(), new JsonSerializerOptions() { WriteIndented = true });
                var firstContent = completion.Content.FirstOrDefault();
                if (firstContent != null && firstContent.Text != null)
                {
                    // Replace "\n" with Environment.NewLine for proper line breaks                        
                    string formattedText = firstContent.Text
.Replace("\r\n", Environment.NewLine) // Replace Windows-style line breaks
.Replace("\n", Environment.NewLine);
                    //\n\n
                    //formattedText = formattedText.Replace("\n\n", Environment.NewLine);
                    return formattedText; // Return the formatted text
                }
                else
                {
                    return "No valid text content found.";
                }
            }
            else
            {
                return "No response received.";
            }
        }
        catch (Exception ex)
        {
            return ex.Message;
        }
    }

Step 3: Test and Analyze the API:

With the API built, the final step is to test its functionality.

The article uses Postman to upload a sample image containing few grocery items. The API then returns a response from the Azure Computer Vision and Azure Open AI service, providing a detailed breakdown of the items detected and the recipes which can be prepared. This confirms that the service is successfully identifying and categorizing potentially recipes.

Below is the screen shot:

Here we can see the response and the image which we have used for the sample.

Below is the response:

{
    "items": [
        "a group of food items",
        "a watermelon and a banana",
        "a close-up of a red apple",
        "a red can with a red lid",
        "a close-up of bananas and watermelon",
        "a white container with a lid",
        "a close-up of a box",
        "a bunch of spaghetti tied with a red band",
        "close-up of a bottle of liquid"
    ],
    "mealsfinal": "Based on the food items you listed, 
                   here are creative meal ideas you can prepare:\r\n\r\n### 
1. **Fruit Salad with a Twist**\r\n- 
**Ingredients:
** Watermelon, banana, apple\r\n- 
**Instructions:
** Chop watermelon, banana, and apple into bite-sized pieces. Combine in a bowl. Optional: Drizzle with a little honey (if the bottle of liquid is honey) or a splash of juice for extra flavor. Chill before serving for a refreshing snack or dessert.\r\n\r\n### 
2. **Spaghetti with Simple Tomato Sauce**\r\n- 
**Ingredients:
** Spaghetti, red can (likely tomato sauce), bottle of liquid (possibly olive oil or vinegar)\r\n- 
**Instructions:
** Cook spaghetti as per package instructions. In a pan, heat some of the bottled liquid (if olive oil) and add the contents of the red can (tomato sauce). Simmer, season with salt, pepper, and any herbs you have. Toss with cooked spaghetti.\r\n\r\n### 
3. **Fruit Breakfast Yogurt Parfait**\r\n- 
**Ingredients:
** Banana, apple, watermelon, white container (possibly yogurt)\r\n- 
**Instructions:** In a glass or bowl, layer yogurt from the white container with sliced fruit. Top with a drizzle of honey (if that's the liquid), nuts, or granola if available.\r\n\r\n### 
4. **Sweet Watermelon-Banana Smoothie**\r\n- 
**Ingredients:** Watermelon, banana, apple (optional), bottle of liquid (if juice, milk, or coconut water)\r\n- 
**Instructions:** Blend watermelon chunks, banana, and peeled apple with a splash of the liquid. Pour over ice for a cooling drink.\r\n\r\n### 
5. **Pan-Fried Apple-Banana Topping for Pasta (Sweet Dessert Pasta)**\r\n- 
**Ingredients:
** Spaghetti, banana, apple, bottle of liquid (use if olive oil, butter, or honey)\r\n- 
**Instructions:
** Cook spaghetti and drain. In the same pot/pan, heat a small amount of the bottled liquid (olive oil or butter). Add chopped apples and bananas, sauté for 3-5 minutes until tender. Toss fruit with pasta, drizzle a little honey (if available) on top for a unique sweet dessert pasta.\r\n\r\n---\r\n\r\n
**Tip:** The contents of the close-up box and specifics of the white container or bottle of liquid may open up more options! Let me know what's inside for tailored recipes."
}