# Azure OpenAI, meet Canopy

> Overview of Canopy's new integration with Azure OpenAI Studio.

Audrey Sage · 2024-02-15

[Canopy](https://www.pinecone.io/blog/canopy-rag-framework/), the [open-source RAG framework](https://github.com/pinecone-io/canopy) from Pinecone, is now compatible with [Azure OpenAI Studio](https://azure.microsoft.com/en-us/products/ai-services/openai-service).

![](https://cdn.sanity.io/images/vr8gru94/production/5366771330dc6a2b48e5ba37d5e2b21a81b97bdb-1740x592.png)


By integrating with Azure OpenAI Studio, Canopy users are able to [chat with their documents](https://github.com/pinecone-io/canopy?tab=readme-ov-file#evaluation-chat-tool) and [launch RAG applications into production](https://github.com/pinecone-io/canopy?tab=readme-ov-file#running-canopy-server-in-production) knowing their models are backed by the [enterprise-ready features](https://azure.microsoft.com/en-us/products/ai-services/openai-service/?ef_id=_k_Cj0KCQiAwvKtBhDrARIsAJj-kTi5-uFHJB62-oV0M_1oPu0Ih3PDrKFGaZ1f05JPlfaGyrv7szVLZggaAn9SEALw_wcB_k_&OCID=AIDcmm5edswduu_SEM__k_Cj0KCQiAwvKtBhDrARIsAJj-kTi5-uFHJB62-oV0M_1oPu0Ih3PDrKFGaZ1f05JPlfaGyrv7szVLZggaAn9SEALw_wcB_k_&gad_source=1&gclid=Cj0KCQiAwvKtBhDrARIsAJj-kTi5-uFHJB62-oV0M_1oPu0Ih3PDrKFGaZ1f05JPlfaGyrv7szVLZggaAn9SEALw_wcB#ProductOverview) of Microsoft Azure.

## Canopy

Canopy is an open-source, production-ready RAG framework that allows you to build and productionize RAG applications easily. Canopy comes with a CLI for rapid iteration of proof-of-concepts and [various options](https://github.com/pinecone-io/canopy?tab=readme-ov-file#running-canopy-server-in-production) for launching your final product into production.



By default, Canopy uses OpenAI’s [text-embedding-3-small](https://platform.openai.com/docs/guides/embeddings/embedding-models) as its embedding model and OpenAI’s[ gpt-3.5-turbo](https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo) as its LLM. However, Canopy is compatible with any publicly available embedding model or LLM from [Anyscale](https://github.com/pinecone-io/canopy/blob/v0.6.0/config/anyscale.yaml), [Cohere](https://github.com/pinecone-io/canopy/blob/main/src/canopy/config_templates/cohere.yaml), [OpenAI](https://github.com/pinecone-io/canopy/blob/main/src/canopy/config_templates/default.yaml), and, now, [Azure OpenAI Studio](https://github.com/pinecone-io/canopy/blob/main/src/canopy/config_templates/azure.yaml).

## Azure OpenAI

Azure OpenAI Studio is a cloud service that allows you to deploy your own instances of OpenAI’s powerful models through Microsoft Azure. By accessing OpenAI models through Azure, you can host private instances of OpenAI models, easily fine-tune those models, and keep the full lifecycle of your work in a corporate, enterprise-grade platform.



**Azure OpenAI vs OpenAI**

While using OpenAI models directly is easy and beginner-friendly, OpenAI does not offer the security controls and enterprise-ready features that a service like Azure does. Some of the differences between the two services are outlined below:

|                        | Azure OpenAI                                      | OpenAI                      |
|------------------------|---------------------------------------------------|-----------------------------|
| **Auth**                   | API keys, Azure account, MFA                      | API keys                    |
| **Data security**          | [Encrypted](https://learn.microsoft.com/en-us/azure/ai-services/openai/encrypt-data-at-rest) at rest and in transit                  | Unclear; [personal data is accessible](https://openai.com/policies/privacy-policy) to an extent |
| **Compliance**             | [HIPAA](https://learn.microsoft.com/en-us/answers/questions/1245418/hipaa-compliance), [FedRAMP](https://devblogs.microsoft.com/azuregov/azure-openai-service-achieves-fedramp-high-authorization/), [GDPR](https://learn.microsoft.com/en-us/legal/gdpr), [ISOs, SOC 1/2/3](https://learn.microsoft.com/en-us/azure/compliance/offerings/offering-soc-2), [etc](https://techcommunity.microsoft.com/t5/azure-architecture-blog/security-best-practices-for-genai-applications-openai-in-azure/ba-p/4027885).       | [GDPR, SOC 2/3, CCPA](https://trust.openai.com/?itemUid=45220873-6e51-4dbb-b1b1-37d66ee9ef95&source=click)         |
| **General security**       | [Threat detection, network security](https://learn.microsoft.com/en-us/security/benchmark/azure/baselines/azure-openai-security-baseline), security audits, [VNETs and private endpoints](https://learn.microsoft.com/en-us/azure/ai-services/cognitive-services-virtual-networks?context=%2Fazure%2Fai-services%2Fopenai%2Fcontext%2Fcontext&tabs=portal), [RBAC](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/role-based-access-control) | Unclear; see [Privacy Portal](https://privacy.openai.com/policies). |
| **Recovery **              | Infra has built-in [disaster recovery and backups](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/business-continuity-disaster-recovery)  | N/A                         |
| **SLAs**                   | [Azure Cognitive Services SLA](https://azure.microsoft.com/support/legal/sla/cognitive-services/v1_1/)                      | N/A                         |
| **Fine tuning**            | [Yes](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/fine-tuning?pivots=programming-language-studio&tabs=turbo%2Cpython)                                               | No                          |
| **Rate limits**            | Azure OpenAI [rate limits](https://learn.microsoft.com/en-us/azure/ai-services/openai/quotas-limits)                          | OpenAI [rate limits](https://platform.openai.com/docs/guides/rate-limits/usage-tiers?context=tier-free)          |
*Note: the above table is a cursory summary; do your own research before taking action from this information.*




For enterprise users, one of the chief motivations for using Azure OpenAI Studio is that all model calls stay within the user’s Azure ecosystem – this helps maintain privacy, allows for customization, and simplifies security and compliance.

Note: access to Azure OpenAI Studio is limited. You need to [apply](https://customervoice.microsoft.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbR7en2Ais5pxKtso_Pz4b1_xUNTZBNzRKNlVQSFhZMU9aV09EVzYxWFdORCQlQCN0PWcu) to use the service.

## Build with Canopy and Azure OpenAI Studio

The best way to get a feel for how Canopy works with Azure OpenAI Studio is to build with it! Follow along in [this notebook](https://github.com/pinecone-io/examples/tree/master/learn/generation/canopy/canopy-azure-openai-studio) to see Canopy in action.



The demo notebook linked above takes you through the process of building an end-to-end RAG application with Canopy and Azure OpenAI Studio. You will create a Pinecone account, apply for Azure OpenAI Studio access, deploy Azure models, and build individual Canopy components to complete your RAG application.



![](https://cdn.sanity.io/images/vr8gru94/production/5de8965a353e1c39fb3322e46bef4f896ae46a70-2448x1274.png)


Above is a preview of your final product: a RAG application that answers questions based on the context retrieved from your Canopy index.

## The Canopy CLI

One of the most exciting things about Canopy is its CLI. Engineers use the CLI during the proof-of-concept phase of the RAG application development lifecycle.



Through the `canopy chat` command, you can chat back and forth with the documents in your Canopy index. The `ChatEngine`’s answers will differ according to the parameters you set in a configuration file.



You can also use the CLI to quickly build a Canopy index (`canopy new`), upsert documents into that Canopy index (`canopy upsert <file path to data file>`), and start up the Canopy server (`canopy start`).



When using the CLI commands with Azure OpenAI Studio, you will need to modify the [appropriate configuration file](https://github.com/pinecone-io/canopy/blob/v0.6.0/config/azure.yaml) (`azure.yaml` in this case).



**Config modifications**

The first modification you’ll make to the example Azure configuration file is replacing dummy text in the `LLM` section with the name of your Azure-deployed LLM. This will be the LLM’s “deployment name,” _not_ the name of the underlying OpenAI model. For instance, I could deploy a model named “canopy-azure-llm” whose underlying LLM is “gpt-35-turbo.” The “deployment name” that I would put in my config file would be “canopy-azure-llm,” not “gpt-35-turbo.”



![](https://cdn.sanity.io/images/vr8gru94/production/a8f3ae7fd0a2fca51749712576c25777d0a7a7b9-1928x564.png)




![](https://cdn.sanity.io/images/vr8gru94/production/d52b4c7402f814ae23ac889767b5bd420a9f1328-2316x974.png)


The next thing you’ll do is replace more dummy text, this time in the `ContextEngine` section, with your Azure-deployed embedding model’s deployment name:

![](https://cdn.sanity.io/images/vr8gru94/production/f1b026cddf011dddadcaf16820e66400d3599c08-1930x318.png)


You can continue modifying the example Azure config file to your heart’s content. Replacing the two instances of dummy text outlined above, though, is mandatory to start using the CLI.



You can use these configuration files programmatically in notebooks and scripts, too; see the “Load from config section” in the [demo notebook](https://github.com/pinecone-io/examples/tree/master/learn/generation/canopy/canopy-azure-openai-studio).



**Canopy Chat**

With your Azure configuration file modified and saved, you can spin up the canopy server, pointing it to your Azure config, by executing `canopy start --config=canopy/config/azure.yaml` from within the `src` directory. You’ll see the Uvicorn server start on your local machine.



Note: you’ll need to set [the appropriate environment variables](https://github.com/pinecone-io/canopy?tab=readme-ov-file#mandatory-environment-variables) for this step to work.



![](https://cdn.sanity.io/images/vr8gru94/production/4b26683e3ebf4e17ced406bdfc4f196836fa538c-2200x748.png)


To chat with the docs in your Canopy index, open a new terminal window (ensure your environment variables are still set in this window). In this new window, execute `canopy chat --no-rag`. It’s fun to include the `--no-rag` flag when chatting with your documents because it shows you how your LLM’s answer changes with and without the addition of the context fetched from your Canopy index.



Note that in order to showcase the `no-rag` option, you’ll need to set your OpenAI API key environment variable (`OPENAI_API_KEY`).



![](https://cdn.sanity.io/images/vr8gru94/production/c4fec71e4eb3bdb177ba8e20b9666cb539ad68e0-2722x1250.png)


You can see above that without the context from the documents in your Canopy index, your LLM has no way of knowing which "Aitchison" you are referencing in your question!

## Stay in touch

Integrating with enterprise-grade hosting platforms like Azure OpenAI Studio is just the beginning for Canopy.  Check out Canopy's latest official releases ([7](https://github.com/pinecone-io/canopy/releases/tag/v0.7.0) and [8](https://github.com/pinecone-io/canopy/releases/tag/v0.8.0)) and submit [issues and feature requests](https://github.com/pinecone-io/canopy/issues) on Github.



We [can’t wait](https://community.pinecone.io/) to see what you build.