A ChatGPT That Only Reads Your Documents: What Azure OpenAI Made Possible in April 2023

The loudest objection to ChatGPT in early 2023 was never capability. It was the data: users pasting sensitive information into a public prompt box with no idea where it goes. Azure OpenAI, which Microsoft made generally available in early 2023, answers that objection directly. You can think of it as a self-hosted instance of ChatGPT where you build on models with your own business data, and that data is not used to help train the public model. Microsoft documents the privacy posture here: Data, privacy, and security for Azure OpenAI Service (opens in new tab).

What made April 2023 interesting was a sample project Microsoft released alongside it: an application that consumes your internal data, indexes it, and gives users a chat interface to query it. The project lives on GitHub: azure-search-openai-demo (opens in new tab).

Microsoft's example uploads company PDFs about an organization's HR benefit plans so employees can ask questions in chat. The chat even cites the content where it surfaced each answer.

How did the sample hold up under testing?#

We spun up our own instance of the sample, used Microsoft's demo data, and then injected documents of our own to see how quickly the index picked up new information and cited it. Three findings from that April 2023 test:

• The service cites the benefit information in the sample data very accurately, and it shows the evidence it searched through to produce each answer.
• After adding a new document to the dataset, it took about 1 to 2 minutes before the chat could cite the new data. No restart or reboot of the service was needed.
• We uploaded the popular M365 feature matrix from m365maps.com (opens in new tab) and got accurate answers to Microsoft 365 licensing questions.

Can it read a SharePoint environment?#

The short answer is yes. It can use SharePoint document libraries to surface answers across PDFs, CSVs, Word docs, and similar files. It cannot surface information from SharePoint Lists. The setup tutorial: SharePoint indexer (preview) for Azure Cognitive Search (opens in new tab).

The service also supports data from:

• Azure Storage
• Azure SQL
• Azure Cosmos DB

The permissions catch#

This is the caveat that should shape any deployment conversation. While the service can ingest data from SharePoint document libraries, it does not inherit the native permissions on those documents. If a file was locked to the leadership team in SharePoint, the index does not know that. As of April 2023, access control is something you build yourself, by creating separate RBAC at the index level in Azure. Plan for that work before pointing the indexer at anything sensitive.

What does it cost to run?#

These are the Azure resources the sample deploys:

Using the Azure Calculator with some placeholder estimates, the cost came out to roughly $132 per month as of April 2023, with a heavy disclaimer: everything depends on usage, both how often users hit the chat and how many documents you scan. Form Recognizer carries the highest cost at $50 per 1,000 pages scanned, so initial ingestion is likely your biggest upfront spend. Switching Form Recognizer to the free tier, limited to 500 pages per month, is one way to pull the number down.

The sensible approach is the one that worked for us: spin up the service, add some documents, and test with a pilot group before quoting anyone a monthly number. Our entire test cost less than $20.

Is "ChatGPT as a service" a real MSP offering?#

We think the possibilities here are worth taking seriously. A customer with substantial data in Azure or SharePoint Online could get real value from indexing and querying their own internal documentation, and an MSP could package exactly that. On pricing, two models fit: a flat fee if you expect cost to stay consistent over time, or pure usage-based billing with markup, the same way many MSPs bill Azure services today.

There is a strategic angle too. For customers not yet in Azure, this is an unusually compelling introduction, because the benefit shows up at the individual user level rather than in an infrastructure diagram nobody outside IT cares about.

Our one hesitation, stated plainly in April 2023: it is possible Microsoft bakes this capability into the Microsoft 365 Copilot services coming later. Indexing SharePoint Online for advanced search and chat looks like low-hanging fruit for Copilot. Other data sources, like SQL databases and Azure Storage, seem likely to remain open territory either way.

Where to see it in action#

Microsoft Mechanics walks through the architecture in this overview video (opens in new tab), and there is a full demo of the sample being deployed and tested (opens in new tab).