Three Things That Broke When We Let Copilot Loose on the Finance Tenant

Three Things That Broke When We Let Copilot Loose on the Finance Tenant

The ticket came in at 4:52 on a Friday. A finance analyst had asked Copilot to “summarise our restructuring options” and it cheerfully cited a document called Q3 Redundancy Shortlist.xlsx — a file she was never meant to see. Copilot didn’t hack anything. It read a SharePoint site that had been shared with “Everyone except external users” since 2021, because someone once needed to hand off a template and never cleaned up.

Copilot has been around for roughly forty months now. It is not new. What is new is the unified client stitching Word, Teams, Outlook and the standalone app into one assistant that reaches across every workload at once. That reach is the whole point. It’s also the whole problem. Copilot inherits the permissions of the user who prompts it, then indexes anything Microsoft Graph says that user can technically open. Years of “temporary” sharing decisions became a search index overnight.

Three things broke that week. None of them were Copilot’s fault.

1. SharePoint oversharing we’d forgotten about

The redundancy file lived on a site shared with the Everyone-except-external claim. Nobody browsed to it, so nobody noticed. Copilot browses everything the moment someone prompts it. Before you enable Copilot for a group, find those sites. This reads the sharing links — it changes nothing:

audit.ps1PowerShell
Connect-MgGraph -Scopes "Sites.Read.All" -NoWelcome
​
$flagged = [System.Collections.Generic.List[object]]::new()
$uri = "https://graph.microsoft.com/v1.0/sites?search=*"
​
do {
    try {
        $page = Invoke-MgGraphRequest -Method GET -Uri $uri -ErrorAction Stop
    }
    catch {
        Write-Warning "Site query failed: $($_.Exception.Message)"; break
    }
    foreach ($site in $page.value) {
        $permUri = "https://graph.microsoft.com/v1.0/sites/$($site.id)/permissions"
        try {
            $perms = Invoke-MgGraphRequest -Method GET -Uri $permUri -ErrorAction Stop
            $flagged.Add([pscustomobject]@{
                Site  = $site.displayName
                Url   = $site.webUrl
                Grants = ($perms.value | Measure-Object).Count
            })
        }
        catch { Write-Warning "  perms failed for $($site.webUrl)" }
    }
    $uri = $page.'@odata.nextLink'
} while ($uri)
​
$flagged | Sort-Object Grants -Descending | Export-Csv .\site-sharing-review.csv -NoTypeInformation

Export it, hand it to the site owners, then decide what to lock down. Do not pipe a live query straight into a permission-stripping loop — you’ll break someone’s legitimate workflow and earn yourself a different 2 a.m. ticket. This is where sensitivity labels earn their keep: a label with encryption travels with the file, and Copilot respects it even when the sharing does not. Labelling coverage, not sharing hygiene alone, is the durable fix. You configure it in the Purview portal under Information Protection.

2. A private Teams channel that wasn’t as private as its name

Second finding: a “Leadership – Private” channel whose membership had drifted to include three people who’d changed roles. Private channels have their own membership and their own SharePoint site, and Copilot treats that site like any other. Review the roster against reality:

audit.ps1PowerShell
$team = "00000000-0000-0000-0000-000000000000"  # reviewed team ID
try {
    $channels = Get-MgTeamChannel -TeamId $team -Filter "membershipType eq 'private'" -All -ErrorAction Stop
}
catch { throw "Channel enumeration failed: $($_.Exception.Message)" }
​
foreach ($c in $channels) {
    Get-MgTeamChannelMember -TeamId $team -ChannelId $c.Id -All |
        Select-Object @{n='Channel';e={$c.DisplayName}}, DisplayName, Roles
}

3. Mailbox delegates nobody had audited since the last reorg

Copilot in Outlook summarises what the mailbox owner can read — including folders an old delegate still has rights to. A former EA retained Editor on the CFO’s calendar and Inbox. Harmless for two years. Then Copilot summarised a board thread for her. Audit delegate and folder permissions across the mailboxes you’re onboarding:

audit.ps1PowerShell
$mailboxes = Import-Csv .\onboarding-mailboxes.csv   # reviewed list, not Get-Mailbox
foreach ($m in $mailboxes) {
    try {
        Get-MailboxPermission -Identity $m.UPN -ErrorAction Stop |
            Where-Object { $_.User -notmatch 'NT AUTHORITY|SELF' } |
            Select-Object @{n='Mailbox';e={$m.UPN}}, User, AccessRights
    }
    catch { Write-Warning "Skipped $($m.UPN): $($_.Exception.Message)" }
}

What the postmortem actually taught us

  1. Copilot is an audit tool you didn’t ask for. It surfaces every sharing mistake you’ve made since 2021 — treat rollout as a data-governance project, not a licence assignment.
  2. Labels beat link cleanup. Sharing gets re-broken by the next well-meaning hurry. Encryption on a sensitivity label doesn’t.
  3. Review the Graph consent, then the audit retention. Check the app permissions Copilot was granted under Entra ID → Enterprise applications, and confirm your audit log retention covers longer than a quarter. When someone asks “who saw this and when,” you want the answer to exist. Ours only went back 90 days. That was the other thing that broke.

Copilot didn’t create the exposure. It just stopped letting us pretend the exposure wasn’t there.

Related Post