Azure DevOps suddenly not letting you manage Environments ? Sorry, it's for your own security!

4 min read.

This article explains how to get around new security limitations (did I say “limitations”? I meant “features”) in Azure DevOps to make it tolerable to configure your Azure DevOps Deployment Environments. Because the recent changes to Azure DevOps have made it, quite frankly, intolerable.

But – let’s take a step back before we take the required 15 steps forward (or sidestep the whole issue – now that’s what I call a proper shortcut, and I’ll share it with you too).

Background

Table of Contents

I’m working for a company developing and offering a SaaS-product for some 2 million users. The offering consists of a bunch of pods running on AKS – some of which are standard and run for every customer, some of which are customized enough that they run for just one customer, and never in more than one pod at a time.

And all of these customers might be running any of the recent major or minor Omnia releases, because like any serious software company, of course we roll our updates out in waves, feature flags, A-B -testing, ninja updates and zero downtime green-blue -deployments.

So, as you can probably imagine, that’s a lot of pods. And a lot of different pods require a lot of different images, and a lot of different versions of those images.

This is not something that a human should manage. And of course, a human does not manage this. A magical, grey eminence known as “DevOps” manages this.

But how does “DevOps” manifest itself? In our case, it’s a bunch of software engineers who understand about Dev and Ops (so far, so good) even if they concentrate on Ops (still tolerable) and over a hundred Azure DevOps pipelines building, testing, pushing and deploying code almost non-stop.

Yeah, it does get a bit out of hand. But I digress.

This works great, and usually a human does not have to touch much. But sometimes, one needs to configure a new pipeline to connect to a new (or existing) environment to deploy code to. And that’s been fine, up until some weeks ago.

And that brings us to our problem.

Problem

We use Environments to control which Pipelines can deploy code to which environments (with a lowercase “e”). These environments are separate “tenants” and used by different customers or internal teams, and should not be mixed, so there’s a bit of a security boundary between them.

In Azure DevOps, you can use Environments to achieve this by using deployment jobs instead of, well, just “jobs”.

jobs:
# using "deployment" instead of "job" means this'll be a deployment job which gives us access to "environment" and other cool stuff
- deployment: ${{ parameters.name }}

  displayName: ${{ parameters.displayName }}
  pool:
    vmImage: $(Image)
  environment: ${{ parameters.environment }}

And this is great! You can have Environment-specific controls like approvals and checks, and also limit which pipelines can access which Environment.

It’s usually fun and easy to give a new pipelines access to new Environments. Actually, it’s usually possible to just let all of the pipelines access any new or current Environment and instead just have a gate to limit deployments to Environments that shouldn’t blindly accept any pipeline’s inputs.

And this way of working worked nicely for us for a couple of years. But alas – one day, a few weeks ago, I ran into this error when creating a new environment:

I was the person creating the Environment, and hence had “Administrator” permissions on the Environment. I also happened to be Project Administrator. And while I COULD add the pipelines to have access to this Environment one-by-one, we had quite a few pipelines that might deploy to this Environment.

We already used approvals/gates for the Environment so we didn’t need this additional hindrance that we didn’t ask for. So I submitted a ticket to Microsoft to fix it: https://developercommunity.visualstudio.com/t/youre-not-authorized-to-open-access-to/10518872

Microsoft’s responses are private and a bit confusing, so I thought I’d explain what to do. In their responses, they mention “Environments-hub” and link to Microsoft’s documentation that describes the necessary permissions to be located under the following path:

Set all environment permissions

From within your project, select Project settings .
Select Environments under Pipelines.
Set permissions, and then Save your changes.

Now, if you’ve worked with Azure DevOps, you know this doesn’t make much sense. This is exactly the same as omitting the first step altogether. You’ll modify permissions for your whole Project at the same time by just clicking Environments -> “…” -> Security and setting the permissions there.

But also, you can’t set your own permissions:

Not even if you’re Project Administrator. Or God. And apparently, “normal” Project Administrators can’t even see Environments anymore, not even if they’re added to “Environments-hub” to Administrators, directly:

(I tried with a different account to see if it could be added – it could, but it can’t see, access or modify anything even if it is the Administrator)

Sound complicated? Well, it sure is. And also very frustrating.

So… How do you solve this conundrum?

Solution

If you just want to get rid of the issue, you can just make everyone “Team Project Collection Administrators” through the Organization-level permissions.

Then you can do this:

After hitting “Open access” you’ll get this:

And after selecting “Open access” all of your YAML pipelines get access to this Environment (Classic pipelines could access it anyway because Microsoft only wants to worsen the life of YAML pipeline users) – but they still respect your approvals / release gates.

Problem solved! Right?

The only caveat is that all of your power users are now Organization-level God users with full permissions to everything.

Is this backwards? Yes.

Is this a huge degradation of security? Also yes.

But hey – it’s the way Azure DevOps now works. If you don’t want to add every single pipeline to every single Environment one-by-one, make everyone who needs to deal with Environments an Organization/Team Project Collection -level admins.

Saying that kind of makes me want to vomit, but perhaps this is a great additional sales opportunity for “GitHub Advanced Security” now? :)

Author
Recent Posts

Antti K. Koskela

Developer / Speaker / Consultant at Koskila / Precio Fishbone / Norppandalotti Software Co / Alter - Experience Ideas Ltd

Antti Koskela is a proud digital native nomadic millennial full stack developer (is that enough funny buzzwords? That's definitely enough funny buzzwords!), who works as Solutions Architect for Precio Fishbone, building delightful Digital Workplaces.

He's been a developer from 2004 (starting with PHP and Java), and he's been working on .NET projects, Azure, Office 365, SharePoint and a lot of other stuff. He's also Microsoft MVP for Azure.

This is his personal professional (e.g. professional, but definitely personal) blog.

Latest posts by Antti K. Koskela (see all)