Investigation: Switch to the Grafana UI to fully replace Alert Manager UI

[Rotfuks]

Motivation

While the prometheus alert manager has a UI the mimir alert manager relies on grafana for a UI. We should investigate if there are some use cases covered by the prometheus alert manager UI but not by Grafana.

Todo

• investigate what the prometheus alert manager ui can do and is used for
• Investigate if those use cases can all be covered by the grafana alert ui

Outcome

• We are confident that we are not loosing any functionality with that migration

[hervenicol]

Stuff I do with alertmanager and I have to learn with grafana:

• look for current silences
  • what are currently deployed silences?
  • what alerts do they match?
  • are current alerts silenced? by which silence?
• look for alert labels to create a silence

We can do all of these with grafana.
Actually, since we don't have the "alertmanager" link in the opsgenie alerts anymore, I now use grafana via the "explore" link.
But I can't help but feel a bit lost when doing it with grafana.
I guess it's just a matter of changing my habits.

[Rotfuks]

Would a recipe/documentation of those use cases and how you'd do them in grafana help?

[hervenicol]

We currently have no doc around that, and switching to grafana as the unified WebUI makes the experience more consistent.
I don't see much interest in writing a "how to use grafana" doc. This should be the upstream doc.

If we really want to write something around "how to manage alerts and silences in grafana" that could be some kind of demo or blog post.
But our CAPA installations already push for using grafana, and AFAIK nobody complained.

An announcement with a bit of communication should be enough when we remove classic alertmanager from CAPI installations.

[TheoBrigitte]

Quick comparison of Alertmanager vs Grafana UIs

		Alertmanage	Grafana
Active alerts		✅	✅
	Alert labels	✅	✅
	Alert description	✅	✅
	Is alert silenced indicator	✅	✅
	Silence alert	✅	✅
	See source	✅	❌
Silences		✅	✅
	Silence labels	✅	✅
	Silence fields	✅	✅
	Silence affected alerts	✅	✅
	remove silence	✅	✅

I opened a issue for the broken See source link grafana/grafana#96155

[QuentinBisson]

@TheoBrigitte I'm confused, where is the ops-recipe link in the old alertmanager?

If I check on alertmanager we only have this:

But this is not a link right?

We have the same on grafana

[QuentinBisson]

The see source is related to the fact we're not using the mimir's alertmanager. We solved it in the notification template but this is currently redirecting to a Prometheus based query. This is not an issue with grafana

[QuentinBisson]

So I think we're done here :)

Maybe we should demo how to use the new alerting UI but we should be good :)

[TheoBrigitte]

You're right the ops-recipe is not a link in both cases. Maybe it should.
I'll see how the source link works once I have Mimir's alertmanager running.