Recommeneded project structure for alertmanager helm chart

I'd recommend to keep project structure simple and straight forward:

chart-name/
  # any static content to include into templates
  files/
  # all needed dynamic templates
  templates/
  # chart description
  Chart.yaml
  # default values
  values.yaml

configuration file and config CRDs

VMAlermanager could use VMAlertmanagerConfig CRD for extending configuration and provide flexibable alerts routing.

I'd recommend to use VMAlertmanager.spec.configRawYaml as default input with stab values and global params:

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanager
metadata:
 name: main
 labels: {{ include "monitoring.labels" $ | nindent 4 }}
 namespace: {{ $.Release.Namespace }}
spec:
 podMetadata:
   labels:
     role: internal-alertmanager
 image:
   tag: {{ .Values.vmalertmanager.imageTag }}
 configNamespaceSelector:
   matchLabels:
     kubernetes.io/metadata.name: {{ .Release.Namespace }}
 extraArgs:
   web.external-url: "https://alertmanager.{{ .Values.clusterDomain }}"
   cluster.label: "main"
 templates:
   - name: alertmanager-templates-internal
     key: slack.tmpl
 # important config option to use cluster wide alerts routing
 disableNamespaceMatcher: true
 configRawYaml: |
   global:
     resolve_timeout: 5m
   route:
     group_wait: 10s
     group_interval: 2m
     group_by: ["alertgroup", "resource_id"]
     repeat_interval: 12h
     receiver: 'blackhole'
   receivers:
     # by default route to dev/null
     - name: blackhole

And make actual alerts routing with dedicated VMAlertmanagerConfigs:

# templates/alertmanager-slack.yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanagerConfig
metadata:
  name: slack-internal
  namespace: {{ $.Release.Namespace }}
  labels: {{ include "monitoring.labels" $ | nindent 4 }}
spec:
  route:
    # default receiver for group
    receiver: internal-slack
    group_interval: 1m
    routes:
      - receiver: internal-slack
        matchers:
        - alertname != alert-notification
        continue: true
      - receiver: actionable-events
        matchers:
        - alertname !~ "LabelValuesExceedLimit|LabelCountExceedLimit"
        continue: false

  receivers:
    - name: internal-slack
      slack_configs:
        # secret created by command
        - api_url:
            key: SLACK_HOOK
            name: alertmanager-secrets
          channel: "{{ $.Values.slackMainChannel }}"
          {{`
          send_resolved: true
          title: '{{ template "__title" . }}'
          actions:
            - type: button
              text: "Query :mag:"
              url: "{{ (index .Alerts 0).GeneratorURL }}"
            - type: button
              text: "Silence :no_bell:"
              url: '{{ template "__alert_silence_link" . }}'
            - type: button
              text: "Runbook :book:"
              url: '{{ template "__runbook_link" . }}'
            - type: button
              text: "Dashboard :grafana:"
              url: '{{ template "__grafana_link" . }}'
            - type: button
              text: "logs :terminal:"
              url: '{{ template "__logs_link" . }}'
            - type: button
              text: '{{ template "__link_button_text" . }}'
              url: "{{ .CommonAnnotations.link_url }}"
          text: '{{ template "_internal_slack_notification" . }}'
          `}}
    - name: actionable-events
      slack_configs:
        # secret created by command
        - api_url:
            key: SLACK_HOOK
            name: alertmanager-secrets
          channel: "{{ $.Values.actionableChannel }}"
          send_resolved: true
          # by wrapping with `` it's possible to use nested templates
          {{`
          title: '{{ template "__title" . }}'
          actions:
            - type: button
              text: "Query :mag:"
              url: "{{ (index .Alerts 0).GeneratorURL }}"
            - type: button
              text: "Silence :no_bell:"
              url: '{{ template "__alert_silence_link" . }}'
            - type: button
              text: "Runbook :book:"
              url: '{{ template "__runbook_link" . }}'
            - type: button
              text: "Dashboard :grafana:"
              url: '{{ template "__grafana_link" . }}'
            - type: button
              text: "logs :terminal:"
              url: '{{ template "__logs_link" . }}'
            - type: button
              text: '{{ template "__link_button_text" . }}'
              url: "{{ .CommonAnnotations.link_url }}"
          text: '{{ template "_internal_slack_notification" . }}'
          `}}

It's possible to conditionally include multiple configs. Or even delegate config creation to other users.

notification templates

It's highly recommened to keep alertmanager templates at files/ folder and included it as is into configmap. Main idea behind it is to re-use static well-tested templates and provide input to via labels/annotations. It greatly simplifies templates modifications.

Lets consider the following example:

# files/slack.tmpl
{{ define "__alert_silence_link" -}}
    {{ .ExternalURL }}/#/silences/new?filter=%7B
    {{- range .CommonLabels.SortedPairs -}}
        {{- if ne .Name "alertname" -}}
            {{- .Name }}%3D"{{- .Value -}}"%2C%20
        {{- end -}}
    {{- end -}}
    alertname%3D"{{ .CommonLabels.alertname }}"%7D
{{- end }}

Configmap with files include:

# templates/alertmanager-templates.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: alertmanager-templates-internal
  namespace: {{ $.Release.Namespace }}
  labels: {{ include "monitoring.labels" $ | nindent 4 }}
data:
{{ (.Files.Glob "files/**.tmpl").AsConfig | indent 2 }}