- Investigate this incident: what happened to cause this incident and why? Determining the root causes is your ultimate goal. Examples: logs analysis, diagramming the request path, reviewing headdumps
- Mitigate this incident: what immediate actions can we take to resolve and manage this specific event? Examples: rolling back, cherry-picking, pushing configs, communicating with affected users
- Repair damage from this incident: how can we resolve immediate or collateral damage from this incident? Examples: restoring data, fixing machines, removing traffic re-routes
- Detect future incidents: how can we the time to accurately detect a similar failure? Examples: monitoring, alerting, plausibility checks on input/output
- Mitigate future incidents: how can we decrease the severity and/or duration of future incidents like this? How can we reduce the percentage of users affected by this class of failure next time it happens? *Examples: graceful degradation, dropping non-critical resu
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# List of kube_*_labels metrics - not all of these are useful for alerting. | |
# kube_certificatesigningrequest_labels | |
kube_certificatesigningrequest_labels * on (certificatesigningrequest) group_right(label_team) ( | |
...insert expression here | |
) | |
# kube_cronjob_labels | |
kube_cronjob_labels * on (cronjob, namespace) group_right(label_team) ( | |
...insert expression here | |
) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
function onEdit(e) { | |
var row = e.range.getRow(); | |
var col = e.range.getColumn(); | |
if(col === 2 && e.source.getActiveSheet().getName() === "help_requests") { | |
if(e.source.getActiveSheet().getRange(row, 1).getValue() == "") { | |
e.source.getActiveSheet().getRange(row, 1).setValue(new Date()); | |
} | |
} | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
for bucket in $(gsutil ls -p project_name); do gsutil label ch -l "bucket:$(echo $bucket | cut -c 6- | sed 's/.$//' | sed 's/\./_/g')" $bucket; done |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# | |
# POD linkerd-web-54f586f959-6mjrt (1 of 1) | |
# | |
# HELP request_total Total count of HTTP requests. | |
# TYPE request_total counter | |
request_total{direction="inbound",tls="no_identity",no_tls_reason="not_provided_by_remote"} 1756 | |
request_total{authority="linkerd-controller-api.linkerd.svc.cluster.local:8085",direction="outbound",dst_control_plane_ns="linkerd",dst_deployment="linkerd-controller",dst_namespace="linkerd",dst_pod="linkerd-controller-789b557586-4pmbb",dst_pod_template_hash="789b557586",dst_service="linkerd-controller-api",dst_serviceaccount="linkerd-controller",tls="true",server_id="linkerd-controller.linkerd.serviceaccount.identity.linkerd.cluster.local"} 536 | |
request_total{authority="linkerd-controller-api.linkerd.svc.cluster.local:8085",direction="outbound",dst_control_plane_ns="linkerd",dst_deployment="linkerd-controller",dst_namespace="linkerd",dst_pod="linkerd-controller-789b557586-lwfvt",dst_pod_template_hash="789b557586",dst_service="linkerd-controller-api",dst_serviceaccount="linkerd-controller |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Extra verbosity to see underlying APIs: | |
kubectl --v=99 get pods busybox | |
Namespaces: | |
kubectl get ns | |
kubectl create ns linux | |
kubectl get ns/linux -o yaml | |
kubectl delete ns/linux | |
Specify PODs namespace: |