Create shell script step by step.
- Extract target documents according to the cyclic synchronization.
- In this sample case, we assume that
cron(8)
trigger every hour. - Have 5 minutes as a margin to avoid omissions.
- In this sample case, we assume that
- Query with ISO 8601 date format made by
date(1)
mongoexport(1)
can't recognize BSONISODate("2023-02-23")
- Need quate(
"
) the field names in query. (not for sort order fields)
AUTH="--username=root --password=example --authenticationDatabase=admin"
Q=$(date -d-1hour5min +'{"createdAt": {"$gt": {"$date": "%Y-%m-%dT%H:%M:%S%z"}}}') \
sudo podman exec -i mongodb mongoexport --quiet $AUTH --db=db1 --collection=items --query="$Q" --sort='{createdAt: 1}'
- In my case, MongoDB running on docker. If you don't need it, remove
podman(1)
execution.
sudo podman exec -i mongodb bash <<'EOF'
AUTH="--username=root --password=example --authenticationDatabase=admin"
export () {
Q="$(date -d-1hour5min +'{"updatedAt":{"$gt":{"$date":"%Y-%m-%dT%H:%M:%S%z"}}}')"
mongoexport --quiet $AUTH --db="$1" --collection=items --query="$Q" --sort='{updatedAt:1}'
}
diff --suppress-common-lines <(export db1) <(export db2)
EOF
- Use
mongoimport(1)
to upsert collection.
sudo podman exec -i mongodb bash <<'EOF'
AUTH="--username=root --password=example --authenticationDatabase=admin"
export () {
Q="$(date -d-1hour5min +'{"updatedAt":{"$gt":{"$date":"%Y-%m-%dT%H:%M:%S%z"}}}')"
mongoexport --quiet $AUTH --db="$1" --collection=items --query="$Q" --sort='{updatedAt:1}'
}
diff --suppress-common-lines <(export db1) <(export db2) \
| awk -v sync_to="mongoimport $AUTH --collection=items --type=json --mode=upsert --db=" '
/^< / { l2r = l2r substr($0, 3) "\n" }
/^> / { r2l = r2l substr($0, 3) "\n" }
END {
printf "%s", l2r | sync_to "db2"
printf "%s", r2l | sync_to "db1"
}'
EOF
This script works reasonably well. However, if the same document is changed during the synchronization period, it must be overwritten with the later changed value, which is not handled. Let's remedy that in the next step.
-
Sorting by
updateAt
, because the same_id
entry ondb1.items
anddb2.items
will be overwritten by a later update if it has been changed. -
The option
--slurp
liftitems
into a formal json array, somongoimport(1)
will accept that input with--jsonArray
.
sudo podman exec -i mongodb bash <<'EOF'
AUTH="--username=root --password=example --authenticationDatabase=admin"
export () {
Q="$(date -d-1hour5min +'{"updatedAt":{"$gt":{"$date":"%Y-%m-%dT%H:%M:%S%z"}}}')"
mongoexport --quiet $AUTH --db="$1" --collection=items --query="$Q" --sort='{updatedAt:1}'
}
import () {
mongoimport $AUTH --db="$1" --collection=items --jsonArray --mode=upsert
}
diff --suppress-common-lines <(export db1) <(export db2) \
| sed -n 's/^[<>] //p' | jq --slurp 'sort_by(.updatedAt)' \
| tee >(import db1) >(import db2) >/dev/null
EOF
An unnecessary effect of this script is that it udpates unchanged documents, but that would be an acceptable trade-off for simplicity.
- In NixOS, add the Systemd/Timers configuration to
configuration.nix(5)
. - Note: Indentation must be removed for here-document terminator (
EOF
) to work properly inscript
(at least forEOF
line).
systemd = {
timers."mongodbsync" = {
wantedBy = [ "timers.target" ];
partOf = [ "mongodbsync.service" ];
timerConfig.OnCalendar = [ "*-*-* *:00:00" ];
};
services."mongodbsync" = {
serviceConfig.Type = "oneshot";
serviceConfig.User = "root";
script = ''
${pkgs.podman}/bin/podman exec -i mongodb bash <<-'EOF'
AUTH="--username=root --password=example --authenticationDatabase=admin"
export () {
Q="$(date -d-1hour5min +'{"updatedAt":{"$gt":{"$date":"%Y-%m-%dT%H:%M:%S%z"}}}')"
mongoexport --quiet $AUTH --db="$1" --collection=items --query="$Q" --sort='{updatedAt:1}'
}
import () {
mongoimport --quiet $AUTH --db="$1" --collection=items --jsonArray --mode=upsert
}
diff --suppress-common-lines <(export db1) <(export db2) \
| sed -n 's/^[<>] //p' | jq --slurp 'sort_by(.updatedAt)' \
| tee >(import db1) >(import db2) >/dev/null
EOF'';
};
};
Remember to do nixos-rebuild switch
.