Alauda Cache Service E1

Upgrade a Disaster Recovery Group from Redis 6.0 to Redis 7.2

Draft

This procedure is a draft. It describes the upgrade path implemented in Alauda Cache Service E1 v5.1.0, but it has not yet completed production validation. Rehearse it in a non-production environment, take a verified backup of every member first, and contact Alauda support before running it against production data.

Redis 6.0 carries the frozen legacy replication module. Redis 7.2 carries the new module, which adds Active-Active mode, mandatory peer authentication, and rotation-safe credentials. Moving an existing Disaster Recovery group from 6.0 to 7.2 is an in-place version upgrade of every member, followed by re-wiring the links.

TOC

What this upgrade involvesPrerequisitesProcedureStep 1: Quiesce the groupStep 2: Delete the connectionsStep 3: Upgrade each member to Redis 7.2Step 4: Confirm the peer-auth bindingStep 5: Re-create the connectionsStep 6: VerifyAutomatic peer-auth bindingIf the binding is not there yetAfter the upgradeKnown gaps in this draft

What this upgrade involves

Replication is interrupted for the duration of the upgrade

The two module generations do not interoperate — cross-generation replication is not a designed path. A group cannot be upgraded member by member while continuing to replicate. Replication stops when the first member is upgraded and resumes only after every member is on Redis 7.2 and the links have been re-created.

Plan a maintenance window that covers the whole group, and keep the client traffic on a single member for the duration.

Changes that take effect with the new module:

AreaRedis 6.0Redis 7.2
Link authenticationThe instance's default account, bound automatically through spec.activeRedis.redisUserRefA peer-auth RedisUser bound with spec.activeRedis.redisUserRef; a dedicated custom account is recommended
Peer transportProxy RESP port (6379)The same proxy RESP port (6379) — one endpoint carries both transports
PersistenceRDB recommendedRDB only — appendonly=yes is rejected
Database countStandarddatabases must be <= 16
Active-Active modeNot availableAvailable

Prerequisites

Before starting, confirm each of the following for every member of the group.

  1. The operator is on v5.1.0 or later in every cluster that hosts a member.

  2. A verified backup exists for every member. See Backup and Restore.

  3. Persistence is RDB-only. If any member sets customConfig.appendonly: "yes", the upgrade is rejected by admission. Change it to no before the upgrade.

    $ kubectl -n default get redis s6-dc1 -o jsonpath='{.spec.customConfig.appendonly}'

  4. The database count is 16 or fewer. If any member sets customConfig.databases above 16, the upgrade is rejected. The module refuses to load beyond 16 databases and the pods would crash-loop.

    $ kubectl -n default get redis s6-dc1 -o jsonpath='{.spec.customConfig.databases}'

  5. The upstream proxy endpoint stays reachable. The new module replicates over the same proxy endpoint the Redis 6.0 links already used, so a Disaster Recovery group needs no new port opened. Leave the existing firewall rules and load-balancer entries for the proxy's RESP port in place, so the links can be re-created at the end.

  6. Record the current topology. Note each member's serviceID, and which member holds the ActiveRedisConnection pointing at which upstream. You will re-create these links.

    $ kubectl -n default get activeredisconnections -o wide
$ kubectl -n default get redis s6-dc1 -o jsonpath='{.spec.activeRedis.serviceID}'

Procedure

Step 1: Quiesce the group

Direct client write traffic to the member you intend to keep as the upstream, and confirm the downstream members have caught up.

$ kubectl -n default get activeredisconnections
NAME              INSTANCE   STATUS    MESSAGE   AGE
conn-dc2-to-dc1   s6-dc2     Healthy             8d

Confirm status.shards[].syncStatus is PartialSync and the offsets are advancing before proceeding.

Step 2: Delete the connections

Remove the ActiveRedisConnection in each downstream cluster. Use the default Detach teardown policy — the members are returning to the group after the upgrade, so their module-side bookkeeping should be retained.

$ kubectl -n default delete activeredisconnections conn-dc2-to-dc1
Do not use Decommission here

Decommission permanently releases the peer's Oplog retention and tombstone garbage-collection floors and cannot be reversed. It is for a datacenter that is gone for good, not for an upgrade.

Step 3: Upgrade each member to Redis 7.2

Change spec.version on each member. The instance performs a rolling restart.

$ kubectl -n default patch redis s6-dc1 --type=merge --patch='{"spec": {"version": "7.2"}}'

Wait for the instance to return to Ready before upgrading the next member:

$ kubectl -n default get redis s6-dc1
Upgrade order

Because the links were deleted in Step 2, there is no live replication to order the upgrade around, so members may be upgraded in any order.

The ordering rule matters for later, within-generation upgrades of a live group: the safe order is downstream before upstream, because a downstream can apply an equal-or-older upstream's Oplog but never a newer one. Bumping an upstream ahead of a live downstream returns an admission warning, and the module holds the affected edge — replication to that downstream pauses until the version skew is resolved.

Step 4: Confirm the peer-auth binding

The new module requires a peer-auth credential on every instance. On an instance that already had cross-datacenter replication enabled, the platform binds one automatically — see Automatic peer-auth binding.

Check what each member ended up with:

$ kubectl -n default get redis s6-dc1 -o jsonpath='{.spec.activeRedis.redisUserRef}'
rfr-acl-s6-dc1-default

$ kubectl -n default get redisusers rfr-acl-s6-dc1-default
NAME                     INSTANCE   USERNAME   PHASE     AGE
rfr-acl-s6-dc1-default   s6-dc1     default    Success   8d

If the binding is empty, or the RedisUser phase is not Success, resolve it as described in If the binding is not there yet before continuing.

Every member needs the same credential

The module accepts an inbound peer only when the username it presents matches the local peer-auth username. The automatic binding uses each instance's default account, so members that were already replicating to each other converge on the same credential — that shared default password is exactly what their Redis 6.0 links were authenticating with.

If you bound credentials manually on any member, verify that every member carries the same username and the same password value before creating any link.

Step 5: Re-create the connections

Create the ActiveRedisConnection in each downstream cluster. There is no credential field any more — the link uses the instance's peer-auth binding — and the upstream address is the same one the Redis 6.0 link used:

$ cat << EOF | kubectl -n default create -f -
apiVersion: redis.middleware.alauda.io/v1alpha1
kind: ActiveRedisConnection
metadata:
  name: conn-dc2-to-dc1
spec:
  instance: s6-dc2
  addresses:
  - 192.168.1.10:6379
  teardownPolicy: Detach
EOF

Admission resolves the peer-auth binding, runs the pre-flight inspection with the credential the data path will use, and dials the upstream endpoint. A failure at this point names the specific problem.

Step 6: Verify

$ kubectl -n default get activeredisconnections conn-dc2-to-dc1
NAME              INSTANCE   STATUS    MESSAGE   AGE
conn-dc2-to-dc1   s6-dc2     Healthy             45s

Confirm that:

  • the connection status is Healthy;
  • status.credentialMode is peer-auth-global;
  • status.shards[].status is Connected and the offsets advance;
  • the upstream's ActiveRedis reports the expected status.downstreamPeerCount;
  • a key written on the upstream appears on each downstream.

Then restore normal client traffic.

Automatic peer-auth binding

The new module makes the peer-auth binding mandatory, which would otherwise block the very update that performs the upgrade. The platform handles that by writing down the credential the Redis 6.0 link has been using all along.

A Redis 6.0 link never carried a username: its credential held a password and nothing else, so the module sent a one-argument AUTH that the proxy resolved to the instance's default account. That account is the link credential. The operator therefore binds the instance's own default-account RedisUser:

Instance architectureBound RedisUser
Clusterdrc-acl-<instance-name>-default
Sentinel, standalonerfr-acl-<instance-name>-default

Nothing is provisioned. That account already exists — the instance's own reconcile creates it — and the password in it is the one the link uses, so the binding changes no byte on the wire.

The binding is applied on the first reconcile of any Redis 6.0 instance that has replication enabled and carries none, so it is normally already in place before you begin. If the version change arrives first, the defaulting webhook fills in the same name.

Scope of the automatic path
  • A fresh instance created on Redis 7.2 with replication enabled must be given a binding explicitly — it is rejected without one.
  • A mistyped manual binding surfaces as RedisUser not found rather than being silently created.
  • A binding that is already set is never overwritten.
The automatic binding ties the link to the instance password

Because it binds the default account, every member of the group must carry the same default password for the links to authenticate — which was already true of the Redis 6.0 links, and rotating that password now also rotates the replication credential.

To keep the two independent, create a dedicated custom RedisUser on every member — the same username and the same password value everywhere — and bind that instead; see Step 1 of the Disaster Recovery setup. Do it before Step 5, and on every member.

If the binding is not there yet

The binding is deferred for one reason only: the default-account RedisUser does not exist yet. The instance's own reconcile creates it, and the binding lands on a following pass. If it is still empty once the instance is Ready, set it yourself:

$ kubectl -n default patch redis s6-dc1 --type=merge \
    --patch='{"spec": {"activeRedis": {"redisUserRef": "rfr-acl-s6-dc1-default"}}}'

After the upgrade

Once the group is running on Redis 7.2, the following become available:

  • Active-Active mode, in which every datacenter accepts writes instead of following a single upstream. See Set Up Active-Active Replication. Note that spec.activeRedis.mode selects the wiring for the instance; moving an established Disaster Recovery group to Active-Active is a separate change and is not covered by this procedure.
  • Rotation-safe credentials. Peer links no longer freeze the credential at wiring time, so rotating the peer-auth password no longer breaks the link on the next reconnect.
  • Decommission teardown, for permanently retiring a datacenter.

Known gaps in this draft

  • The procedure requires a full-group maintenance window. There is no supported way to upgrade a group while keeping replication live.
  • Downgrading from Redis 7.2 back to Redis 6.0 is not supported. If the upgrade must be abandoned, restore the members from the backups taken in the prerequisites.
  • The Web Console does not provide a guided flow for this upgrade; the steps above are CLI-driven.