I continuosly use the Rds Aurora and I tend to manage all resources via cloudformation service. This time I planned to test how to promote a cross region read replica due to a failer of the primary cluster. I'm not going to explore how to do the disaster recovery but I'm testing how the two key properties used for the CRRR in the DB cluster CFN resource can be manage and if it is possible to change them afterwards.
ReplicationSourceIdentifier The Amazon Resource Name (ARN) of the source Amazon RDS MySQL DB instance or DB cluster, if this DB cluster is created as a Read Replica.
- Required: No
- Type: String
- Update requires: No interruption
SnapshotIdentifier The identifier for the DB cluster snapshot from which you want to restore.
- Required: No
- Type: String
- Update requires: Replacement
ReplicationSourceIdentifier
I try to create an Aurora cluster with ReplicationSourceIdentifier point to a test cluster in another region.
Extract from yaml file
DatabaseDbCluster:
Type: AWS::RDS::DBCluster
Properties:
ReplicationSourceIdentifier: arn:aws:rds:eu-west-1:000000000007:cluster:sebba-database-test-4-10000000008
AvailabilityZones:
- eu-west-1a
- eu-west-1b
- eu-west-1c
BackupRetentionPeriod: 35
DBClusterParameterGroupName:
Ref: DatabaseDbClusterParameterGroup
DBSubnetGroupName:
Ref: DatabaseDbSubnetGroup
Engine: aurora
DatabaseName: sebba_service_database
MasterUsername: root
MasterUserPassword: root&root
PreferredBackupWindow: 01:00-01:30
PreferredMaintenanceWindow: Fri:02:00-Fri:02:30
VpcSecurityGroupIds:
- Ref: DatabaseSecurityGroup
Prerequisites for using ReplicationSourceIdentifier property
The DB cluster must NOT have these properties otherwise the creation fails
- Cannot specify database name for cross region replication cluster (removed DatabaseName property)
- Cannot specify user name for cross region replication cluster (removed MasterUsername and MasterUserPassword properties)
- Replication from cluster in same region is not supported
How to create the stack in London region
cd ~/python3/scripts/database
aws cloudformation --profile bt --region eu-west-2 create-stack --template-body file://sebba_service_database_lnd.yaml --parameters file://sebba_service_database_lnd_params/5.json --stack-name sebba-service-database-lnd-test-5
ls ~/python3/scripts/database/sebba_service_database_lnd*
- sebba_service_database_lnd.yaml
- sebba_service_database_lnd_params/5.json
Promote Read Replica and Remove the connection to the master writer in Ireland
Once the cluster and its Rds instance was setup in London region, I promoted Read replica the instance, this makes the cluster in London indipendently from the former master writer in Ireland. The scope of this action is to see how the template with the replication source indentifier reacts to this change.
1) Managing properties of the template with still the ReplicationSourceIdentifier property
The db cluster resource in the source template has still the the
ReplicationSourceIdentifier: arn:aws:rds:eu-west-1:000000000007:cluster:sebba-database-test-4-10000000008
In the source template were changed two properties:
- DB Cluster The retention backup value from 35 to 10
- Add a SG role
This results in
Action Logical ID Physical ID Resource Type Replacement
Modify DatabaseDbCluster sebba-service-database-lnd-test-databasedbcluster-1j9sm6uv3jrmi AWS::RDS::DBCluster False
Modify DatabaseSecurityGroup sg-372fa15f AWS::EC2::SecurityGroup False
Modify DatabaseSecurityGroupIngress DatabaseSecurityGroupIngress AWS::EC2::SecurityGroupIngress Conditional
The change set is what I expected.
The execution of the change set succeed
2) Removing the ReplicationSourceIdentifier once the cluster runs by itself (FAILED)
The "ReplicationSourceIdentifier" is removed from the source template
ReplicationSourceIdentifier: arn:aws:rds:eu-west-1:000000000007:cluster:sebba-database-test-4-10000000008
Action Modify
- Logical ID DatabaseDbCluster
- Physical ID sebba-service-database-lnd-test-databasedbcluster-1j9sm6uv3jrmi
- Resource Type AWS::RDS::DBCluster
- Replacement False
It fails because some mandatory properties are missing ( I.E. MasterUserPassword cannot be empty)
3) Removing the ReplicationSourceIdentifier and add the requirement properties (REPLACE the DBCLuster resource)
Added these properties to the previous template
- DatabaseName: sebba
- MasterUsername: root
- MsterUserPassword: root&root
a
aws cloudformation --profile bt --region eu-west-2 create-change-set --template-body file://sebba_service_database_lnd.yaml --parameters file://sebba_service_database_lnd_params/5.json --stack-name sebba-service-database-lnd-test-5 --change-set-name sebba-service-database-lnd-test-5-$RANDOM
Change set
1 Logical ID Database01DbInstance
- Action Modify
- Physical ID sd1x2bbtbdx3seq
- Resource Type AWS::RDS::DBInstance
- Replacement True
2 Logical ID DatabaseDbCluster
- Action Modify
- Physical ID sebba-service-database-lnd-test-databasedbcluster-1j9sm6uv3jrmi
- Resource Type AWS::RDS::DBCluster
- Replacement True
Questions, Does a replacement of a DBCluster resource wipe out all data?
Yes it does!
Replacement of a DBCluster resource wipes out all Databases! So keep in mind it!
How can re-use the template with the CRR Replica?
I believe the correct procedure here is to
- Stop the cluster
- Take a snapshot
- Terminate the stack
- Re-create the stack with the SnapshotIdentifier cluster property
To be tested.
Testing SnapshotIdentifier
I'd like to restore a cluster from a snapshot and then clean up the template removing the property SnapshotIdentifier and add the mandatory properties
DatabaseDbCluster:
Type: AWS::RDS::DBCluster
Properties:
AvailabilityZones:
- eu-west-2a
- eu-west-2b
BackupRetentionPeriod: 10
DBClusterParameterGroupName:
Ref: DatabaseDbClusterParameterGroup
DBSubnetGroupName:
Ref: DatabaseDbSubnetGroup
Engine: aurora
PreferredBackupWindow: 01:00-01:30
PreferredMaintenanceWindow: Fri:02:00-Fri:02:30
VpcSecurityGroupIds:
- Ref: DatabaseSecurityGroup
SnapshotIdentifier: arn:aws:rds:eu-west-2:000000000007:cluster-snapshot:sebba-service-database-lnd-test-5-snapshot-sebba-service-database-lnd-test-databasedbcluster-jedbc40nn6he-133c6cb3ud4lm
Tags:
- Key: Name
Value:
Fn::Join:
- "-"
- - Ref: ServiceNameDatabase
- 'db-cluster'
====
How the template has been changed in order to remove the unwanted property
- SnapshotIdentifier: arn:aws:rds:eu-west-2:000000000007:cluster-snapshot:sebba-service-database-lnd-test-5-snapshot-sebba-service-database-lnd-test-databasedbcluster-jedbc40nn6he-133c6cb3ud4lm
+ DatabaseName: sebba_service_database
+ MasterUsername: root
+ MasterUserPassword: root&root
Removing the SnapshotIdentifier property is NOT feasible due to the nature of CFN which is configured to replace the DBCluster therefore then delete all data
CONCLUSION
The cross region read replica initiated via Cloudformation is a quick solution to keep your data save in another region and spin up new resources quickly and consistently. Playing with the two key properties used for the disaster recovery allow the user to link the cluster in the different regions, the issue to address is when you want to promote the DR cluster because either the primary master cluster went down or you want to use another region as primary for you website, it is not smooth as it should be. You would imagine keeping your template tidy and clean from legacy code, but if you remove these two properties, Cfn requires the mandatory properties of db name, master username and password and adding or changing these values cause a replacement of the Db cluster resource and a consequently loosing of all data.