The MariaDB Galera cluster is a master-master solution that provides High Availability, resilient and very good performance.
It keeps continuously the nodes all in sync for transactions. If you experience an issue with Run out of disk space you may be able to solve your problem by following these steps
How Galera works
Data modification happens on a node then there is an Optimistic locking control it succeeds if it can commit on all nodes
Challenges of DDL
The schema change can happen on any node, it may affect query results though.
The schema must be upgraded before any other DML which can cause long transactions.
DDL changes are replicated in the same order regarding other transactions. The whole cluster is blocked for the duration of the entire DDL operation, the upgrade should be performed safely by scheduling a maintenance window and have the cluster not be accessible until DDL finishes.
The DDL operations should be run without a maintenance window only if they are quick.
Run out of disk space
The GaleraDB nodes should all have the same specs RAM, CPU, and disk space in order to have even performance and eventually the same potential issues on all nodes.
For example, is a DDL table may require to use of the temporary space for creating the temporary table, it can end up in a "Run out of disk space" error.
If the disk sizes are different amongst the nodes, you can have the error on just one node. Galera does not mark it as a failed transaction or a corrupt disk instead it carries on reporting that all nodes on sync.
References
- Change the replication mode Link [A schema change inconsistency with Galera Cluster for MySQL]
- How to avoid pitfalls in Schema Upgrade Link
Solution
You need to force a resync operation of the node. You can enforce it by stopping the MariaDB server, delete the grastate.dat file in the datadir, it is usually located there, and restart the server.
This is an expensive operation since the whole datadir will be sync so I recommend doing it during low traffic or even in a maintenance window.