Error
Error Code:
1590
MariaDB Error 1590: Master Incident Reported
Description
This error indicates that a significant event or problem, referred to as an 'incident,' has occurred on the replication master server. The slave server is reporting this incident, signifying a potential disruption or failure in the replication process. It typically occurs when the master encounters an unrecoverable error or a state that prevents normal replication.
Error Message
The incident %s occured on the master. Message: %s
Known Causes
4 known causesMaster Server Crash
The MariaDB master server unexpectedly terminated, became unresponsive, or suffered a critical software failure, leading to an incident report.
Hardware/OS Failure on Master
An underlying operating system or hardware issue (e.g., disk failure, power loss) on the master server caused a critical incident affecting MariaDB operations.
Unrecoverable Data Corruption
The master encountered severe data corruption or an unrecoverable transaction that forced an incident state, halting normal operations.
Master Resource Exhaustion
The master server ran out of critical system resources such as memory, disk space, or file descriptors, leading to an incident.
Solutions
4 solutions available1. Investigate Master Server Logs easy
Examine the MariaDB error logs on the master server for the specific incident reported.
1
Connect to your master MariaDB server via SSH.
2
Locate the MariaDB error log file. Common locations include /var/log/mysql/error.log, /var/log/mariadb/mariadb.log, or as specified in your my.cnf configuration file.
3
Use a command-line tool like `tail` or `grep` to view the log for recent errors, paying close attention to the timestamp that corresponds to the replication error.
sudo tail -n 100 /var/log/mariadb/mariadb.log
sudo grep "ERROR" /var/log/mariadb/mariadb.log4
Analyze the error messages found in the master's logs. These will provide the root cause of the incident, which is crucial for resolving the replication issue.
2. Check Replication Status on Slave easy
Verify the replication status on the slave server to understand the extent of the problem.
1
Connect to your slave MariaDB server using the MariaDB client.
2
Execute the `SHOW SLAVE STATUS` command to get detailed information about the replication process.
SHOW SLAVE STATUS;3
Look for `Slave_IO_Running` and `Slave_SQL_Running` to be 'Yes'. If either is 'No', this indicates a problem. Also, examine `Last_IO_Error` and `Last_SQL_Error` for specific error messages that might provide clues.
4
If `Seconds_Behind_Master` is increasing significantly, it suggests the slave is falling behind, which can lead to more severe replication issues.
3. Restart Replication Threads medium
Attempt to restart the slave's replication threads if they have stopped unexpectedly.
1
Connect to your slave MariaDB server using the MariaDB client.
2
Stop the slave's replication threads.
STOP SLAVE;3
Start the slave's replication threads.
START SLAVE;4
Check the replication status again using `SHOW SLAVE STATUS;` to see if the threads have resumed and if the error is resolved.
SHOW SLAVE STATUS;4. Reset Slave and Re-sync Data (Thorough Solution) advanced
If simpler methods fail, reset the slave's replication configuration and re-synchronize its data from the master.
1
Ensure you have a recent backup of your master database.
2
On the slave server, stop replication.
STOP SLAVE;3
On the slave server, reset the replication configuration.
RESET SLAVE;4
Take a consistent snapshot or dump of your master database. For large databases, consider using `mariadb-dump` with `--single-transaction` or an LVM snapshot.
mariadb-dump -u root -p --single-transaction --all-databases > master_dump.sql5
Transfer the dump file to the slave server.
6
On the slave server, drop all existing databases (be extremely cautious and ensure you have backups).
DROP DATABASE database_name;
-- Repeat for all databases7
On the slave server, import the master dump.
mariadb -u root -p < master_dump.sql8
On the slave server, reconfigure replication by obtaining the current binary log file and position from the master.
SHOW MASTER STATUS;
-- Note the File and Position values9
On the slave server, configure replication with the obtained master log file and position.
CHANGE MASTER TO MASTER_HOST='master_ip_address', MASTER_USER='replication_user', MASTER_PASSWORD='replication_password', MASTER_LOG_FILE='master-bin.XXXXXX', MASTER_LOG_POS=XXXXXX;10
On the slave server, start replication.
START SLAVE;11
Monitor `SHOW SLAVE STATUS;` to confirm replication is running correctly and the slave is catching up.
SHOW SLAVE STATUS;