We ran into an issue which was causing HBase to fail. When the job was rerun then it would work again. In the logs we got 2 errors.
IOException: Could not seek StoreFileScanner
Sometime we got.
IOException: Invalid HFile block magic: \x00\x00 ...
Commands to verify the issue.
hbase hfile -mbsf /hbase/<path_to_file>/cf/<file>
-m : Print meta Dataset
-b : Print block
-s : Stats
-f : file to scan
Further we can also check using hdfs fsck /<path_to_file> command to verify if we have bad blocks.
More details on this here https://ahmedzbyr.github.io/hdfs/hadoop/hdfs-check-blocks/
Possible cause of this issue.
- Block corruption.
- Sector issue on the disk.
We found that it was a issue with the disk and we replaced it and the issue was resolved.