getmerge Operation not Permitted
getmerge command takes a source directory and a destination file as input and concatenates files in src into the destination local file.
Recent Posts
Cloud IAM - GCP Learning Notes.
Google Cloud offers IAM, which lets you give more granular access to specific Google Cloud resources and prevents unwanted access to other resources.
Cloud Storge Bucket - GCP Learning Notes.
Cloud Storage is Google Cloud’s object storage service, and it allows worldwide storage and retrieval of any amount of data at anytime.
Long Running Jobs in YARN distcp.
Usually distcp and other batch jobs run for a very long time. This is fine if you are running a HADOOP environment without kerberos. When we kerberize a clus...
ERROR - java heap space caused using distcp.
When you see this error default thing to do is the set the HADOOP_CLIENT_OPTS config on your env. In My case did a typo HADDOP_CLIENT_OPTS which caused us sp...
Bulk Upload Hfiles doBulkLoad - LoadIncrementalHfiles failed. This is recoverable. Hbase Encryption Zone.
We were working with a team who were uploading data into HBASE using the doBulkLoad, also we had moved the /hbase to encryption zone. The issue started after...
Checking HDFS health using fsck.
When we have large data sets on the cluster, there will be corruptions of blocks. This could be due to disk or any other.
HDFS Block Corrupted - HBase Error Could not seek StoreFileScanner Invalid HFile block magic
We ran into an issue which was causing HBase to fail. When the job was rerun then it would work again. In the logs we got 2 errors.
MongoDB to Neo4j Using neo4j_doc_manager.
We had a requirement where we wanted to have all the data which is in mongodb to be replicated on neo4j to show few graphs. Here is quick way to demonstrate ...
Variable Substitution in YAML Config pyyaml [Python]
PyYAML - The next generation YAML parser and emitter for Python.