Cloud Storge Bucket - GCP Learning Notes.
Cloud Storage is Google Cloud’s object storage service, and it allows worldwide storage and retrieval of any amount of data at anytime.
Recent Posts
Long Running Jobs in YARN distcp.
Usually distcp and other batch jobs run for a very long time. This is fine if you are running a HADOOP environment without kerberos. When we kerberize a clus...
ERROR - java heap space caused using distcp.
When you see this error default thing to do is the set the HADOOP_CLIENT_OPTS config on your env. In My case did a typo HADDOP_CLIENT_OPTS which caused us sp...
Bulk Upload Hfiles doBulkLoad - LoadIncrementalHfiles failed. This is recoverable. Hbase Encryption Zone.
We were working with a team who were uploading data into HBASE using the doBulkLoad, also we had moved the /hbase to encryption zone. The issue started after...
Checking HDFS health using fsck.
When we have large data sets on the cluster, there will be corruptions of blocks. This could be due to disk or any other.
HDFS Block Corrupted - HBase Error Could not seek StoreFileScanner Invalid HFile block magic
We ran into an issue which was causing HBase to fail. When the job was rerun then it would work again. In the logs we got 2 errors.
MongoDB to Neo4j Using neo4j_doc_manager.
We had a requirement where we wanted to have all the data which is in mongodb to be replicated on neo4j to show few graphs. Here is quick way to demonstrate ...
Variable Substitution in YAML Config pyyaml [Python]
PyYAML - The next generation YAML parser and emitter for Python.
Chef Kitchen Test Deployment on Multiple Nodes with Private Interface.
What is Kitchen? Kitchen provides a test harness to execute infrastructure code on one or more platforms in isolation.
Cropping Bulk Images Using Python.
I was working on getting post headers for my post on this blog. I had couple of images from unsplash. But the header for the post need to be a little more ho...