ansible on RHEL 6.6.
Ansible is a radically simple IT automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchest...
tarball.
This is a simple zookeeper playbook, to quickly start zookeeper running on a single or more nodes, in a clustered mode.
This is a simple Storm Cluster Setup. We are using a dedicated Zookeeper Cluster/Node, instead of the standalone zkserver. Below is how we will deploy our c...
This is a simple Kafka setup. In this setup we are running kafka over a dedicated zookeeper service. (NOT the standalone zookeeper which comes with kafka)
tarball.
Setting up Hadoop using Ansible, we will be using cdh5 tarball for installation of the cluster.
The SpagoBI project is a free software/open source initiative by the SpagoBI Labs of Engineering Group. It aims to realize the most complete 100% open source...
Deployment error during setting zookeeper on a new cluster using Cloudera Manager UI, when an earlier cluster containing zookeeper earlier was built using Cl...
We were deploying one of the cluster on our lab environment which is used by everyone. So the lab has it own share of stale information on it.
We had a cluster environment which we needed to update the /etc/hosts file. Which would help communicate between the server over a private network. Our serve...
Data Bags are a way to store information on the chef-server which all the cookbooks can access. Few more additional advantages are that we can encrypt the da...
This is a quick reference for few of the commands which I use often.
Migrating an old Nagios backup to a new nagiosxi server. Migration is simple in Nagios XI, you a backup and restore it. Once that is done we need to update/a...
HTTPS is a protocol for secure communication over a computer network which is widely used on the Internet. HTTPS consists of communication over Hypertext Tra...
The standalone installation of Chef server creates a working installation on a single server. This installation is also useful when you are installing Chef s...
Pandoc is an opensource utility to create documents from markdown. We can create PDF, Doc, doc, html and other formats. And can be also used to convert html ...
Nagios monitors your entire IT infrastructure to ensure systems, applications, services, and business processes are functioning properly. In the event of a f...
RPM (Redhat Package Manager) is the most popular package utility and is used mostly on RHEL, Centos and Fedora. RPM helps user/admins to build, install, que...
Was working on a monitoring project, need to create a large file to test notifications. Here is how we can do that.
Checking port available on a remote machine using nc command instead of telnet. Same command can be used to check on a remote server as well, change the 127....
Linux Unified Key Setup-on-disk-format (or LUKS) allows you to encrypt partitions on your Linux computer. This is particularly important when it comes to mo...
How to move an existing data directory in mysql to a new location. We were running out of space and had to move the existing data directory to a new drive. B...
We were running out of disk space on one of the databases server, we need to get information on what the current table/database usage was. Below are few comm...
python 2.7.x on Centos 6.5/6.6
By default centos comes with python 2.6. In most of the cases we might need python 2.7 or later to be installed. Below are few ways to install python 2.7 on ...
What is Kitchen? Kitchen provides a test harness to execute infrastructure code on one or more platforms in isolation.
We had a cluster environment which we needed to update the /etc/hosts file. Which would help communicate between the server over a private network. Our serve...
Kitchen-Vagrant has the capability to spin up a windows instance for testing. To make it work you will need the vagrant-winrm to be installted on the worksta...
Data Bags are a way to store information on the chef-server which all the cookbooks can access. Few more additional advantages are that we can encrypt the da...
knife-windows plugin adds additional functionality to the Chef Knife CLI tool for configuring / interacting with nodes running Microsoft Windows.
The standalone installation of Chef server creates a working installation on a single server. This installation is also useful when you are installing Chef s...
This cookbook [currently as of now] can be used to setup a Cloudera Manager Server (Management Server) running on MySQL/Postgres database. But the intended u...
The Chef Development Kit (ChefDK) brings the best-of-breed development tools built by the awesome Chef community to your workstation with just a few clicks. ...
knife-windows plugin adds additional functionality to the Chef Knife CLI tool for configuring / interacting with nodes running Microsoft Windows.
The standalone installation of Chef server creates a working installation on a single server. This installation is also useful when you are installing Chef s...
The Chef Development Kit (ChefDK) brings the best-of-breed development tools built by the awesome Chef community to your workstation with just a few clicks. ...
Apache Kafka is a distributed streaming platform. Kafka 2.0 supports Kerberos authentication, Enabling Kerberos Authentication Using the Wizard on cloudera m...
Deployment error during setting zookeeper on a new cluster using Cloudera Manager UI, when an earlier cluster containing zookeeper earlier was built using Cl...
We were deploying one of the cluster on our lab environment which is used by everyone. So the lab has it own share of stale information on it.
Python API for cloudera is really nice, apart from getting the cluster setup, we can also do configuration and automation. We use a lot of automation using C...
This is the second follow up post. In the earlier post
This is the second follow up post. In the earlier post Setting Up Cloudera Manager Services Using Cloudera API [Part 1] we install the cloudera management se...
Cloudera API is a very convenient way to setup a cluster and do more.
This is a basic steps to get connected with cloudera manager.
0.0.0.0 [Cloudera]
We were working on setting up a cluster, but the Hue URL was set to a private IP of the server. As we had setup all the nodes to access each other using a pr...
This cookbook [currently as of now] can be used to setup a Cloudera Manager Server (Management Server) running on MySQL/Postgres database. But the intended u...
Python API for cloudera is really nice, apart from getting the cluster setup, we can also do configuration and automation. We use a lot of automation using C...
This is the second follow up post. In the earlier post
This is the second follow up post. In the earlier post Setting Up Cloudera Manager Services Using Cloudera API [Part 1] we install the cloudera management se...
Cloudera API is a very convenient way to setup a cluster and do more.
This is a basic steps to get connected with cloudera manager.
Deployment error during setting zookeeper on a new cluster using Cloudera Manager UI, when an earlier cluster containing zookeeper earlier was built using Cl...
We were deploying one of the cluster on our lab environment which is used by everyone. So the lab has it own share of stale information on it.
We had a cluster environment which we needed to update the /etc/hosts file. Which would help communicate between the server over a private network. Our serve...
GNU find searches the directory tree rooted at each given file name by evaluating the given expression from left to right, according to the rules of preceden...
CouchDB is a database that completely embraces the web. Store your data with JSON documents. Access your documents and query your indexes with your web brows...
After a longtime I was on my windows machine and had to make it feel more like my linux machine. So install the thing what everyone else does cygwin. Surpise...
Data Bags are a way to store information on the chef-server which all the cookbooks can access. Few more additional advantages are that we can encrypt the da...
In this blog post we will be setting up a postgres database on docker using Dockerfile.
In this blog post we will be setting up a postgres database on docker.
neo4j_doc_manager.
We had a requirement where we wanted to have all the data which is in mongodb to be replicated on neo4j to show few graphs. Here is quick way to demonstrate ...
Was setting up jekyll using docker, but was not able to install ruby on the docker machine as apt-add-repository was not working. Below is my Dockerfile when...
Pandoc is an opensource utility to create documents from markdown. We can create PDF, Doc, doc, html and other formats. And can be also used to convert html ...
Welcome back to our series on datastream workflows! In our previous article, we explored the basics of setting up a datastream workflow.
This guide provides detailed instructions on setting up a datastream connection from Cloud SQL (MySQL) to BigQuery. We aim to simplify the process by using d...
We will walk you through the process of setting up HashiCorp Vault to effectively manage your secrets. This guide is an gettinf started with vault with terra...
In this blog post, we will venture into exporting data from Firestore and Datastore modes. Find the code on Github.
Data is invaluable, and as any developer would testify, ensuring its security and availability in the event of unintended deletion or modification is paramou...
Ensuring consistency and streamlining infrastructure provisioning is crucial for effective cloud management. Creating custom virtual machine (VM) images on G...
Using Ansible within the initialization script of a Google Compute Engine (GCE) instance can streamline and automate the configuration and setup of your virt...
Security is paramount when setting up resources in the cloud or on-premises. It encompasses various layers of protection, including network security, encrypt...
null_resource with local-exec and remote-exec in Terraform
Terraform, a widely adopted Infrastructure as Code (IaC) tool, offers a rich array of resources and provisioners for efficiently managing and configuring inf...
BigQuery, Google’s fully-managed and serverless data warehouse, empowers organizations to analyze massive datasets with remarkable speed and efficiency. But ...
Google BigQuery is a powerful and cost-effective data warehouse that enables you to analyze massive datasets quickly. However, to get the most out of BigQuer...
Managing resources in your Google Cloud Platform (GCP) projects is a complex task, especially when dealing with multiple projects. Fortunately, Google offers...
null_resource
When working with Terraform, it’s crucial to ensure that your infrastructure follows certain conditions or constraints. While Terraform provides validation m...
Validation in Terraform is an essential practice to detect and prevent errors early in the infrastructure provisioning process. By incorporating robust valid...
Kerberos authentication in Google Dataproc clusters offers enhanced security and user isolation. It’s a crucial step for multi-tenancy, encryption, and user ...
TL;DR In this post we will be setting up a scheduled job to take backup for Bigtable table in avro format.
Cloud VPN securely connects your peer network to your Virtual Private Cloud (VPC) network through an IPsec VPN connection.
Stackdriver driver provides monitoring, logging, error reporting, tracing and debugging services. Currently known as Google Cloud’s operations suite.
Cloud SQL is a fully managed service which includes MySQL, PostgreSQL, Microsoft SQL Server. And for horizontal scalability, consider using Cloud Spanner.
Google Cloud offers IAM, which lets you give more granular access to specific Google Cloud resources and prevents unwanted access to other resources.
Cloud Storage is Google Cloud’s object storage service, and it allows worldwide storage and retrieval of any amount of data at anytime.
CMAN v3 is a Corsync plugin that monitors the names and number of active cluster nodes in order to deliver membership and quorum information to clients (such...
HAProxy is the Reliable, High Performance TCP/HTTP Load Balancer and it works nicely with Deveo Cluster setup.
NGINX for HTTP load balancing.
NGINX is a free, open-source, high-performance HTTP server and reverse proxy, as well as an IMAP/POP3 proxy server. NGINX is known for its high performance, ...
getmerge command takes a source directory and a destination file as input and concatenates files in src into the destination local file.
YARN distcp.
Usually distcp and other batch jobs run for a very long time. This is fine if you are running a HADOOP environment without kerberos. When we kerberize a clus...
distcp.
When you see this error default thing to do is the set the HADOOP_CLIENT_OPTS config on your env. In My case did a typo HADDOP_CLIENT_OPTS which caused us sp...
doBulkLoad - LoadIncrementalHfiles failed. This is recoverable. Hbase Encryption Zone.
We were working with a team who were uploading data into HBASE using the doBulkLoad, also we had moved the /hbase to encryption zone. The issue started after...
HDFS health using fsck.
When we have large data sets on the cluster, there will be corruptions of blocks. This could be due to disk or any other.
HBase Error Could not seek StoreFileScanner Invalid HFile block magic
We ran into an issue which was causing HBase to fail. When the job was rerun then it would work again. In the logs we got 2 errors.
We were deploying one of the cluster on our lab environment which is used by everyone. So the lab has it own share of stale information on it.
We had a cluster environment which we needed to update the /etc/hosts file. Which would help communicate between the server over a private network. Our serve...
Python API for cloudera is really nice, apart from getting the cluster setup, we can also do configuration and automation. We use a lot of automation using C...
This is the second follow up post. In the earlier post
This is the second follow up post. In the earlier post Setting Up Cloudera Manager Services Using Cloudera API [Part 1] we install the cloudera management se...
Cloudera API is a very convenient way to setup a cluster and do more.
This is a basic steps to get connected with cloudera manager.
These are a set of testing which we can do on a Hadoop environment. These are basic testing to make sure the environment is setup correctly.
0.0.0.0 [Cloudera]
We were working on setting up a cluster, but the Hue URL was set to a private IP of the server. As we had setup all the nodes to access each other using a pr...
Setting up yum repos on RHEL using httpd. We will be setting up httpd and yum repo on top of it. So that we can access yum over http.
If using access_provider = ldap, this option is mandatory. It specifies an LDAP search filter criteria that must be met for the user to be granted access on ...
Upgrading Cloudera Manager 5 to the Latest Cloudera Manager, In most cases it is possible to complete the following upgrade without shutting down most CDH se...
Apache Hive is a powerful data warehousing application built on top of Hadoop; it enables you to access your data using Hive QL, a language that is similar t...
There are inherent structural differences between how Windows and Linux handle system users. The user schemas used in Active Directory and standard LDAPv3 di...
There are inherent structural differences between how Windows and Linux handle system users. The user schemas used in Active Directory and standard LDAPv3 di...
Pentaho Data Integration (PDI) is a suite of open source Business Intelligence (BI) products which provide data integration, OLAP services, reporting, dashb...
We have 2 domains forests in our environment, ABC and XYZ. We were not able to authenticate normal users from either of the domains. Most of the information ...
eth1 to eth0 on EDGE node.
Adding route to all the slaves which reside on a private network to communicate with External Server directly using an EDGE node using Interface Forwarding.
tarball.
This is a simple zookeeper playbook, to quickly start zookeeper running on a single or more nodes, in a clustered mode.
This is a simple Storm Cluster Setup. We are using a dedicated Zookeeper Cluster/Node, instead of the standalone zkserver. Below is how we will deploy our c...
This is a simple Kafka setup. In this setup we are running kafka over a dedicated zookeeper service. (NOT the standalone zookeeper which comes with kafka)
tarball.
Setting up Hadoop using Ansible, we will be using cdh5 tarball for installation of the cluster.
The Apache Cassandra database is the right choice when you need scalability and high availability without compromising performance. Linear scalability and pr...
Using HBase in production often requires that you turn many knobs to make it hum as expected. More Here http://hbase.apache.org/0.94/book/performance.html
HBase is an open source, non-relational, distributed database modeled after Google’s BigTable and written in Java. It is developed as part of Apache Software...
Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Kafka is a distributed, partitioned, replicated commit log service. It pro...
sysctl.conf parameters.
Performace tuning Hadoop at kernel level. sysctl is an interface that allows you to make changes to a running Linux kernel. With /etc/sysctl.conf you can con...
Swappiness is a Linux kernel parameter that controls the relative weight given to swapping out runtime memory, as opposed to dropping pages from the system p...
First lets setup the Hbase Configuration Files. For pseudo-distributed replace with 'localhost'
Enable Authorization on HBase.
doBulkLoad - LoadIncrementalHfiles failed. This is recoverable. Hbase Encryption Zone.
We were working with a team who were uploading data into HBASE using the doBulkLoad, also we had moved the /hbase to encryption zone. The issue started after...
HBase Error Could not seek StoreFileScanner Invalid HFile block magic
We ran into an issue which was causing HBase to fail. When the job was rerun then it would work again. In the logs we got 2 errors.
Using HBase in production often requires that you turn many knobs to make it hum as expected. More Here http://hbase.apache.org/0.94/book/performance.html
getmerge command takes a source directory and a destination file as input and concatenates files in src into the destination local file.
HDFS health using fsck.
When we have large data sets on the cluster, there will be corruptions of blocks. This could be due to disk or any other.
HBase Error Could not seek StoreFileScanner Invalid HFile block magic
We ran into an issue which was causing HBase to fail. When the job was rerun then it would work again. In the logs we got 2 errors.
This is the second follow up post. In the earlier post
Apache Hive is a powerful data warehousing application built on top of Hadoop; it enables you to access your data using Hive QL, a language that is similar t...
Host file contains IP followed by the FQDN which can be used to reach that IP address. Host file takes precedence over your DNS servers. In Microsoft operati...
Setting up yum repos on RHEL using httpd. We will be setting up httpd and yum repo on top of it. So that we can access yum over http.
HTTPS is a protocol for secure communication over a computer network which is widely used on the Internet. HTTPS consists of communication over Hypertext Tra...
0.0.0.0 [Cloudera]
We were working on setting up a cluster, but the Hue URL was set to a private IP of the server. As we had setup all the nodes to access each other using a pr...
python 2.7.x on Centos 6.5/6.6
By default centos comes with python 2.6. In most of the cases we might need python 2.7 or later to be installed. Below are few ways to install python 2.7 on ...
postgres and mysql.
After the zabbix setup is done, we would like to take backup of the database. Here we are doing a complete backup. But if we are looking for only configurati...
iptables - Port forwarding from one interface to another.
iptables is a user-space application program that allows a system administrator to configure the tables provided by the Linux kernel firewall (implemented as...
Apache Kafka is a distributed streaming platform. Kafka 2.0 supports Kerberos authentication, Enabling Kerberos Authentication Using the Wizard on cloudera m...
Deployment error during setting zookeeper on a new cluster using Cloudera Manager UI, when an earlier cluster containing zookeeper earlier was built using Cl...
This is a simple Kafka setup. In this setup we are running kafka over a dedicated zookeeper service. (NOT the standalone zookeeper which comes with kafka)
Apache Kafka is a distributed streaming platform. Kafka 2.0 supports Kerberos authentication, Enabling Kerberos Authentication Using the Wizard on cloudera m...
Python API for cloudera is really nice, apart from getting the cluster setup, we can also do configuration and automation. We use a lot of automation using C...
We have 2 domains forests in our environment, ABC and XYZ. We were not able to authenticate normal users from either of the domains. Most of the information ...
Kitchen-Vagrant has the capability to spin up a windows instance for testing. To make it work you will need the vagrant-winrm to be installted on the worksta...
Data Bags are a way to store information on the chef-server which all the cookbooks can access. Few more additional advantages are that we can encrypt the da...
KVM is a kernel-based Virutal Machine which grows quickly in maturity and popularity in the Linux server market. Red Hat officially dropped Xen in favor of K...
getmerge command takes a source directory and a destination file as input and concatenates files in src into the destination local file.
Deployment error during setting zookeeper on a new cluster using Cloudera Manager UI, when an earlier cluster containing zookeeper earlier was built using Cl...
We were deploying one of the cluster on our lab environment which is used by everyone. So the lab has it own share of stale information on it.
We had a cluster environment which we needed to update the /etc/hosts file. Which would help communicate between the server over a private network. Our serve...
Python API for cloudera is really nice, apart from getting the cluster setup, we can also do configuration and automation. We use a lot of automation using C...
This is the second follow up post. In the earlier post
This is the second follow up post. In the earlier post Setting Up Cloudera Manager Services Using Cloudera API [Part 1] we install the cloudera management se...
Cloudera API is a very convenient way to setup a cluster and do more.
This is a basic steps to get connected with cloudera manager.
0.0.0.0 [Cloudera]
We were working on setting up a cluster, but the Hue URL was set to a private IP of the server. As we had setup all the nodes to access each other using a pr...
After a longtime I was on my windows machine and had to make it feel more like my linux machine. So install the thing what everyone else does cygwin. Surpise...
Monitoring for mongodb using nagiosxi is straight forword but you might have some issues when we are setting up. Here are few issues which might come up usin...
CouchDB is a database that completely embraces the web. Store your data with JSON documents. Access your documents and query your indexes with your web brows...
MongoDB is an open-source document database, and leading NoSQL database. MongoDB is written in c++. Below is a brief document about installing a mongodb on a...
Data Bags are a way to store information on the chef-server which all the cookbooks can access. Few more additional advantages are that we can encrypt the da...
GNU find searches the directory tree rooted at each given file name by evaluating the given expression from left to right, according to the rules of preceden...
When we were working on a nagios monitoring system we were migrating from a 32bit nagios to a 64bit Architecture. Most of the graphs are not working as the R...
HTTPS is a protocol for secure communication over a computer network which is widely used on the Internet. HTTPS consists of communication over Hypertext Tra...
The standalone installation of Chef server creates a working installation on a single server. This installation is also useful when you are installing Chef s...
Pandoc is an opensource utility to create documents from markdown. We can create PDF, Doc, doc, html and other formats. And can be also used to convert html ...
Checking port available on a remote machine using nc command instead of telnet. Same command can be used to check on a remote server as well, change the 127....
Linux Unified Key Setup-on-disk-format (or LUKS) allows you to encrypt partitions on your Linux computer. This is particularly important when it comes to mo...
How to move an existing data directory in mysql to a new location. We were running out of space and had to move the existing data directory to a new drive. B...
We were running out of disk space on one of the databases server, we need to get information on what the current table/database usage was. Below are few comm...
python 2.7.x on Centos 6.5/6.6
By default centos comes with python 2.6. In most of the cases we might need python 2.7 or later to be installed. Below are few ways to install python 2.7 on ...
postgres and mysql.
After the zabbix setup is done, we would like to take backup of the database. Here we are doing a complete backup. But if we are looking for only configurati...
There might be situations where you have a remote/zabbix server which does not have internet connectivity, due to security or other reasons. So we create a ...
zabbix-extra on Zabbix Version 2.4.
SNMP Builder/Extra is an add-in for Zabbix. It provides new web interface components to browse MIB trees and values. SNMP OIDs can then be automatically conv...
The System Security Services Daemon (SSSD) provides a set of daemons to manage access to remote directories and authentication mechanisms. It provides Name S...
Setting up yum repos on RHEL using httpd. We will be setting up httpd and yum repo on top of it. So that we can access yum over http.
If using access_provider = ldap, this option is mandatory. It specifies an LDAP search filter criteria that must be met for the user to be granted access on ...
Upgrading Cloudera Manager 5 to the Latest Cloudera Manager, In most cases it is possible to complete the following upgrade without shutting down most CDH se...
Apache Hive is a powerful data warehousing application built on top of Hadoop; it enables you to access your data using Hive QL, a language that is similar t...
There are inherent structural differences between how Windows and Linux handle system users. The user schemas used in Active Directory and standard LDAPv3 di...
There are inherent structural differences between how Windows and Linux handle system users. The user schemas used in Active Directory and standard LDAPv3 di...
iptables - Port forwarding from one interface to another.
iptables is a user-space application program that allows a system administrator to configure the tables provided by the Linux kernel firewall (implemented as...
squid as a sibling to an already existing Parent squid.
Squid is a caching and forwarding web proxy. It has a wide variety of uses, from speeding up a web server by caching repeated requests; to caching web, DNS a...
CMAN v3 is a Corsync plugin that monitors the names and number of active cluster nodes in order to deliver membership and quorum information to clients (such...
eth1 to eth0 on EDGE node.
Adding route to all the slaves which reside on a private network to communicate with External Server directly using an EDGE node using Interface Forwarding.
parted.
GUID Partition Table (GPT) is a standard for the layout of the partition table on a physical hard disk, using globally unique identifiers (GUID). Although it...
A Network File System (NFS) allows remote hosts to mount file systems over a network and interact with those file systems as though they are mounted locally....
tarball.
This is a simple zookeeper playbook, to quickly start zookeeper running on a single or more nodes, in a clustered mode.
This is a simple Storm Cluster Setup. We are using a dedicated Zookeeper Cluster/Node, instead of the standalone zkserver. Below is how we will deploy our c...
This is a simple Kafka setup. In this setup we are running kafka over a dedicated zookeeper service. (NOT the standalone zookeeper which comes with kafka)
tarball.
Setting up Hadoop using Ansible, we will be using cdh5 tarball for installation of the cluster.
Apache Spark is an in-memory distributed data analysis platform– primarily targeted at speeding up batch analysis jobs, iterative machine learning jobs, inte...
You can enter a linux system without entring a password using below steps. We will be creating a ssh jey which will be share between the servers, which will ...
The Apache Cassandra database is the right choice when you need scalability and high availability without compromising performance. Linear scalability and pr...
The SpagoBI project is a free software/open source initiative by the SpagoBI Labs of Engineering Group. It aims to realize the most complete 100% open source...
KVM is a kernel-based Virutal Machine which grows quickly in maturity and popularity in the Linux server market. Red Hat officially dropped Xen in favor of K...
Using HBase in production often requires that you turn many knobs to make it hum as expected. More Here http://hbase.apache.org/0.94/book/performance.html
HBase is an open source, non-relational, distributed database modeled after Google’s BigTable and written in Java. It is developed as part of Apache Software...
Was installing Java today, this is an easy install thanks to ppa:webupd8team/java, but when I tried it was not working, but has worked for me all this while.
zabbix-java-gateway on Centos 6.5
Zabbix 2.0 adds native support for monitoring JMX applications by introducing a new Zabbix daemon called Zabbix Java gateway. Zabbix Java gateway is a daemon...
HAProxy is the Reliable, High Performance TCP/HTTP Load Balancer and it works nicely with Deveo Cluster setup.
npm behind a proxy
npm stands for Node Package Manager, and is the default package manager for the JavaScript runtime environment Node.js.
Below are the list required to integrate NodeJS and Kafka. This is a simple HOWTO to get started.
Kafka-node is a Node.js client with Zookeeper integration for Apache Kafka 0.8.1 and later. The Zookeeper integration does the following jobs:
kafka-node
Now that we have Kafka and NodeJS ready. Lets some data to our Kafka Cluster.
What we are trying to achieve ?
Node.js is an open-source, cross-platform runtime environment for developing server-side web applications. Node.js applications are written in JavaScript and...
Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Kafka is a distributed, partitioned, replicated commit log service. It pro...
Got into this issue on the server, when trying to install mysql-server.
Have updated and explanation in the conf file below.
sysctl is an interface that allows you to make changes to a running Linux kernel. With /etc/sysctl.conf you can configure various Linux networking and system...
Current IP of the server : 10.130.18.11. Virtual IPs being assigned to server : 10.130.18.22, 10.130.18.23, 10.130.18.24 to our server.. Go to network-scrip...
Changing Timezone in Ubuntu server.
sysctl.conf parameters.
Performace tuning Hadoop at kernel level. sysctl is an interface that allows you to make changes to a running Linux kernel. With /etc/sysctl.conf you can con...
The Java Development Kit (JDK) is an implementation of either one of the Java SE, Java EE or Java ME platforms released by Oracle Corporation in the form of ...
Default passwd is set as username@123 As we are setting the chage, user needs to change the passwd on first login.
nginx
Nginx (pronounced engine x) is a web server with a strong focus on high concurrency, performance and low memory usage. It can also act as a reverse proxy ser...
Tsung (formerly known as idx-Tsunami) is a stress testing tool written in the Erlang language and distributed under the GPL license. It can currently stress ...
Swappiness is a Linux kernel parameter that controls the relative weight given to swapping out runtime memory, as opposed to dropping pages from the system p...
Siege is an http load testing and benchmarking utility. It was designed to let web developers measure their code under duress, to see how it will stand up to...
heartbeat for nginx.
Heartbeat is a daemon that provides cluster infrastructure (communication and membership) services to its clients. This allows clients to know about the pres...
mod_jk and httpd on Centos - In-Memory Session Replication.
In an earlier post we did a basic session based replication, but the session was not redundant.
Setting up SpagoBI 5.1 on Tomcat with MySQL post. Details on Tomcat Clustering.
NGINX for HTTP load balancing.
NGINX is a free, open-source, high-performance HTTP server and reverse proxy, as well as an IMAP/POP3 proxy server. NGINX is known for its high performance, ...
httpd is the Apache HyperText Transfer Protocol (HTTP) server program. It is designed to be run as a standalone daemon process. When used like this it will c...
Easy way to collect files recursively over a sftp server is to connect to the server over scp and do scp -r. Problem was that the device we were connecting ...
Linux Unified Key Setup-on-disk-format (or LUKS) allows you to encrypt partitions on your Linux computer. This is particularly important when it comes to mo...
Monitoring for mongodb using nagiosxi is straight forword but you might have some issues when we are setting up. Here are few issues which might come up usin...
MongoDB is an open-source document database, and leading NoSQL database. MongoDB is written in c++. Below is a brief document about installing a mongodb on a...
We were working on nagios and found that after our migration, service group summary was not working. You might get below error on the screen and the solution...
Zabbix history table gets really big, and if you are in a situation where you want to clean it up. Then we can do so, using the below steps.
This is a quick reference for few of the commands which I use often.
Migrating an old Nagios backup to a new nagiosxi server. Migration is simple in Nagios XI, you a backup and restore it. Once that is done we need to update/a...
HTTPS is a protocol for secure communication over a computer network which is widely used on the Internet. HTTPS consists of communication over Hypertext Tra...
Nagios monitors your entire IT infrastructure to ensure systems, applications, services, and business processes are functioning properly. In the event of a f...
postgres and mysql.
After the zabbix setup is done, we would like to take backup of the database. Here we are doing a complete backup. But if we are looking for only configurati...
There might be situations where you have a remote/zabbix server which does not have internet connectivity, due to security or other reasons. So we create a ...
In zabbix we dont have a better way to capture snmptraps. we have to manually create an item and corresponfding trigger to handle a trap arriving from the de...
zabbix-extra on Zabbix Version 2.4.
SNMP Builder/Extra is an add-in for Zabbix. It provides new web interface components to browse MIB trees and values. SNMP OIDs can then be automatically conv...
Receiving SNMP traps is the opposite to querying SNMP-enabled devices. In this case the information is sent from a SNMP-enabled device and is collected or “t...
zabbix-java-gateway on Centos 6.5
Zabbix 2.0 adds native support for monitoring JMX applications by introducing a new Zabbix daemon called Zabbix Java gateway. Zabbix Java gateway is a daemon...
Zabbix is the ultimate enterprise-level software designed for monitoring availability and performance of IT infrastructure components. Zabbix is open source ...
This script can be used to monitor Namenode Parameters. This script can be used to Generate Zabbix Import XML or Send monitoring data to Zabbix server.
This section describes how to get monitoring data from Zabbix server through the command line and export these to a CSV formatted file.
This script is to create a xml import file from a csv file. We use 2 csv file for this.
Send Zabbix metric by reading from a XML file. This needs package zbxsend from : https://github.com/ahmedzbyr/zbxsend
Module used to generate Zabbix import xml for snmp traps, Below are steps to create a Zabbix Import for Traps. These traps are captured from the snmptrap fun...
How to move an existing data directory in mysql to a new location. We were running out of space and had to move the existing data directory to a new drive. B...
We were running out of disk space on one of the databases server, we need to get information on what the current table/database usage was. Below are few comm...
We were working on nagios and found that after our migration, service group summary was not working. You might get below error on the screen and the solution...
Monitoring for mongodb using nagiosxi is straight forword but you might have some issues when we are setting up. Here are few issues which might come up usin...
This is a quick reference for few of the commands which I use often.
When we were working on a nagios monitoring system we were migrating from a 32bit nagios to a 64bit Architecture. Most of the graphs are not working as the R...
HTTPS is a protocol for secure communication over a computer network which is widely used on the Internet. HTTPS consists of communication over Hypertext Tra...
Nagios monitors your entire IT infrastructure to ensure systems, applications, services, and business processes are functioning properly. In the event of a f...
postgres and mysql.
After the zabbix setup is done, we would like to take backup of the database. Here we are doing a complete backup. But if we are looking for only configurati...
There might be situations where you have a remote/zabbix server which does not have internet connectivity, due to security or other reasons. So we create a ...
zabbix-extra on Zabbix Version 2.4.
SNMP Builder/Extra is an add-in for Zabbix. It provides new web interface components to browse MIB trees and values. SNMP OIDs can then be automatically conv...
zabbix-java-gateway on Centos 6.5
Zabbix 2.0 adds native support for monitoring JMX applications by introducing a new Zabbix daemon called Zabbix Java gateway. Zabbix Java gateway is a daemon...
This script can be used to monitor Namenode Parameters. This script can be used to Generate Zabbix Import XML or Send monitoring data to Zabbix server.
This section describes how to get monitoring data from Zabbix server through the command line and export these to a CSV formatted file.
This script is to create a xml import file from a csv file. We use 2 csv file for this.
Send Zabbix metric by reading from a XML file. This needs package zbxsend from : https://github.com/ahmedzbyr/zbxsend
Module used to generate Zabbix import xml for snmp traps, Below are steps to create a Zabbix Import for Traps. These traps are captured from the snmptrap fun...
We were working on nagios and found that after our migration, service group summary was not working. You might get below error on the screen and the solution...
Monitoring for mongodb using nagiosxi is straight forword but you might have some issues when we are setting up. Here are few issues which might come up usin...
Migrating an old Nagios backup to a new nagiosxi server. Migration is simple in Nagios XI, you a backup and restore it. Once that is done we need to update/a...
HTTPS is a protocol for secure communication over a computer network which is widely used on the Internet. HTTPS consists of communication over Hypertext Tra...
Nagios monitors your entire IT infrastructure to ensure systems, applications, services, and business processes are functioning properly. In the event of a f...
Checking port available on a remote machine using nc command instead of telnet. Same command can be used to check on a remote server as well, change the 127....
npm behind a proxy
npm stands for Node Package Manager, and is the default package manager for the JavaScript runtime environment Node.js.
Below are the list required to integrate NodeJS and Kafka. This is a simple HOWTO to get started.
Kafka-node is a Node.js client with Zookeeper integration for Apache Kafka 0.8.1 and later. The Zookeeper integration does the following jobs:
kafka-node
Now that we have Kafka and NodeJS ready. Lets some data to our Kafka Cluster.
What we are trying to achieve ?
Node.js is an open-source, cross-platform runtime environment for developing server-side web applications. Node.js applications are written in JavaScript and...
Have updated and explanation in the conf file below.
npm behind a proxy
npm stands for Node Package Manager, and is the default package manager for the JavaScript runtime environment Node.js.
Pentaho Data Integration (PDI) is a suite of open source Business Intelligence (BI) products which provide data integration, OLAP services, reporting, dashb...
Python virtual environment creates a isoloated workspace of python work. This helps in creating project specific virtual environment without worrying about ...
neo4j_doc_manager.
We had a requirement where we wanted to have all the data which is in mongodb to be replicated on neo4j to show few graphs. Here is quick way to demonstrate ...
pyyaml [Python]
PyYAML - The next generation YAML parser and emitter for Python.
I was working on getting post headers for my post on this blog. I had couple of images from unsplash. But the header for the post need to be a little more ho...
python 2.7.x on Centos 6.5/6.6
By default centos comes with python 2.6. In most of the cases we might need python 2.7 or later to be installed. Below are few ways to install python 2.7 on ...
List Comprehensions provides easy and functional way to create list in python. We could make a single line of code which otherwise would take a few lines. l...
Python is a clear and powerful object-oriented programming language, comparable to Perl, Ruby, Scheme, or Java. This is basic documentation for getting star...
parted.
GUID Partition Table (GPT) is a standard for the layout of the partition table on a physical hard disk, using globally unique identifiers (GUID). Although it...
Deployment error during setting zookeeper on a new cluster using Cloudera Manager UI, when an earlier cluster containing zookeeper earlier was built using Cl...
We were deploying one of the cluster on our lab environment which is used by everyone. So the lab has it own share of stale information on it.
We had a cluster environment which we needed to update the /etc/hosts file. Which would help communicate between the server over a private network. Our serve...
GNU find searches the directory tree rooted at each given file name by evaluating the given expression from left to right, according to the rules of preceden...
When we were working on a nagios monitoring system we were migrating from a 32bit nagios to a 64bit Architecture. Most of the graphs are not working as the R...
RPM (Redhat Package Manager) is the most popular package utility and is used mostly on RHEL, Centos and Fedora. RPM helps user/admins to build, install, que...
Was working on a monitoring project, need to create a large file to test notifications. Here is how we can do that.
RPM (Redhat Package Manager) is the most popular package utility and is used mostly on RHEL, Centos and Fedora. RPM helps user/admins to build, install, que...
When we were working on a nagios monitoring system we were migrating from a 32bit nagios to a 64bit Architecture. Most of the graphs are not working as the R...
postgres and mysql.
After the zabbix setup is done, we would like to take backup of the database. Here we are doing a complete backup. But if we are looking for only configurati...
There might be situations where you have a remote/zabbix server which does not have internet connectivity, due to security or other reasons. So we create a ...
In zabbix we dont have a better way to capture snmptraps. we have to manually create an item and corresponfding trigger to handle a trap arriving from the de...
zabbix-extra on Zabbix Version 2.4.
SNMP Builder/Extra is an add-in for Zabbix. It provides new web interface components to browse MIB trees and values. SNMP OIDs can then be automatically conv...
Receiving SNMP traps is the opposite to querying SNMP-enabled devices. In this case the information is sent from a SNMP-enabled device and is collected or “t...
The SpagoBI project is a free software/open source initiative by the SpagoBI Labs of Engineering Group. It aims to realize the most complete 100% open source...
Apache Spark is an in-memory distributed data analysis platform– primarily targeted at speeding up batch analysis jobs, iterative machine learning jobs, inte...
squid as a sibling to an already existing Parent squid.
Squid is a caching and forwarding web proxy. It has a wide variety of uses, from speeding up a web server by caching repeated requests; to caching web, DNS a...
HTTPS is a protocol for secure communication over a computer network which is widely used on the Internet. HTTPS consists of communication over Hypertext Tra...
The System Security Services Daemon (SSSD) provides a set of daemons to manage access to remote directories and authentication mechanisms. It provides Name S...
If using access_provider = ldap, this option is mandatory. It specifies an LDAP search filter criteria that must be met for the user to be granted access on ...
There are inherent structural differences between how Windows and Linux handle system users. The user schemas used in Active Directory and standard LDAPv3 di...
There are inherent structural differences between how Windows and Linux handle system users. The user schemas used in Active Directory and standard LDAPv3 di...
This is a simple Storm Cluster Setup. We are using a dedicated Zookeeper Cluster/Node, instead of the standalone zkserver. Below is how we will deploy our c...
Apache Spark is an in-memory distributed data analysis platform– primarily targeted at speeding up batch analysis jobs, iterative machine learning jobs, inte...
Kitchen-Vagrant has the capability to spin up a windows instance for testing. To make it work you will need the vagrant-winrm to be installted on the worksta...
Tsung (formerly known as idx-Tsunami) is a stress testing tool written in the Erlang language and distributed under the GPL license. It can currently stress ...
Siege is an http load testing and benchmarking utility. It was designed to let web developers measure their code under duress, to see how it will stand up to...
These are a set of testing which we can do on a Hadoop environment. These are basic testing to make sure the environment is setup correctly.
postgres and mysql.
After the zabbix setup is done, we would like to take backup of the database. Here we are doing a complete backup. But if we are looking for only configurati...
There might be situations where you have a remote/zabbix server which does not have internet connectivity, due to security or other reasons. So we create a ...
zabbix-extra on Zabbix Version 2.4.
SNMP Builder/Extra is an add-in for Zabbix. It provides new web interface components to browse MIB trees and values. SNMP OIDs can then be automatically conv...
Receiving SNMP traps is the opposite to querying SNMP-enabled devices. In this case the information is sent from a SNMP-enabled device and is collected or “t...
Was setting up jekyll using docker, but was not able to install ruby on the docker machine as apt-add-repository was not working. Below is my Dockerfile when...
Kitchen-Vagrant has the capability to spin up a windows instance for testing. To make it work you will need the vagrant-winrm to be installted on the worksta...
Monitoring for mongodb using nagiosxi is straight forword but you might have some issues when we are setting up. Here are few issues which might come up usin...
CouchDB is a database that completely embraces the web. Store your data with JSON documents. Access your documents and query your indexes with your web brows...
MongoDB is an open-source document database, and leading NoSQL database. MongoDB is written in c++. Below is a brief document about installing a mongodb on a...
Data Bags are a way to store information on the chef-server which all the cookbooks can access. Few more additional advantages are that we can encrypt the da...
GNU find searches the directory tree rooted at each given file name by evaluating the given expression from left to right, according to the rules of preceden...
When we were working on a nagios monitoring system we were migrating from a 32bit nagios to a 64bit Architecture. Most of the graphs are not working as the R...
HBase is an open source, non-relational, distributed database modeled after Google’s BigTable and written in Java. It is developed as part of Apache Software...
Was installing Java today, this is an easy install thanks to ppa:webupd8team/java, but when I tried it was not working, but has worked for me all this while.
Changing Timezone in Ubuntu server.
Not enough physical memory is available to power this virtual machine
If you are using VMWare Workstation (or VMWare player) on Windows 8.1 and have just update Windows, specifically KB2995388, you may receive this error messag...
Below are the list required to integrate NodeJS and Kafka. This is a simple HOWTO to get started.
Kafka-node is a Node.js client with Zookeeper integration for Apache Kafka 0.8.1 and later. The Zookeeper integration does the following jobs:
kafka-node
Now that we have Kafka and NodeJS ready. Lets some data to our Kafka Cluster.
What we are trying to achieve ?
Node.js is an open-source, cross-platform runtime environment for developing server-side web applications. Node.js applications are written in JavaScript and...
Have updated and explanation in the conf file below.
sysctl is an interface that allows you to make changes to a running Linux kernel. With /etc/sysctl.conf you can configure various Linux networking and system...
Kitchen-Vagrant has the capability to spin up a windows instance for testing. To make it work you will need the vagrant-winrm to be installted on the worksta...
After a longtime I was on my windows machine and had to make it feel more like my linux machine. So install the thing what everyone else does cygwin. Surpise...
knife-windows plugin adds additional functionality to the Chef Knife CLI tool for configuring / interacting with nodes running Microsoft Windows.
Host file contains IP followed by the FQDN which can be used to reach that IP address. Host file takes precedence over your DNS servers. In Microsoft operati...
Was working on a monitoring project, need to create a large file to test notifications. Here is how we can do that.
The Chef Development Kit (ChefDK) brings the best-of-breed development tools built by the awesome Chef community to your workstation with just a few clicks. ...
Not enough physical memory is available to power this virtual machine
If you are using VMWare Workstation (or VMWare player) on Windows 8.1 and have just update Windows, specifically KB2995388, you may receive this error messag...
YARN distcp.
Usually distcp and other batch jobs run for a very long time. This is fine if you are running a HADOOP environment without kerberos. When we kerberize a clus...
distcp.
When you see this error default thing to do is the set the HADOOP_CLIENT_OPTS config on your env. In My case did a typo HADDOP_CLIENT_OPTS which caused us sp...
Zabbix history table gets really big, and if you are in a situation where you want to clean it up. Then we can do so, using the below steps.
postgres and mysql.
After the zabbix setup is done, we would like to take backup of the database. Here we are doing a complete backup. But if we are looking for only configurati...
There might be situations where you have a remote/zabbix server which does not have internet connectivity, due to security or other reasons. So we create a ...
In zabbix we dont have a better way to capture snmptraps. we have to manually create an item and corresponfding trigger to handle a trap arriving from the de...
zabbix-extra on Zabbix Version 2.4.
SNMP Builder/Extra is an add-in for Zabbix. It provides new web interface components to browse MIB trees and values. SNMP OIDs can then be automatically conv...
Receiving SNMP traps is the opposite to querying SNMP-enabled devices. In this case the information is sent from a SNMP-enabled device and is collected or “t...
zabbix-java-gateway on Centos 6.5
Zabbix 2.0 adds native support for monitoring JMX applications by introducing a new Zabbix daemon called Zabbix Java gateway. Zabbix Java gateway is a daemon...
Zabbix is the ultimate enterprise-level software designed for monitoring availability and performance of IT infrastructure components. Zabbix is open source ...
This script can be used to monitor Namenode Parameters. This script can be used to Generate Zabbix Import XML or Send monitoring data to Zabbix server.
This section describes how to get monitoring data from Zabbix server through the command line and export these to a CSV formatted file.
This script is to create a xml import file from a csv file. We use 2 csv file for this.
Send Zabbix metric by reading from a XML file. This needs package zbxsend from : https://github.com/ahmedzbyr/zbxsend
Module used to generate Zabbix import xml for snmp traps, Below are steps to create a Zabbix Import for Traps. These traps are captured from the snmptrap fun...
Deployment error during setting zookeeper on a new cluster using Cloudera Manager UI, when an earlier cluster containing zookeeper earlier was built using Cl...
This is the second follow up post. In the earlier post
This is the second follow up post. In the earlier post Setting Up Cloudera Manager Services Using Cloudera API [Part 1] we install the cloudera management se...