getmerge Operation not Permitted

getmerge command takes a source directory and a destination file as input and concatenates files in src into the destination local file.

• Optionally -nl can be set to enable adding a newline character (LF) at the end of each file.
• -skip-empty-file can be used to avoid unwanted newline characters in case of empty files.

Examples:

hadoop fs -getmerge -nl /src /opt/output.txt
hadoop fs -getmerge -nl /src/file1.txt /src/file2.txt /output.txt

Exit Code: Returns 0 on success and non-zero on error.

During one of the processing. We get getmerge: Operation not permitted issue when we execute the below command. We found that the permission and the CRC file were created using different accounts.

hdfs dfs -getmerge /data/Part-000* /data/on_server/combine_file.txt 

When the first user1 created the getmerge and then user2 tried to do the same we get the above error.

Solution: Remove the .crc file as and the destination file and retry.

-rw-r--r-- 1 user1 grp    0 Mar 1 11:06 combine_file.txt
-rw-r--r-- 1 user2 grp   16 Mar 1 11:01 .combine_file.txt.crc

Ideally all processing has to be done by one process user or a service account.