Subscribe to Stuck in an Infiniteloop        RSS Feed
-----

Apache Nifi Sensitive Value Encryption

Icon Leave Comment
This post is for my sanity and posterity if anyone else runs across this issue in the future and does not have the good fortune to come across a mailing list archive from last February.


Apache Nifi allows processors to have "sensitive values", i.e. once you type in the value in the properties of a processor, it is no longer visible. It is encrypted and stored in the flow.xml file on disk thusly:

      <property>
        <name>Password</name>
        <value>enc{AE06E2E77C38A0EA899DB37FB7F6E05FFBA6529B2E9F90C914962FF2DD594020}</value>
      </property>



This becomes a problem if you forget the value and need to reference it later, such as a SFTP password that you didn't set up. I beat my head against the wall for hours earlier this week before coming across this mailing list exchange. In this back and forth, a user was using ruby to generate sensitive values encrypted and put them in a flow.xml programmically. He was running into issues as I was, but in the reverse direction. I originally tried to use this hex encoded ciphertext with the command line openssl tool. This failed. The reason this failed was because the third party library Jasypt Nifi uses for string encryption uses a 16 byte salt instead of an 8 byte salt and no magic numbers/headers.

In the nifi.properties file you can specify a master passphrase and the encryption algorithm you wish to use. As a default, the passphrase is blank and defaults to "nififtw!" and the algorithm is PBEWITHMD5AND256BITAES-CBC-OPENSSL.

That algorithm is the "standard" that openssl uses on the command line with aes-256-cbc. This is 1 iteration of MD5 of the password and a randomly generated 8 byte salt to create the key and initialization vector for the cipher. The magic string "Salted__" is written at the beginning of the file followed by the 8 byte salt and finally the ciphertext. If you want your encryption/decryption to be compatible with the command line equivalent

echo password | openssl aes-256-cbc -salt -in file_in -o file_out -pass stdin

and

echo password | openssl enc -d -aes-256-cbc -salt -in file_in -o file_out -pass stdin

you can shell out to the command or use the openssl library to do it yourself. This brings us to EVP_BytesToKey(). This function is the one that takes the password and 8 byte salt to generate the key and init vector. It requires an 8 byte salt or it will throw up.

All of this is a roundabout way to say that not using an 8 byte salt will ensure that openssl cannot decrypt your sensitive values.

Moreover, the EncryptContent processor cannot decrypt it either. The algorithm options in that processor are:

NiFi KDF legacy -> this is 1000 rounds of MD5 with a salt equal to the block size (16 bytes in the case of AES)
OpenSSL PKCS v1.5 EVP_BytesToKey -> this is the command line equivalent from above
Bcrypt
Scrypt
PBKDF2

It was mind boggling that none of the tools in NiFi itself could decrypt this value. This issue exists up to and in Apache NiFi 1.1.2 as of this writing.

In the mailing list, one of the devs rewrote EVP_BytesToKey in ruby to allow an arbitrary long salt size. The default ruby openssl module wraps the above code and will fail on a non 8 byte salt. I have taken this function and written a script that will take the hex encoding from the enc{} block in the flow.xml and output the decrypted value.

#!/usr/bin/env ruby

require 'openssl'
require 'getoptlong'

#This script will attempt to decrypt sensitive values in Apache NiFi's flow.xml
#See the HELP block below for more information

def bin_to_hex(s)
  s.each_byte.map { |b| b.to_s(16).rjust(2, '0') }.join
end

def evp_bytes_to_key(key_len, iv_len, md, salt, data, count)
  key = ''.bytes
  key_ix = 0
  iv = ''.bytes
  iv_ix = 0
  md_buf = ''.bytes
  n_key = key_len
  n_iv = iv_len
  i = 0
  salt_length = salt.length
  if data == nil
    return [key, iv]
  end
  add_md = 0
  while true
    md.reset
    if add_md > 0
      md.update md_buf
    end
    add_md += 1
    md.update data
    if nil != salt
      md.update salt[0..salt_length-1]
    end
    md_buf = md.digest
    (1..count-1).each do
      md.reset
      md.update md_buf
      md_buf = md.digest
    end
    i = 0
    if n_key > 0
      while true
        if n_key == 0
          break
        end
        if i == md_buf.length
          break
        end
        key[key_ix] = md_buf[i]
        key_ix += 1
        n_key -= 1
        i += 1
      end
    end
    if n_iv > 0 && i != md_buf.length
      while true
        if n_iv == 0
          break
        end
        if i == md_buf.length
          break
        end
        iv[iv_ix] = md_buf[i]
        iv_ix += 1
        n_iv -= 1
        i += 1
      end
    end
    if n_key == 0 && n_iv == 0
      break
    end
  end
  (0..md_buf.length-1).each do |j|
    md_buf[j] = '0'
  end
  [key, iv]
end

HELP=<<ENDHELP
Usage: ./decrypt_sensitive.rb -i /path/to/file -p[optional]

  --help,-h Print this help message
  -i        path to input file that is hex string inside of "enc{}" from the flow.xml
  -p        passphrase from nifi.properties: "nifi.sensitive.props.key"
            if not provided it will default to "nififtw!"

  This program will attempt to decrypt sensitive configuration items from the flow.xml
  in a NiFi instance. Hex encoding is 16 byte salt and the remainder is the ciphertext.
ENDHELP

opts = GetoptLong.new(
  ["--help","-h", GetoptLong::NO_ARGUMENT],
  ["-i", GetoptLong::REQUIRED_ARGUMENT],
  ["-p", GetoptLong::REQUIRED_ARGUMENT]
)


if ARGV.length < 1
  puts HELP
  exit
end

path = nil
master_passphrase = nil

opts.each do |opt, arg|
  case opt
    when "--help"
      puts HELP
      exit
    when "-i"
      path = arg
    when "-p"
      master_passphrase = arg
  end
end

master_passphrase = "nififtw!" unless master_passphrase != nil
puts "Master passphrase: #{master_passphrase}"

payload = File.read(path).strip

#this is the first 16 bytes of the enc {} nonsense in the flow.xml
master_salt = payload[0..31]
payload_bytes = [payload[32..-1].strip].pack('H*')

puts "Salt: #{master_salt} 16"
puts "Payload: #{payload[32..-1].strip}"

cipher = OpenSSL::Cipher.new 'AES-256-CBC'
cipher.decrypt

# If the salt was 8 bytes, this would work, but NiFi Jasypt uses a 16 byte salt
# This is the OpenSSL implementation
# cipher.pkcs5_keyivgen master_passphrase, master_salt, 1, OpenSSL::Digest::MD5.new

iterations = 1
(key, iv) = evp_bytes_to_key cipher.key_len, cipher.iv_len, OpenSSL::Digest::MD5.new, [master_salt].pack('H*'), master_passphrase, iterations

key = key.join
iv = iv.join

hex_key = bin_to_hex(key)
hex_iv = bin_to_hex(iv)

puts ""
puts "Output of EVP_BytesToKey"
puts "Hex key: #{hex_key} #{key.length}"
puts "Hex  IV: #{hex_iv} #{iv.length}"

puts ""

cipher.key = key
cipher.iv = iv

#decrypted = cipher.update plaintext
decrypted = cipher.update(payload_bytes)
decrypted << cipher.final

puts "Plaintext decrypted #{decrypted}"



Using the hex encoded encryption from the above in a file called enc_1:

Quote

$> ./decrypt_sensitive.rb -i enc_1 -p testpassword
Master passphrase: testpassword
Salt: AE06E2E77C38A0EA899DB37FB7F6E05F 16
Payload: FBA6529B2E9F90C914962FF2DD594020

Output of EVP_BytesToKey
Hex key: 2fb794e8adf21b2aa960c8e87caa74d411ffcba2369b5dac38726c10437c6249 32
Hex IV: b4103a85d1ece0eacba1ef327058da1d 16

Plaintext decrypted NewPasswordTest

0 Comments On This Entry

 

May 2017

S M T W T F S
  123456
78910111213
14151617181920
212223 24 252627
28293031     

Tags

    Recent Entries

    Search My Blog

    1 user(s) viewing

    1 Guests
    0 member(s)
    0 anonymous member(s)