Flink Savepoint metadata file is not getting created properly

On triggering flink savepoint as described in the document (SavepointDocument). The _metadata file gets created but the content malformed or incomplete "I G" .. Tried with flink version 1.4.x, 1.5.x, 1.6.0,1.6.1, 1.6.2, 1.7.0. But it works with Flink version 1.3.x, configuration and Savepoint CLI command executed are exactly the same.

Seems like the Savepoint stopped working for version > 1.3.x

Logs for 1.4.x, 1.5.x, 1.6.x, 1.7.x (Doesn't work)

2018-12-19 20:45:02,173 INFO org.apache.flink.client.cli.CliFrontend - -------------------------------------------------------------------------------- 2018-12-19 20:45:02,176 INFO org.apache.flink.client.cli.CliFrontend - Starting Command Line Client (Version: 1.7.0, Rev:49da9f9, Date:28.11.2018 @ 17:59:06 UTC) 2018-12-19 20:45:02,176 INFO org.apache.flink.client.cli.CliFrontend - OS current user: centos 2018-12-19 20:45:03,104 INFO org.apache.flink.client.cli.CliFrontend - Current Hadoop/Kerberos user: centos 2018-12-19 20:45:03,104 INFO org.apache.flink.client.cli.CliFrontend - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.144-b01 2018-12-19 20:45:03,104 INFO org.apache.flink.client.cli.CliFrontend - Maximum heap size: 3531 MiBytes 2018-12-19 20:45:03,104 INFO org.apache.flink.client.cli.CliFrontend - JAVA_HOME: (not set) 2018-12-19 20:45:03,110 INFO org.apache.flink.client.cli.CliFrontend - Hadoop version: 2.7.5 2018-12-19 20:45:03,111 INFO org.apache.flink.client.cli.CliFrontend - JVM Options: 2018-12-19 20:45:03,111 INFO org.apache.flink.client.cli.CliFrontend - -Dlog.file=/home/centos/flink_install/flink-1.7.0/log/flink-master.local 2018-12-19 20:45:03,111 INFO org.apache.flink.client.cli.CliFrontend - -Dlog4j.configuration=file:/home/centos/flink_install/flink-1.7.0/conf/log4j-cli.properties 2018-12-19 20:45:03,111 INFO org.apache.flink.client.cli.CliFrontend - -Dlogback.configurationFile=file:/home/centos/flink_install/flink-1.7.0/conf/logback.xml 2018-12-19 20:45:03,111 INFO org.apache.flink.client.cli.CliFrontend - Program Arguments: 2018-12-19 20:45:03,111 INFO org.apache.flink.client.cli.CliFrontend - list 2018-12-19 20:45:03,112 INFO org.apache.flink.client.cli.CliFrontend - Classpath: /home/centos/flink_install/flink-1.7.0/lib/flink-python_2.11-1.7.0.jar:/home/centos/flink_install/flink-1.7.0/lib/flink-shaded-hadoop2-uber-1.7.0.jar:/home/centos/flink_install/flink-1.7.0/lib/log4j-1.2.17.jar:/home/centos/flink_install/flink-1.7.0/lib/slf4j-log4j12-1.7.15.jar:/home/centos/flink_install/flink-1.7.0/lib/flink-dist_2.11-1.7.0.jar::: 2018-12-19 20:45:03,112 INFO org.apache.flink.client.cli.CliFrontend - -------------------------------------------------------------------------------- 2018-12-19 20:45:03,121 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, 9x.xx.xx.1 2018-12-19 20:45:03,121 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123 2018-12-19 20:45:03,121 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 8192m 2018-12-19 20:45:03,122 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 20480m 2018-12-19 20:45:03,122 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 10 2018-12-19 20:45:03,122 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1 2018-12-19 20:45:03,123 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.savepoints.dir, file:///mount/data/savepoints 2018-12-19 20:45:03,124 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 8081 2018-12-19 20:45:03,124 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.address, 9x.xx.xx.1 2018-12-19 20:45:03,629 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to centos (auth:SIMPLE) 2018-12-19 20:45:03,684 INFO org.apache.flink.client.cli.CliFrontend - Running 'list' command. 2018-12-19 20:45:04,245 INFO org.apache.flink.runtime.rest.RestClient - Rest client endpoint started. 2018-12-19 20:45:04,662 INFO org.apache.flink.client.cli.CliFrontend - Waiting for response... 2018-12-19 20:45:04,937 INFO org.apache.flink.client.cli.CliFrontend - Successfully retrieved list of jobs 2018-12-19 20:45:04,951 INFO org.apache.flink.runtime.rest.RestClient - Shutting down rest endpoint. 2018-12-19 20:45:04,954 INFO org.apache.flink.runtime.rest.RestClient - Rest endpoint shutdown complete.

Metadata file content (Not working versions)

I`g-^@^@^@^B^@^@^@^@^@^@^@^A^@^@^@^@^@^@^@^@

Logs for 1.3.x (Works 1.3.2 and 1.3.3)

2018-12-19 19:50:06,863 INFO org.apache.flink.client.CliFrontend - Starting Command Line Client (Version: 1.3.3, Rev:93e9a6e, Date:14.03.2018 @ 05:13:37 UTC) 2018-12-19 19:50:06,864 INFO org.apache.flink.client.CliFrontend - Current user: centos 2018-12-19 19:50:06,864 INFO org.apache.flink.client.CliFrontend - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.144-b01 2018-12-19 19:50:06,864 INFO org.apache.flink.client.CliFrontend - Maximum heap size: 3531 MiBytes 2018-12-19 19:50:06,864 INFO org.apache.flink.client.CliFrontend - JAVA_HOME: (not set) 2018-12-19 19:50:06,870 INFO org.apache.flink.client.CliFrontend - Hadoop version: 2.7.2 2018-12-19 19:50:06,870 INFO org.apache.flink.client.CliFrontend - JVM Options: 2018-12-19 19:50:06,870 INFO org.apache.flink.client.CliFrontend - -Dlog.file=/home/centos/flink_install/flink-1.3.3/log/flink-centos-client-flink-master.local.log 2018-12-19 19:50:06,870 INFO org.apache.flink.client.CliFrontend - -Dlog4j.configuration=file:/home/centos/flink_install/flink-1.3.3/conf/log4j-cli.properties 2018-12-19 19:50:06,870 INFO org.apache.flink.client.CliFrontend - -Dlogback.configurationFile=file:/home/centos/flink_install/flink-1.3.3/conf/logback.xml 2018-12-19 19:50:06,871 INFO org.apache.flink.client.CliFrontend - Program Arguments: 2018-12-19 19:50:06,871 INFO org.apache.flink.client.CliFrontend - savepoint 2018-12-19 19:50:06,871 INFO org.apache.flink.client.CliFrontend - d3c377067c466d5fc040068b53f60931 2018-12-19 19:50:06,871 INFO org.apache.flink.client.CliFrontend - /mnt/savepoints 2018-12-19 19:50:06,871 INFO org.apache.flink.client.CliFrontend - Classpath: /home/centos/flink_install/flink-1.3.3/lib/flink-python_2.11-1.3.3.jar:/home/centos/flink_install/flink-1.3.3/lib/flink-shaded-hadoop2-uber-1.3.3.jar:/home/centos/flink_install/flink-1.3.3/lib/log4j-1.2.17.jar:/home/centos/flink_install/flink-1.3.3/lib/slf4j-log4j12-1.7.7.jar:/home/centos/flink_install/flink-1.3.3/lib/flink-dist_2.11-1.3.3.jar::: 2018-12-19 19:50:06,872 INFO org.apache.flink.client.CliFrontend - -------------------------------------------------------------------------------- 2018-12-19 19:50:06,872 INFO org.apache.flink.client.CliFrontend - Using configuration directory /home/centos/flink_install/flink-1.3.3/conf 2018-12-19 19:50:06,872 INFO org.apache.flink.client.CliFrontend - Trying to load configuration file 2018-12-19 19:50:06,880 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, 9x.xx.xx.1 2018-12-19 19:50:06,881 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123 2018-12-19 19:50:06,881 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 8192m 2018-12-19 19:50:06,881 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 20480m 2018-12-19 19:50:06,882 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 10 2018-12-19 19:50:06,882 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1 2018-12-19 19:50:06,883 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.savepoints.dir, file:///mount/data/savepoints 2018-12-19 19:50:06,883 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 8081 2018-12-19 19:50:06,884 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.address, 9x.xx.xx.1 2018-12-19 19:50:07,248 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, 9x.xx.xx.1 2018-12-19 19:50:07,248 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123 2018-12-19 19:50:07,248 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 8192m 2018-12-19 19:50:07,249 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 20480m 2018-12-19 19:50:07,249 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 10 2018-12-19 19:50:07,249 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1 2018-12-19 19:50:07,250 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.savepoints.dir, file:///mount/data/savepoints 2018-12-19 19:50:07,250 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 8081 2018-12-19 19:50:07,250 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.address, 9x.xx.xx.1 2018-12-19 19:50:07,311 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to centos (auth:SIMPLE) 2018-12-19 19:50:07,368 INFO org.apache.flink.client.CliFrontend - Running 'savepoint' command. 2018-12-19 19:50:07,379 INFO org.apache.flink.client.CliFrontend - Retrieving JobManager. 2018-12-19 19:50:07,455 INFO org.apache.flink.client.CliFrontend - Using address /9x.xx.xx.1:6123 to connect to JobManager. 2018-12-19 19:50:07,455 INFO org.apache.flink.client.program.StandaloneClusterClient - Starting client actor system. 2018-12-19 19:50:07,457 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils - Trying to select the network interface and address to use by connecting to the leading JobManager. 2018-12-19 19:50:07,458 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils - TaskManager will try to connect for 10000 milliseconds before falling back to heuristics 2018-12-19 19:50:07,460 INFO org.apache.flink.runtime.net.ConnectionUtils - Retrieved new target address /9x.xx.xx.1:6123. 2018-12-19 19:50:08,047 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 2018-12-19 19:50:08,140 INFO Remoting - Starting remoting 2018-12-19 19:50:08,479 INFO Remoting - Remoting started; listening on addresses :[akka.tcp://flink@flink-master.local:49096] 2018-12-19 19:50:08,714 INFO org.apache.flink.client.CliFrontend - Triggering savepoint for job d3c377067c466d5fc040068b53f60931. 2018-12-19 19:50:08,717 INFO org.apache.flink.client.CliFrontend - Waiting for response... 2018-12-19 19:50:08,848 INFO org.apache.flink.client.CliFrontend - Savepoint completed. Path: /mnt/savepoints/savepoint-d3c377-805efc6c8142 2018-12-19 19:50:08,848 INFO org.apache.flink.client.CliFrontend - You can resume your program from this savepoint with the run command.

Metadata file content (working versions)

Ig-^@^@^@^B^@^@^@^@^@^@^@^A^@^@^@^@^@^@^@^Vtx%,<9f>^Oòa<90>'<8c>êQÈ^T¾^@^@^@^F^@^@^@<80>^@^@^@^A^@^@^@^F^@^@^@^@ÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Aÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Bÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Cÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Dÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Eÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@´ð$^Lgß\ÒÐ^RyÚΪ7±^@^@^@^F^@^@^@<80>^@^@^@^A^@^@^@^F^@^@^@^@ÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Aÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Bÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Cÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Dÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Eÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^SÑ<8a>^X K^]Z<82>^[<9a>'P?2^?^@^@^@^F^@^@^@<80>^@^@^@^A^@^@^@^F^@^@^@^@ÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Aÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Bÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Cÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Dÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Eÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@d:^_^X¯^VN^P)(^Ap^?^T^Ro^@^@^@^F^@^@^@<80>^@^@^@^A^@^@^@^F^@^@^@^@ÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Aÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Bÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Cÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Dÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Eÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^\^Z#<9b>ä/*Û<9b>q½qÿ?Bm^@^@^@^F^@^@^@<80>^@^@^@^A^@^@^@^F^@^@^@^@ÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Aÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Bÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Cÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Dÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^Eÿÿÿÿÿÿÿÿ^@^@^@^@^@^@^@^@^@^@^@^@^@^@Ð\ÞJ(^@^@^@^A^@^@^@^F^@^@^@^@ÿÿÿÿÿÿÿÿ^@^@^@^A^A^@$d880fde8-5dd9-4bcf-8086-7c74cfcb4088^@^@^AJ^A¬í^@^Esr^@^Qjava.util.HashMap^E^GÚÁÃ^VÑ^C^@^BF^@ loadFactorI^@ thresholdxp?@^@^@^@^@^@^Aw^H^@^@^@^B^@^@^@^Asr^@Iorg.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition ^E[/ØÜX5^B^@^CI^@ cachedHashI^@ partitionL^@^Etopict^@^RLjava/lang/String;xp>±]O^@^@^@^Bt^@^NWFXEvent-topicsr^@^Njava.lang.Long;<8b>ä<90>Ì<8f>#ß^B^@^AJ^@^Evaluexr^@^Pjava.lang.Number<86>¬<95>^]^K<94>à<8b>^B^@^@xp^@^@^@^@^@^@^Gýx^@^@^@^@^@^@^@^@^@^@^@^@^@^Aÿÿÿÿÿÿÿÿ^@^@^@^A^A^@$9c08c8d2-d2f7-459d-9656-02a601e11d56^@^@^AJ^A¬í^@^Esr^@^Qjava.util.HashMap^E^GÚÁÃ^VÑ^C^@^BF^@ loadFactorI^@ thresholdxp?@^@^@^@^@^@^Aw^H^@^@^@^B^@^@^@^Asr^@Iorg.apache.flink.streaming.connectors.kafka.internals.KafkaTopicPartition ^E[/ØÜX5^B^@^CI^@ cachedHashI^@