Skip to main content
  1. Posts/

tensorflow checkpoint 学习笔记

·2 mins
Note: This article is available in Chinese only. 本文暂无英文版本。 View original

参考资料:

What is the TensorFlow checkpoint meta file?

TensorFlow: Restoring variables from from multiple checkpoints

合并模型的时候发现.meta一直在累加,而其他数据文件没有改变。因此来探究一下checkpoint的几个文件的含义。

This file contains a serialized MetaGraphDef protocol buffer. The MetaGraphDef is designed as a serialization format that includes all of the information required to restore a training or inference process (including the GraphDef that describes the dataflow, and additional annotations that describe the variables, input pipelines, and other relevant information). For example, the MetaGraphDef is used by TensorFlow Serving to start an inference service based on your trained model. We are investigating other tools that could use the MetaGraphDef for training.

Assuming that you still have the Python code for your model, you do not need the MetaGraphDefto restore the model, because you can reconstruct all of the information in the MetaGraphDef by re-executing the Python code that builds the model. To restore from a checkpoint, you only need the checkpoint files that contain the trained weights, which are written periodically to the same directory.

提到了.meta文件包含了恢复训练的所有信息。。。主要是为了其他工具可以快速恢复训练?

如果还是用python的话,恢复checkpoint没有必要使用.meta文件。

1  * **meta file**: describes the saved graph structure, includes GraphDef, SaverDef, and so on; then apply `tf.train.import_meta_graph('/tmp/model.ckpt.meta')`, will restore `Saver` and `Graph`.
2  * **index file**: it is a string-string immutable table(tensorflow::table::Table). Each key is a name of a tensor and its value is a serialized BundleEntryProto. Each BundleEntryProto describes the metadata of a tensor: which of the "data" files contains the content of a tensor, the offset into that file, checksum, some auxiliary data, etc.
3  * **data file**: it is TensorBundle collection, save the values of all variables.

Checkpoint
#

checkpoint文件可以理解成一个table,每次save之后会被更新,内容是最近一次的checkpoint文件名。

restore的时候,会先去checkpoint里看到最新的checkpoint文件是什么,然后只加载最新的。

Related

tensorflow variable 学习笔记

参考资料: programmers_guide/variables tf/Variable 之前感觉对tensorflow 的variable的理解不是很深刻…跑个模型啥的倒不会有什么问题,但是涉及分布式,模型并行之类的,感觉有些地方还是要理解得仔细一点比较好。

Distributed Tensorflow : Cannot assign a device for operation save

·1 min
是在使用分布式tensorflow遇到的一个错误 报错如下: InvalidArgumentError (see above for traceback): Cannot assign a device for operation ‘save/Rest│| 2 GeForce GTX 1080 On | 0000:08:00.0 Off | N/A | oreV2_888’: Operation was explicitly assigned to /job:worker/task:0/device:CPU:0 but available │| 24% 39C P8 12W / 180W | 0MiB / 8114MiB | 0% Default | devices are [ /job:localhost/replica:0/task:0/cpu:0, /job:localhost/replica:0/task:0/gpu:0 ]. Make sure the device specification refers to a valid device.

分布式 tensorflow 学习笔记(非最终版)

·4 mins
感觉资料不是很多,先收集资料好了。 tf-distributed官网文档 SO-between-graph和in-graph的区别 inception.README.md SyncReplicasOptimizer SO_How does ps work in distribute Tensorflow? update:在多个nodes(机)上跑。。。tf默认是异步更新的。。。同步的话。。大概需要syncreplicasoptimizer?