详谈tensorflow gfile文件的用法
一、gfile模块是什么
gfile模块定义在tensorflow/python/platform/gfile.py，但其源代码实现主要位于tensorflow/tensorflow/python/lib/io/file_io.py，那么gfile模块主要功能是什么呢？
google上的定义为：
翻译过来为：
没有线程锁的文件I/O操作包装器
...对于TensorFlow的tf.gfile模块来说是一个特别无用的描述！
tf.gfile模块的主要角色是：
1.提供一个接近Python文件对象的API，以及
2.提供基于TensorFlowC++FileSystemAPI的实现。
C++FileSystemAPI支持多种文件系统实现，包括本地文件，谷歌云存储（以gs：//开头）和HDFS（以hdfs：/开头）。TensorFlow将它们导出为tf.gfile，以便我们可以使用这些实现来保存和加载检查点，编写TensorBoardlog以及访问训练数据（以及其他用途）。但是，如果所有文件都是本地文件，则可以使用常规的Python文件API而不会造成任何问题。
以上为google对tf.gfile的说明。
二、gfileAPI介绍
下面将分别介绍每一个gfileAPI！
2-1）tf.gfile.Copy(oldpath,newpath,overwrite=False)
拷贝源文件并创建目标文件，无返回，其形参说明如下：
oldpath：带路径名字的拷贝源文件；
newpath：带路径名字的拷贝目标文件；
overwrite：目标文件已经存在时是否要覆盖，默认为false，如果目标文件已经存在则会报错
2-2）tf.gfile.MkDir(dirname)
创建一个目录，dirname为目录名字，无返回。
2-3）tf.gfile.Remove(filename)
删除文件，filename即文件名，无返回。
2-4）tf.gfile.DeleteRecursively(dirname)
递归删除所有目录及其文件，dirname即目录名，无返回。
2-5）tf.gfile.Exists(filename)
判断目录或文件是否存在，filename可为目录路径或带文件名的路径，有该目录则返回True，否则False。
2-6）tf.gfile.Glob(filename)
查找匹配pattern的文件并以列表的形式返回，filename可以是一个具体的文件名，也可以是包含通配符的正则表达式。
2-7）tf.gfile.IsDirectory(dirname)
判断所给目录是否存在，如果存在则返回True，否则返回False，dirname是目录名。
2-8）tf.gfile.ListDirectory(dirname)
罗列dirname目录下的所有文件并以列表形式返回，dirname必须是目录名。
2-9）tf.gfile.MakeDirs(dirname)
以递归方式建立父目录及其子目录，如果目录已存在且是可覆盖则会创建成功，否则报错，无返回。
2-10）tf.gfile.Rename(oldname,newname,overwrite=False)
重命名或移动一个文件或目录，无返回，其形参说明如下：
oldname：旧目录或旧文件；
newname：新目录或新文件；
overwrite：默认为false，如果新目录或新文件已经存在则会报错，否则重命名或移动成功。
2-11）tf.gfile.Stat(filename)
返回目录的统计数据，该函数会返回FileStatistics数据结构，以dir(tf.gfile.Stat(filename))获取返回数据的属性如下：
2-12）tf.gfile.Walk(top,in_order=True)
递归获取目录信息生成器，top是目录名，in_order默认为True指示顺序遍历目录，否则将无序遍历，每次生成返回如下格式信息(dirname,[subdirname,subdirname,...],[filename,filename,...])。
2-13）tf.gfile.GFile(filename,mode)
获取文本操作句柄，类似于python提供的文本操作open()函数，filename是要打开的文件名，mode是以何种方式去读写，将会返回一个文本操作句柄。
tf.gfile.Open()是该接口的同名，可任意使用其中一个！
2-14）tf.gfile.FastGFile(filename,mode)
该函数与tf.gfile.GFile的差别仅仅在于“无阻塞”，即该函数会无阻赛以较快的方式获取文本操作句柄。
三、API源码
#Copyright2015TheTensorFlowAuthors.AllRightsReserved.
#
#LicensedundertheApacheLicense,Version2.0(the"License");
#youmaynotusethisfileexceptincompliancewiththeLicense.
#YoumayobtainacopyoftheLicenseat
#
#http://www.apache.org/licenses/LICENSE-2.0
#
#Unlessrequiredbyapplicablelaworagreedtoinwriting,software
#distributedundertheLicenseisdistributedonan"ASIS"BASIS,
#WITHOUTWARRANTIESORCONDITIONSOFANYKIND,eitherexpressorimplied.
#SeetheLicenseforthespecificlanguagegoverningpermissionsand
#limitationsundertheLicense.
#==============================================================================
"""FileIOmethodsthatwraptheC++FileSystemAPI.
TheC++FileSystemAPIisSWIGwrappedinfile_io.i.Thesefunctionscallthose
toaccomplishbasicFileIOoperations.
"""
from__future__importabsolute_import
from__future__importdivision
from__future__importprint_function
importos
importuuid
importsix
fromtensorflow.pythonimportpywrap_tensorflow
fromtensorflow.python.frameworkimportc_api_util
fromtensorflow.python.frameworkimporterrors
fromtensorflow.python.utilimportcompat
fromtensorflow.python.utilimportdeprecation
fromtensorflow.python.util.tf_exportimporttf_export
classFileIO(object):
"""FileIOclassthatexposesmethodstoread/writeto/fromfiles.
Theconstructortakesthefollowingarguments:
name:nameofthefile
mode:oneof'r','w','a','r+','w+','a+'.Append'b'forbytesmode.
Canbeusedasaniteratortoiterateoverlinesinthefile.
ThedefaultbuffersizeusedfortheBufferedInputStreamusedforreading
thefilelinebylineis1024*512bytes.
"""
def__init__(self,name,mode):
self.__name=name
self.__mode=mode
self._read_buf=None
self._writable_file=None
self._binary_mode="b"inmode
mode=mode.replace("b","")
ifmodenotin("r","w","a","r+","w+","a+"):
raiseerrors.InvalidArgumentError(
None,None,"modeisnot'r'or'w'or'a'or'r+'or'w+'or'a+'")
self._read_check_passed=modein("r","r+","a+","w+")
self._write_check_passed=modein("a","w","r+","a+","w+")
@property
defname(self):
"""Returnsthefilename."""
returnself.__name
@property
defmode(self):
"""Returnsthemodeinwhichthefilewasopened."""
returnself.__mode
def_preread_check(self):
ifnotself._read_buf:
ifnotself._read_check_passed:
raiseerrors.PermissionDeniedError(None,None,
"Fileisn'topenforreading")
witherrors.raise_exception_on_not_ok_status()asstatus:
self._read_buf=pywrap_tensorflow.CreateBufferedInputStream(
compat.as_bytes(self.__name),1024*512,status)
def_prewrite_check(self):
ifnotself._writable_file:
ifnotself._write_check_passed:
raiseerrors.PermissionDeniedError(None,None,
"Fileisn'topenforwriting")
witherrors.raise_exception_on_not_ok_status()asstatus:
self._writable_file=pywrap_tensorflow.CreateWritableFile(
compat.as_bytes(self.__name),compat.as_bytes(self.__mode),status)
def_prepare_value(self,val):
ifself._binary_mode:
returncompat.as_bytes(val)
else:
returncompat.as_str_any(val)
defsize(self):
"""Returnsthesizeofthefile."""
returnstat(self.__name).length
defwrite(self,file_content):
"""Writesfile_contenttothefile.Appendstotheendofthefile."""
self._prewrite_check()
witherrors.raise_exception_on_not_ok_status()asstatus:
pywrap_tensorflow.AppendToFile(
compat.as_bytes(file_content),self._writable_file,status)
defread(self,n=-1):
"""Returnsthecontentsofafileasastring.
Startsreadingfromcurrentpositioninfile.
Args:
n:Read'n'bytesifn!=-1.Ifn=-1,readstoendoffile.
Returns:
'n'bytesofthefile(orwholefile)inbytesmodeor'n'bytesofthe
stringifinstring(regular)mode.
"""
self._preread_check()
witherrors.raise_exception_on_not_ok_status()asstatus:
ifn==-1:
length=self.size()-self.tell()
else:
length=n
returnself._prepare_value(
pywrap_tensorflow.ReadFromStream(self._read_buf,length,status))
@deprecation.deprecated_args(
None,
"positionisdeprecatedinfavoroftheoffsetargument.",
"position")
defseek(self,offset=None,whence=0,position=None):
#TODO(jhseu):Deletelater.Usedtoomit`position`fromdocs.
#pylint:disable=g-doc-args
"""Seekstotheoffsetinthefile.
Args:
offset:Thebytecountrelativetothewhenceargument.
whence:Validvaluesforwhenceare:
0:startofthefile(default)
1:relativetothecurrentpositionofthefile
2:relativetotheendoffile.offsetisusuallynegative.
"""
#pylint:enable=g-doc-args
self._preread_check()
#Weneededtomakeoffsetakeywordargumentforbackwards-compatibility.
#Thischeckexistssothatwecanconvertbacktohavingoffsetbea
#positionalargument.
#TODO(jhseu):Make`offset`apositionalargumentafter`position`is
#deleted.
ifoffsetisNoneandpositionisNone:
raiseTypeError("seek():offsetargumentrequired")
ifoffsetisnotNoneandpositionisnotNone:
raiseTypeError("seek():offsetandpositionmaynotbeset"
"simultaneously.")
ifpositionisnotNone:
offset=position
witherrors.raise_exception_on_not_ok_status()asstatus:
ifwhence==0:
pass
elifwhence==1:
offset+=self.tell()
elifwhence==2:
offset+=self.size()
else:
raiseerrors.InvalidArgumentError(
None,None,
"Invalidwhenceargument:{}.Validvaluesare0,1,or2."
.format(whence))
ret_status=self._read_buf.Seek(offset)
pywrap_tensorflow.Set_TF_Status_from_Status(status,ret_status)
defreadline(self):
r"""Readsthenextlinefromthefile.Leavesthe'\n'attheend."""
self._preread_check()
returnself._prepare_value(self._read_buf.ReadLineAsString())
defreadlines(self):
"""Returnsalllinesfromthefileinalist."""
self._preread_check()
lines=[]
whileTrue:
s=self.readline()
ifnots:
break
lines.append(s)
returnlines
deftell(self):
"""Returnsthecurrentpositioninthefile."""
self._preread_check()
returnself._read_buf.Tell()
def__enter__(self):
"""Makeusablewith"with"statement."""
returnself
def__exit__(self,unused_type,unused_value,unused_traceback):
"""Makeusablewith"with"statement."""
self.close()
def__iter__(self):
returnself
defnext(self):
retval=self.readline()
ifnotretval:
raiseStopIteration()
returnretval
def__next__(self):
returnself.next()
defflush(self):
"""FlushestheWritablefile.
Thisonlyensuresthatthedatahasmadeitswayoutoftheprocesswithout
anyguaranteesonwhetherit'swrittentodisk.Thismeansthatthe
datawouldsurviveanapplicationcrashbutnotnecessarilyanOScrash.
"""
ifself._writable_file:
witherrors.raise_exception_on_not_ok_status()asstatus:
ret_status=self._writable_file.Flush()
pywrap_tensorflow.Set_TF_Status_from_Status(status,ret_status)
defclose(self):
"""ClosesFileIO.ShouldbecalledfortheWritableFiletobeflushed."""
self._read_buf=None
ifself._writable_file:
witherrors.raise_exception_on_not_ok_status()asstatus:
ret_status=self._writable_file.Close()
pywrap_tensorflow.Set_TF_Status_from_Status(status,ret_status)
self._writable_file=None
@tf_export("gfile.Exists")
deffile_exists(filename):
"""Determineswhetherapathexistsornot.
Args:
filename:string,apath
Returns:
Trueifthepathexists,whetheritsafileoradirectory.
Falseifthepathdoesnotexistandtherearenofilesystemerrors.
Raises:
errors.OpError:PropagatesanyerrorsreportedbytheFileSystemAPI.
"""
try:
witherrors.raise_exception_on_not_ok_status()asstatus:
pywrap_tensorflow.FileExists(compat.as_bytes(filename),status)
excepterrors.NotFoundError:
returnFalse
returnTrue
@tf_export("gfile.Remove")
defdelete_file(filename):
"""Deletesthefilelocatedat'filename'.
Args:
filename:string,afilename
Raises:
errors.OpError:PropagatesanyerrorsreportedbytheFileSystemAPI.E.g.,
NotFoundErrorifthefiledoesnotexist.
"""
witherrors.raise_exception_on_not_ok_status()asstatus:
pywrap_tensorflow.DeleteFile(compat.as_bytes(filename),status)
defread_file_to_string(filename,binary_mode=False):
"""Readstheentirecontentsofafiletoastring.
Args:
filename:string,pathtoafile
binary_mode:whethertoopenthefileinbinarymodeornot.Thischanges
thetypeoftheobjectreturned.
Returns:
contentsofthefileasastringorbytes.
Raises:
errors.OpError:Raisesvarietyoferrorsthataresubtypese.g.
NotFoundErroretc.
"""
ifbinary_mode:
f=FileIO(filename,mode="rb")
else:
f=FileIO(filename,mode="r")
returnf.read()
defwrite_string_to_file(filename,file_content):
"""Writesastringtoagivenfile.
Args:
filename:string,pathtoafile
file_content:string,contentsthatneedtobewrittentothefile
Raises:
errors.OpError:Ifthereareerrorsduringtheoperation.
"""
withFileIO(filename,mode="w")asf:
f.write(file_content)
@tf_export("gfile.Glob")
defget_matching_files(filename):
"""Returnsalistoffilesthatmatchthegivenpattern(s).
Args:
filename:stringoriterableofstrings.Theglobpattern(s).
Returns:
Alistofstringscontainingfilenamesthatmatchthegivenpattern(s).
Raises:
errors.OpError:Iftherearefilesystem/directorylistingerrors.
"""
witherrors.raise_exception_on_not_ok_status()asstatus:
ifisinstance(filename,six.string_types):
return[
#Convertthefilenamestostringfrombytes.
compat.as_str_any(matching_filename)
formatching_filenameinpywrap_tensorflow.GetMatchingFiles(
compat.as_bytes(filename),status)
]
else:
return[
#Convertthefilenamestostringfrombytes.
compat.as_str_any(matching_filename)
forsingle_filenameinfilename
formatching_filenameinpywrap_tensorflow.GetMatchingFiles(
compat.as_bytes(single_filename),status)
]
@tf_export("gfile.MkDir")
defcreate_dir(dirname):
"""Createsadirectorywiththename'dirname'.
Args:
dirname:string,nameofthedirectorytobecreated
Notes:
Theparentdirectoriesneedtoexist.Userecursive_create_dirinsteadif
thereisthepossibilitythattheparentdirsdon'texist.
Raises:
errors.OpError:Iftheoperationfails.
"""
witherrors.raise_exception_on_not_ok_status()asstatus:
pywrap_tensorflow.CreateDir(compat.as_bytes(dirname),status)
@tf_export("gfile.MakeDirs")
defrecursive_create_dir(dirname):
"""Createsadirectoryandallparent/intermediatedirectories.
Itsucceedsifdirnamealreadyexistsandiswritable.
Args:
dirname:string,nameofthedirectorytobecreated
Raises:
errors.OpError:Iftheoperationfails.
"""
witherrors.raise_exception_on_not_ok_status()asstatus:
pywrap_tensorflow.RecursivelyCreateDir(compat.as_bytes(dirname),status)
@tf_export("gfile.Copy")
defcopy(oldpath,newpath,overwrite=False):
"""Copiesdatafromoldpathtonewpath.
Args:
oldpath:string,nameofthefilewho'scontentsneedtobecopied
newpath:string,nameofthefiletowhichtocopyto
overwrite:boolean,iffalseitsanerrorfornewpathtobeoccupiedbyan
existingfile.
Raises:
errors.OpError:Iftheoperationfails.
"""
witherrors.raise_exception_on_not_ok_status()asstatus:
pywrap_tensorflow.CopyFile(
compat.as_bytes(oldpath),compat.as_bytes(newpath),overwrite,status)
@tf_export("gfile.Rename")
defrename(oldname,newname,overwrite=False):
"""Renameormoveafile/directory.
Args:
oldname:string,pathnameforafile
newname:string,pathnametowhichthefileneedstobemoved
overwrite:boolean,iffalseit'sanerrorfor`newname`tobeoccupiedby
anexistingfile.
Raises:
errors.OpError:Iftheoperationfails.
"""
witherrors.raise_exception_on_not_ok_status()asstatus:
pywrap_tensorflow.RenameFile(
compat.as_bytes(oldname),compat.as_bytes(newname),overwrite,status)
defatomic_write_string_to_file(filename,contents,overwrite=True):
"""Writesto`filename`atomically.
Thismeansthatwhen`filename`appearsinthefilesystem,itwillcontain
allof`contents`.Withwrite_string_to_file,itispossibleforthefile
toappearinthefilesystemwith`contents`onlypartiallywritten.
Accomplishedbywritingtoatempfileandthenrenamingit.
Args:
filename:string,pathnameforafile
contents:string,contentsthatneedtobewrittentothefile
overwrite:boolean,iffalseit'sanerrorfor`filename`tobeoccupiedby
anexistingfile.
"""
temp_pathname=filename+".tmp"+uuid.uuid4().hex
write_string_to_file(temp_pathname,contents)
try:
rename(temp_pathname,filename,overwrite)
excepterrors.OpError:
delete_file(temp_pathname)
raise
@tf_export("gfile.DeleteRecursively")
defdelete_recursively(dirname):
"""Deleteseverythingunderdirnamerecursively.
Args:
dirname:string,apathtoadirectory
Raises:
errors.OpError:Iftheoperationfails.
"""
witherrors.raise_exception_on_not_ok_status()asstatus:
pywrap_tensorflow.DeleteRecursively(compat.as_bytes(dirname),status)
@tf_export("gfile.IsDirectory")
defis_directory(dirname):
"""Returnswhetherthepathisadirectoryornot.
Args:
dirname:string,pathtoapotentialdirectory
Returns:
True,ifthepathisadirectory;Falseotherwise
"""
status=c_api_util.ScopedTFStatus()
returnpywrap_tensorflow.IsDirectory(compat.as_bytes(dirname),status)
@tf_export("gfile.ListDirectory")
deflist_directory(dirname):
"""Returnsalistofentriescontainedwithinadirectory.
Thelistisinarbitraryorder.Itdoesnotcontainthespecialentries"."
and"..".
Args:
dirname:string,pathtoadirectory
Returns:
[filename1,filename2,...filenameN]asstrings
Raises:
errors.NotFoundErrorifdirectorydoesn'texist
"""
ifnotis_directory(dirname):
raiseerrors.NotFoundError(None,None,"Couldnotfinddirectory")
witherrors.raise_exception_on_not_ok_status()asstatus:
#Converteachelementtostring,sincethereturnvaluesofthe
#vectorofstringshouldbeinterpretedasstrings,notbytes.
return[
compat.as_str_any(filename)
forfilenameinpywrap_tensorflow.GetChildren(
compat.as_bytes(dirname),status)
]
@tf_export("gfile.Walk")
defwalk(top,in_order=True):
"""Recursivedirectorytreegeneratorfordirectories.
Args:
top:string,aDirectoryname
in_order:bool,TraverseinorderifTrue,postorderifFalse.
Errorsthathappenwhilelistingdirectoriesareignored.
Yields:
Eachyieldisa3-tuple:thepathnameofadirectory,followedbylistsof
allitssubdirectoriesandleaffiles.
(dirname,[subdirname,subdirname,...],[filename,filename,...])
asstrings
"""
top=compat.as_str_any(top)
try:
listing=list_directory(top)
excepterrors.NotFoundError:
return
files=[]
subdirs=[]
foriteminlisting:
full_path=os.path.join(top,item)
ifis_directory(full_path):
subdirs.append(item)
else:
files.append(item)
here=(top,subdirs,files)
ifin_order:
yieldhere
forsubdirinsubdirs:
forsubiteminwalk(os.path.join(top,subdir),in_order):
yieldsubitem
ifnotin_order:
yieldhere
@tf_export("gfile.Stat")
defstat(filename):
"""Returnsfilestatisticsforagivenpath.
Args:
filename:string,pathtoafile
Returns:
FileStatisticsstructthatcontainsinformationaboutthepath
Raises:
errors.OpError:Iftheoperationfails.
"""
file_statistics=pywrap_tensorflow.FileStatistics()
witherrors.raise_exception_on_not_ok_status()asstatus:
pywrap_tensorflow.Stat(compat.as_bytes(filename),file_statistics,status)
returnfile_statistics
以上这篇详谈tensorflowgfile文件的用法就是小编分享给大家的全部内容了，希望能给大家一个参考，也希望大家多多支持毛票票。
声明：本文内容来源于网络，版权归原作者所有，内容由互联网用户自发贡献自行上传，本网站不拥有所有权，未作人工编辑处理，也不承担相关法律责任。如果您发现有涉嫌版权的内容，欢迎发送邮件至：czq8825#qq.com（发邮件时，请将#更换为@）进行举报，并提供相关证据，一经查实，本站将立刻删除涉嫌侵权内容。