TFX Guide Local pipeline example

 

TFX guide에 있는 예시를 간단히 진행해보려 한다. 이를 통해서 TFX의 컴포넌트의 I/O를 간략히 살펴볼 수 있다.

 

https://www.tensorflow.org/tfx/guide/build_local_pipeline

로컬에서 TFX 파이프라인 구축  |  TensorFlow

해당 링크를 차근 차근 따라서 구성을 파악해보자.

 

tfx template copy --model='taxi' --pipeline_name='taxi_tutorial' --destination_path=./ # call taxi tutorial

tfx pipeline create --pipeline_path local_runner.py # create piepeline

tfx run create --pipeline_name taxi_tutorial # run pipeline

 

 

#components I/O (예시: taxi tutorial)

input	component	output
data.csv	CsvExampleGen	data_tfrecord-00000-of-00001.gz
CsvExampleGen.output	StatisticsGen	FeatureStats.pb
StatisticsGen.output	SchemaGen	schema.pbtxt
StatisticsGen.output SchemaGen.output	ExampleValidator	Validation results
CsvExampleGen.output SchemaGen.output preprocessing_fn	Transform
Transform.output SchemaGen.output args	Trainer	saved_model.pb events.out.tfevents.1651040385.dev-05.31488.0.v2
CsvExampleGen.output Trainer.output model_resolver.output	Evaluator
Trainer.output Evaluator.output	Pusher

data_tfrecord-00000-of-00001.gz file 파일 열기

import tensorflow as tf 
raw_dataset = tf.data.TFRecordDataset("path-to-file")

for raw_record in raw_dataset.take(1):
    example = tf.train.Example()
    example.ParseFromString(raw_record.numpy())
    print(example)
    
'''
features {
  feature {
    key: "company"
    value {
      bytes_list {
        value: "Chicago Elite Cab Corp. (Chicago Carriag"
      }
    }
  }
  feature {
    key: "dropoff_census_tract"
    value {
      int64_list {
      }
    }
  }
 '''

FeatureStats.pb 열기 (pb는 protobuf를 의미한다.)

: 미해결

 

schema.pbtxt 열기

'''
feature {
  name: "company"
  type: BYTES
  domain: "company"
  presence {
    min_fraction: 1.0
    min_count: 1
  }
}
feature {
  name: "payment_type"
  type: BYTES
  domain: "payment_type"
  presence {
    min_fraction: 1.0
    min_count: 1
  }
  shape {
    dim {
      size: 1
    }
  }
}
'''

Trainer 결과 열기:

tensorboard --logdir=./ --port=6009