Cannot load saved model from S3 bucket after tensorflow-serving 2.7

[jeongukjae]

Bug Report

If this is a bug report, please fill out the following form in full:

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
• TensorFlow Serving installed from (source or binary): Docker image (tensorflow/serving:2.7.0)
• TensorFlow Serving version: 2.7.0

Describe the problem

Describe the problem clearly here. Be sure to convey here why it's a bug in
TensorFlow Serving.

SavedModel cannot be loaded from S3 bucket after tensorflow-serving 2.7. Server raised error like below:

2022-01-18 02:30:30.032712: E tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:365] FileSystemStoragePathSource encountered a filesystem access error: Could not find base path s3://some-s3-path..... for servable model with error UNIMPLEMENTED: File system scheme 's3' not implemented (file: 's3://some-s3-path.....')

Docker image tensorflow/serving:2.6.2 runs without any error in same configuration.

Exact Steps to Reproduce

Please include all steps necessary for someone to reproduce this issue on their
own machine. If not, skip this section.

docker run --rm -it \
  --env AWS_REGION=some-aws-region \
  --env MODEL_BASE_PATH=some-s3-path \
  tensorflow/serving:2.7.0

Source code / logs

Include any logs or source code that would be helpful to diagnose the problem.
If including tracebacks, please include the full traceback. Large logs and files
should be attached. Try to provide a reproducible test case that is the bare
minimum necessary to generate the problem.

2022-01-18 02:30:30.032712: E tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:365] FileSystemStoragePathSource encountered a filesystem access error: Could not find base path s3://some-s3-path..... for servable model with error UNIMPLEMENTED: File system scheme 's3' not implemented (file: 's3://some-s3-path.....')

---

I think this is because of Modular File System Migration in TF 2.7.0's release note. Is there any way to link tensorflow-io in the build step?

• https://github.com/tensorflow/io/blob/16d6f43e93bd2738b3011f73109708ecf318f195/tensorflow_io/core/filesystems/filesystem_plugins.cc

[siddharth-agrawal]

I asked a question on SO just a few days before this issue was opened: https://stackoverflow.com/questions/70700005/using-s3-bucket-for-savedmodel-with-tensorflow-serving2-7-0-gpu-docker-image. TF support asked me to follow up here, which is what I'm doing.

[jeongukjae]

Is there any progress here?

[mihaimaruseac]

Cloud filesystem implementation moved to https://github.com/tensorflow/io.

Users will have to install https://pypi.org/project/tensorflow-io/ and import it. This has the side effect of loading the plugin for the S3 filesystem so there will be an implementation

So, from 2.7 onwards, users need pip install tensorflow-io and import tensorflow_io and then code will work as from before.

For any 2.6 patch or previous releases there is nothing to change.

[jeongukjae]

@mihaimaruseac Thanks for your response :)

But I think that makes sense in python environment. To fix this issue, maybe tensorflow-io should be linked in bazel or c++ code level. For now, I downgrade tensorflow serving docker image version to load saved model from s3.

[richieyoum]

@mihaimaruseac are you referring to importing tfio in the training script? how will tf serving be impacted from tfio that way?

[haitong]

any updates? @yongtang You removed s3 support in tensorflow in this PR https://github.com/tensorflow/tensorflow/pull/51032/files Do you know how to build s3 support into tensorflow serving? tensorflow serving compiles against tensorflow source code. If tensorflow source code doesn't support s3 anymore, tfserving will fail to load models from s3 too

[hsahovic]

My team wanted to upgrade some of our production systems to tensorflow 2.8.X, but this is a blocking issue. @shan3290 any update on a way to have s3 support with tensorflow serving?

[lumenghe]

Is there any news ? we would like to use tensorflow 2.8 too with s3. thanks

[RakeshRaj97]

Any updates on this issue?

[521bibi]

docker run --rm -p 8501:8501 --name tfs-s3 -e AWS_ACCESS_KEY_ID=minioadmin -e AWS_SECRET_ACCESS_KEY=minioadmin -e S3_ENDPOINT=http://127.0.0.1:9000 -e AWS_REGION=us-east-1 --env MODEL_BASE_PATH=S3://models --env MODEL_NAME=half -t tensorflow/serving:latest
my error is：

The latest version is 2.5.1, right? Why is it also a tfio error? What can I do to use S3?