Skip to content

GH-45747: [C++] Remove deprecated ObjectType and FileStatistics, refactor hdfs code #45998

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
AlenkaF wants to merge 50 commits into
base: main
Choose a base branch
from
Open

GH-45747: [C++] Remove deprecated ObjectType and FileStatistics, refactor hdfs code #45998

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
d9eac43
First remove only ObjectType
AlenkaF Mar 18, 2025
1090265
Remove FileStatistics
AlenkaF Apr 1, 2025
e5e1fb1
Run linter
AlenkaF Apr 1, 2025
63fa38b
Update tests and filesystem/hdfs.cc
AlenkaF Apr 1, 2025
49a3f8b
Move definitions from libarrow.pxd to libarrow_fs.pxd
AlenkaF Apr 1, 2025
6625ffa
Move hdfs code from io to filesystem
AlenkaF Apr 8, 2025
fc3a9fd
Fix a small mess after merge conflict resolve
AlenkaF Apr 8, 2025
434de96
Fix linter errors and revert changes in the test namespace
AlenkaF Apr 9, 2025
a3de460
Add docstrings to python have_libhdfs method
AlenkaF Apr 9, 2025
b4efd81
More linter fixes
AlenkaF Apr 9, 2025
354ca93
And Python linter
AlenkaF Apr 9, 2025
43bd8eb
Remove target_link_libraries for ARROW_IO
AlenkaF May 5, 2025
b827c04
Rename hdfs_io_test
AlenkaF May 5, 2025
19bb688
Remove hdfs_io include in filesystem/api.h
AlenkaF May 5, 2025
c1d8ec8
Merge internal hdfs classes
AlenkaF May 12, 2025
0e24444
Remove commented code
AlenkaF May 19, 2025
11152c6
Fix linter error in CMakeLists
AlenkaF May 19, 2025
54b101c
Add missing linter fix
AlenkaF May 19, 2025
208912c
Make MSVC happy
AlenkaF May 19, 2025
7662531
Change friend Status Make* methods to static Result Make
AlenkaF May 19, 2025
bd3dd8a
Try to remove the correct std::move
AlenkaF May 19, 2025
54b2c72
Define the static method on the outer class
AlenkaF May 20, 2025
b54ec99
Try to safeguard client_ from nullptr and dangling ptr
AlenkaF May 20, 2025
e94b75e
Update hdfs Impl
AlenkaF May 20, 2025
5a5f823
Change io::internal to fs::internal
AlenkaF Jun 10, 2025
f9de075
Run C++ linter
AlenkaF Jun 10, 2025
0b74f47
Remove HdfsDriver from PyArrow Cython file
AlenkaF Jun 10, 2025
95a952b
Run pre-commit
AlenkaF Jun 8, 2026
aa29648
Fix failing CI builds
AlenkaF Jun 8, 2026
9166f4e
Add <unordered_map>
AlenkaF Jun 8, 2026
8f00be3
Update HdfsConnectionConfig comment
AlenkaF Jun 8, 2026
3f361f5
Update meson.build
AlenkaF Jun 8, 2026
b94a306
Fix MakeDirectory
AlenkaF Jun 8, 2026
ebc320d
Update Copy
AlenkaF Jun 8, 2026
39f567c
Move internal hdfs methods back to public
AlenkaF Jun 11, 2026
98f9157
Update tests
AlenkaF Jun 11, 2026
7eed2a6
Run pre-hook
AlenkaF Jun 11, 2026
03fddb5
Remove internal tests
AlenkaF Jun 26, 2026
ba505d0
Add #undef for CHECK_FAILURE
AlenkaF Jun 26, 2026
95fa731
Add newline at the end of the file
AlenkaF Jun 26, 2026
f75a88c
Remove legacy methods
AlenkaF Jun 26, 2026
b0b9142
Remove HdfsConnectionConfig
AlenkaF Jul 1, 2026
0bd2bf6
Use internal::ConnectLibHdfs
AlenkaF Jul 1, 2026
ef5120d
Fix Python bindings
AlenkaF Jul 1, 2026
4ab7fb9
Remove merge conflict leftover
AlenkaF Jul 20, 2026
211d333
Potential fix for pull request finding
AlenkaF Jul 20, 2026
053dcf8
Add back 4 tests
AlenkaF Aug 14, 2026
0625e9e
Fix integration, dlopen linkage and mason issues
AlenkaF Sep 17, 2026
2741e3f
Remove another instance of io hdfs test in integration and clean up t…
AlenkaF Sep 17, 2026
8d861e4
Fix formatting of extra_conf method declaration
AlenkaF Sep 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions ci/scripts/integration_hdfs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,11 +46,9 @@ function use_libhdfs_dir() {
export ARROW_HDFS_TEST_LIBHDFS_REQUIRE=ON
pushd "${build_dir}"

debug/arrow-io-hdfs-test
debug/arrow-hdfs-test

use_libhdfs_dir
debug/arrow-io-hdfs-test
debug/arrow-hdfs-test
use_hadoop_home

Expand Down
13 changes: 4 additions & 9 deletions cpp/src/arrow/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -473,19 +473,11 @@ arrow_add_object_library(ARROW_IO
io/caching.cc
io/compressed.cc
io/file.cc
io/hdfs.cc
io/hdfs_internal.cc
io/interfaces.cc
io/memory.cc
io/slow.cc
io/stdio.cc
io/transform.cc)
foreach(ARROW_IO_TARGET ${ARROW_IO_TARGETS})
target_link_libraries(${ARROW_IO_TARGET} PRIVATE arrow::hadoop)
if(NOT MSVC)
target_link_libraries(${ARROW_IO_TARGET} PRIVATE ${CMAKE_DL_LIBS})
endif()
endforeach()

set(ARROW_MEMORY_POOL_SRCS memory_pool.cc)
if(ARROW_JEMALLOC)
Expand Down Expand Up @@ -1016,7 +1008,7 @@ if(ARROW_FILESYSTEM)
endif()
endif()
if(ARROW_HDFS)
list(APPEND ARROW_FILESYSTEM_SRCS filesystem/hdfs.cc)
list(APPEND ARROW_FILESYSTEM_SRCS filesystem/hdfs.cc filesystem/hdfs_internal.cc)
Comment thread
AlenkaF marked this conversation as resolved.
endif()
if(ARROW_S3)
list(APPEND ARROW_FILESYSTEM_SRCS filesystem/s3fs.cc)
Expand All @@ -1039,6 +1031,9 @@ if(ARROW_FILESYSTEM)
if(ARROW_HDFS)
foreach(ARROW_FILESYSTEM_TARGET ${ARROW_FILESYSTEM_TARGETS})
target_link_libraries(${ARROW_FILESYSTEM_TARGET} PRIVATE arrow::hadoop)
if(NOT MSVC)
target_link_libraries(${ARROW_FILESYSTEM_TARGET} PRIVATE ${CMAKE_DL_LIBS})
endif()
endforeach()
endif()
if(ARROW_S3)
Expand Down
Loading
Loading