Skip to content

feat: implement purge support in DropTable for InMemoryCatalog and SqlCatalog #744

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wgtmac merged 2 commits into from
Jun 17, 2026
Merged

feat: implement purge support in DropTable for InMemoryCatalog and SqlCatalog #744

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f424314
feat: implement purge support in DropTable for InMemoryCatalog and Sq…
WZhuo Jun 15, 2026
7b4bbb8
Use FileIO::DeleteFiles for batch metadata file deletion in InMemoryC…
WZhuo Jun 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion src/iceberg/catalog/memory/in_memory_catalog.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
#include <algorithm>
#include <iterator>

#include "iceberg/file_io.h"
#include "iceberg/table.h"
#include "iceberg/table_identifier.h"
#include "iceberg/table_metadata.h"
Expand Down Expand Up @@ -511,7 +512,22 @@ Result<bool> InMemoryCatalog::TableExists(const TableIdentifier& identifier) con

Status InMemoryCatalog::DropTable(const TableIdentifier& identifier, bool purge) {
std::unique_lock lock(mutex_);
// TODO(Guotao): Delete all metadata files if purge is true.
if (purge && file_io_) {
ICEBERG_ASSIGN_OR_RAISE(auto metadata_location,
root_namespace_->GetTableMetadataLocation(identifier));
ICEBERG_ASSIGN_OR_RAISE(auto metadata,
TableMetadataUtil::Read(*file_io_, metadata_location));
// Delete previous metadata files from the log first, so that if deletion
// fails and is retried, the current metadata file still exists as an
// anchor to locate any remaining old files.
std::vector<std::string> files_to_delete;
files_to_delete.reserve(metadata->metadata_log.size());
for (const auto& entry : metadata->metadata_log) {

@zhjwpku zhjwpku Jun 15, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a blocking comment, but should we use the newly introduced DeleteFiles API so that batch deletion can be leveraged when supported by the underlying FileIO implementation?

files_to_delete.push_back(entry.metadata_file);
}
std::ignore = file_io_->DeleteFiles(files_to_delete);
std::ignore = file_io_->DeleteFile(metadata_location);
}
return root_namespace_->UnregisterTable(identifier);
}

Expand Down
21 changes: 18 additions & 3 deletions src/iceberg/catalog/sql/sql_catalog.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
#include <string>

#include "iceberg/catalog/sql/config.h"
#include "iceberg/file_io.h"
#include "iceberg/table.h"
#include "iceberg/table_identifier.h"
#include "iceberg/table_metadata.h"
Expand Down Expand Up @@ -507,11 +508,25 @@ Result<std::shared_ptr<Transaction>> SqlCatalog::StageCreateTable(

Status SqlCatalog::DropTable(const TableIdentifier& identifier, bool purge) {
ICEBERG_RETURN_UNEXPECTED(ValidateTableIdentifier(identifier));
if (purge) {
// TODO(zhjwpku): Delete the table data and metadata files when purge is requested.
}

const std::string ns_str = NamespaceToString(identifier.ns);

if (purge && file_io_) {
ICEBERG_ASSIGN_OR_RAISE(auto metadata_location,
store_->GetTableMetadataLocation(ns_str, identifier.name));
if (metadata_location.has_value()) {
ICEBERG_ASSIGN_OR_RAISE(auto metadata,
TableMetadataUtil::Read(*file_io_, *metadata_location));
// Delete previous metadata files from the log first, so that if deletion
// fails and is retried, the current metadata file still exists as an
// anchor to locate any remaining old files.
for (const auto& entry : metadata->metadata_log) {
std::ignore = file_io_->DeleteFile(entry.metadata_file);
}
std::ignore = file_io_->DeleteFile(*metadata_location);
}
}

ICEBERG_ASSIGN_OR_RAISE(auto affected, store_->DeleteTable(ns_str, identifier.name));
if (affected == 0) {
return NoSuchTable("Table does not exist: {}", identifier.ToString());
Expand Down
Loading