Add Bird's-Eye-View rendering and tracker integration for ByteTrack#8
Open
Add Bird's-Eye-View rendering and tracker integration for ByteTrack#8
Conversation
Reviewer's GuideAdds a BirdEyeViewTracker/BirdEyeViewRenderer pipeline on top of BYTETracker to generate per-frame bird’s-eye-view BEV images using world ground-point coordinates and class info carried through Object and STrack, while slightly extending BYTETracker’s API for subclassing and inspection of tracked/lost tracks, and wiring this into the TensorRT C++ demo plus brief docs. Sequence diagram for per-frame BEV rendering via BirdEyeViewTrackersequenceDiagram
actor User
participant Main as main_bytetrack_cpp
participant Tracker as BirdEyeViewTracker
participant Base as BYTETracker
participant Renderer as BirdEyeViewRenderer
User->>Main: start demo()
Main->>Tracker: BirdEyeViewTracker(fps, track_buffer,
Main->>Tracker: "bev_outputs", y_min_m, y_max_m, ...)
loop For each video frame
Main->>Main: preprocess frame, build vector<Object> objects
Main->>Tracker: update(objects)
activate Tracker
Tracker->>Base: update(objects)
activate Base
Base->>Base: build detections from Object
Base->>Base: run association and Kalman updates
Base-->>Tracker: vector<STrack> output_tracks
deactivate Base
Tracker->>Base: get_tracked_stracks()
Tracker->>Base: get_lost_stracks()
Tracker->>Tracker: appendTracksToBevInput(tracked, Tracked, bev_tracks)
Tracker->>Tracker: appendTracksToBevInput(lost, Lost, bev_tracks)
Tracker->>Renderer: renderFrame(bev_tracks, bev_frame_index)
deactivate Tracker
Tracker-->>Main: vector<STrack> output_tracks
Main->>Main: draw 2D boxes from output_tracks
Main->>Main: write RGB frame if needed
end
Main-->>User: demo finished, BEV images in bev_outputs/
Flow diagram for BEV data propagation from detections to rendered imagesflowchart LR
A["Detector outputs
vector<Object>
rect, label, prob, gp"] --> B["BYTETracker::update
constructs STrack
with tlwh, score,
class_id, gp"]
B --> C["BYTETracker
tracking loop
updates tracked_stracks
and lost_stracks"]
C --> D["BirdEyeViewTracker::update
calls base update,
then collects:
get_tracked_stracks()
get_lost_stracks()"]
D --> E["appendTracksToBevInput
build vector<BEVTrackPoint>
from STrack.track_id,
class_id, gp, state"]
E --> F["BirdEyeViewRenderer::renderFrame
buildCanvas + drawGridAndAxes"]
F --> G["worldToImage(x, y)
map meters -> pixels"]
G --> H["drawTrackPoint
circle / cross,
id + class label"]
H --> I["cv::imwrite
bev_outputs/bev_XXXXXX.jpg"]
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 3 issues, and left some high level feedback:
- In
BirdEyeViewTrackerand the BEV structs you treattrack_stateas anintand passexpected_stateasint; consider using theTrackStateenum type directly throughout to improve type safety and readability. BirdEyeViewRenderer::ensureOutputDirectorysilently returnsfalseon failure andrenderFrameignores that result; consider surfacing a clear error (e.g., throw or log) so failures to create the output directory are easier to detect.BYTETracker::get_tracked_stracks/get_lost_stracksexpose internal storage by const reference; if there’s any risk of callers caching these references across updates, consider returning a copy or documenting that they must be used only within the same update cycle.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- In `BirdEyeViewTracker` and the BEV structs you treat `track_state` as an `int` and pass `expected_state` as `int`; consider using the `TrackState` enum type directly throughout to improve type safety and readability.
- `BirdEyeViewRenderer::ensureOutputDirectory` silently returns `false` on failure and `renderFrame` ignores that result; consider surfacing a clear error (e.g., throw or log) so failures to create the output directory are easier to detect.
- `BYTETracker::get_tracked_stracks` / `get_lost_stracks` expose internal storage by const reference; if there’s any risk of callers caching these references across updates, consider returning a copy or documenting that they must be used only within the same update cycle.
## Individual Comments
### Comment 1
<location path="deploy/TensorRT/cpp/include/BirdEyeViewTracker.h" line_range="24-25" />
<code_context>
+ BirdEyeViewRenderer& renderer();
+
+private:
+ void appendTracksToBevInput(const vector<STrack>& tracks,
+ int expected_state,
+ vector<BEVTrackPoint>& bev_tracks) const;
+
</code_context>
<issue_to_address>
**suggestion:** Use `TrackState` instead of `int` for `expected_state` to preserve type safety.
Because `STrack::state` is a `TrackState` enum, using `int` here weakens type safety and allows unrelated values. Taking `expected_state` as `TrackState` would clarify the API, prevent implicit conversions, and let `appendTracksToBevInput` compare directly against a `TrackState`.
Suggested implementation:
```c
BirdEyeViewRenderer& renderer();
private:
void appendTracksToBevInput(const vector<STrack>& tracks,
TrackState expected_state,
vector<BEVTrackPoint>& bev_tracks) const;
```
1. In the corresponding implementation file (likely `BirdEyeViewTracker.cpp`), update the definition of `appendTracksToBevInput` to use `TrackState expected_state` instead of `int expected_state`.
2. Ensure all call sites that pass `expected_state` are updated to pass a `TrackState` (e.g., `TrackState::Tracked`) rather than raw integers.
3. If `TrackState` is not already visible in the header, confirm it is brought in via the included headers (`BYTETracker.h` / `STrack` definition). If not, add the appropriate include for the `TrackState` enum.
</issue_to_address>
### Comment 2
<location path="deploy/TensorRT/cpp/include/BirdEyeViewRenderer.h" line_range="17" />
<code_context>
+ std::string class_label;
+ cv::Point3f gp = cv::Point3f(0.0F, 0.0F, 0.0F);
+ bool is_active = true;
+ int track_state = TrackState::Tracked;
+};
+
</code_context>
<issue_to_address>
**suggestion:** Consider storing `track_state` as `TrackState` instead of `int` in BEV structs.
`BEVTrackPoint` and `BEVRenderedTrackInfo` store `track_state` as `int` even though they represent `TrackState`. Using the `TrackState` enum directly would improve type safety and readability; only cast to `int` at serialization/logging boundaries if needed.
Suggested implementation:
```c
struct BEVTrackPoint {
int track_id = -1;
int class_id = -1;
std::string class_label;
cv::Point3f gp = cv::Point3f(0.0F, 0.0F, 0.0F);
bool is_active = true;
TrackState track_state = TrackState::Tracked;
};
```
```c
struct BEVRenderedTrackInfo {
int track_id = -1;
int class_id = -1;
std::string class_label;
cv::Point3f gp = cv::Point3f(0.0F, 0.0F, 0.0F);
bool is_active = true;
TrackState track_state = TrackState::Tracked;
};
```
1. If `BEVRenderedTrackInfo` or `BEVTrackPoint` are defined slightly differently in your local copy (e.g., different member order or missing default initializer), adjust the `SEARCH` blocks to match the exact existing definitions while still changing `int track_state` to `TrackState track_state`.
2. Any serialization, logging, or external interface code that expects `int track_state` should now explicitly cast `track_state` (e.g., `static_cast<int>(track_state)`) at the boundary as needed.
</issue_to_address>
### Comment 3
<location path="deploy/TensorRT/cpp/src/BirdEyeViewRenderer.cpp" line_range="56-57" />
<code_context>
+ drawTrackPoint(canvas, tracks[i], frame_index);
+ }
+
+ const std::string output_path = frameOutputPath(frame_index);
+ cv::imwrite(output_path, canvas);
+}
+
</code_context>
<issue_to_address>
**suggestion:** Handle or surface possible `cv::imwrite` failures when writing BEV frames.
`cv::imwrite` can fail (permissions, disk full, etc.) and is currently ignored. Since writing the BEV frame is a key side-effect, consider checking the return value and logging or otherwise surfacing failures instead of silently dropping outputs.
Suggested implementation:
```cpp
void BirdEyeViewRenderer::renderFrame(const std::vector<BEVTrackPoint>& tracks, int frame_index) {
cv::Mat canvas = buildCanvas();
latest_rendered_tracks_.clear();
latest_rendered_tracks_.reserve(tracks.size());
for (size_t i = 0; i < tracks.size(); ++i) {
drawTrackPoint(canvas, tracks[i], frame_index);
}
const std::string output_path = frameOutputPath(frame_index);
if (!cv::imwrite(output_path, canvas)) {
std::cerr << "BirdEyeViewRenderer: failed to write BEV frame to '" << output_path << "'" << std::endl;
}
#include <cerrno>
```
```cpp
#include <sstream>
#include <sys/stat.h>
#include <sys/types.h>
#include <algorithm>
#include <iostream>
```
If your codebase has a central logging facility (e.g., a `LOG(ERROR)` macro or similar), replace the `std::cerr` line with the appropriate logging call to keep logging consistent across the project.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| std::string class_label; | ||
| cv::Point3f gp = cv::Point3f(0.0F, 0.0F, 0.0F); | ||
| bool is_active = true; | ||
| int track_state = TrackState::Tracked; |
There was a problem hiding this comment.
suggestion: Consider storing track_state as TrackState instead of int in BEV structs.
BEVTrackPoint and BEVRenderedTrackInfo store track_state as int even though they represent TrackState. Using the TrackState enum directly would improve type safety and readability; only cast to int at serialization/logging boundaries if needed.
Suggested implementation:
struct BEVTrackPoint {
int track_id = -1;
int class_id = -1;
std::string class_label;
cv::Point3f gp = cv::Point3f(0.0F, 0.0F, 0.0F);
bool is_active = true;
TrackState track_state = TrackState::Tracked;
};struct BEVRenderedTrackInfo {
int track_id = -1;
int class_id = -1;
std::string class_label;
cv::Point3f gp = cv::Point3f(0.0F, 0.0F, 0.0F);
bool is_active = true;
TrackState track_state = TrackState::Tracked;
};- If
BEVRenderedTrackInfoorBEVTrackPointare defined slightly differently in your local copy (e.g., different member order or missing default initializer), adjust theSEARCHblocks to match the exact existing definitions while still changingint track_statetoTrackState track_state. - Any serialization, logging, or external interface code that expects
int track_stateshould now explicitly casttrack_state(e.g.,static_cast<int>(track_state)) at the boundary as needed.
Comment on lines
+56
to
+57
| const std::string output_path = frameOutputPath(frame_index); | ||
| cv::imwrite(output_path, canvas); |
There was a problem hiding this comment.
suggestion: Handle or surface possible cv::imwrite failures when writing BEV frames.
cv::imwrite can fail (permissions, disk full, etc.) and is currently ignored. Since writing the BEV frame is a key side-effect, consider checking the return value and logging or otherwise surfacing failures instead of silently dropping outputs.
Suggested implementation:
void BirdEyeViewRenderer::renderFrame(const std::vector<BEVTrackPoint>& tracks, int frame_index) {
cv::Mat canvas = buildCanvas();
latest_rendered_tracks_.clear();
latest_rendered_tracks_.reserve(tracks.size());
for (size_t i = 0; i < tracks.size(); ++i) {
drawTrackPoint(canvas, tracks[i], frame_index);
}
const std::string output_path = frameOutputPath(frame_index);
if (!cv::imwrite(output_path, canvas)) {
std::cerr << "BirdEyeViewRenderer: failed to write BEV frame to '" << output_path << "'" << std::endl;
}
#include <cerrno>
#include <sstream>
#include <sys/stat.h>
#include <sys/types.h>
#include <algorithm>
#include <iostream>
If your codebase has a central logging facility (e.g., a LOG(ERROR) macro or similar), replace the std::cerr line with the appropriate logging call to keep logging consistent across the project.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1d48d6af90
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+124
to
+125
| for (int x = static_cast<int>(std::ceil(world_range_.min_x)); x <= static_cast<int>(std::floor(world_range_.max_x)); | ||
| x += grid_step_m_) { |
There was a problem hiding this comment.
drawGridAndAxes increments both grid loops by grid_step_m_ without validating that it is positive. If a caller constructs BirdEyeViewRenderer/BirdEyeViewTracker with grid_step_m <= 0 (a valid value under the current API), the for loop never makes progress and renderFrame hangs the tracking pipeline on the first frame.
Useful? React with 👍 / 👎.
| const float usable_h = static_cast<float>(canvas_height_ - 2 * padding_px_); | ||
|
|
||
| const float nx = (x_m - world_range_.min_x) / x_span; | ||
| const float ny = (y_m - world_range_.min_y) / y_span; |
There was a problem hiding this comment.
worldToImage divides by y_span but the constructor allows y_min_m == y_max_m, which makes y_span zero and produces invalid coordinates (inf/NaN) during rendering. This can surface as corrupted BEV output or OpenCV drawing failures when users pass a degenerate y-range in configuration.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
gp) and class information through detections and tracks so BEV rendering can position and label tracks in world coordinates.Description
BirdEyeViewRenderer(deploy/TensorRT/cpp/include/BirdEyeViewRenderer.h,deploy/TensorRT/cpp/src/BirdEyeViewRenderer.cpp) which builds a BEV canvas, converts world coords to image pixels, draws grid/axes and track markers, and writes per-frame images.BirdEyeViewTracker(deploy/TensorRT/cpp/include/BirdEyeViewTracker.h,deploy/TensorRT/cpp/src/BirdEyeViewTracker.cpp) as aBYTETrackersubclass that calls the baseupdate, collects both tracked and lostSTrackentries, and passes them to theBirdEyeViewRenderer.Object(inBYTETracker.h) to includegp(cv::Point3f) and propagatelabel/gpintoSTrackvia a newSTrackconstructor signature; updatedSTrack(include/STrack.h,src/STrack.cpp) to storeclass_idandgpand to copy them inupdate/re_activate.BYTETracker::updatevirtual and destructor virtual, addedget_tracked_stracks()andget_lost_stracks()accessors inBYTETracker, and updated the TensorRT demo (deploy/TensorRT/cpp/src/bytetrack.cpp) to instantiateBirdEyeViewTrackerand write BEV outputs.deploy/TensorRT/cpp/BEV_INTEGRATION.mdthat shows usage and notes on passinggpfrom detectors.Testing
STrack.cpp,BYTETracker.cpp,BirdEyeViewRenderer.cpp, andBirdEyeViewTracker.cppsuccessfully.BirdEyeViewTrackerand produced per-frame BEV images under the configuredbev_outputsdirectory without runtime errors.Codex Task
Summary by Sourcery
Integrate a bird's-eye-view (BEV) visualization into the TensorRT C++ ByteTrack demo while propagating world coordinates and class metadata through tracking.
New Features:
Enhancements:
Documentation: