Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 15 additions & 5 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,18 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)

### [Added]

- Ingest Service
- Processor Service
- REST API
- Mock Device into Ingest Service
- Docker compose system architecture
- Enhanced documentation and open source project structure
- Comprehensive user guides and developer documentation
- GitHub issue and pull request templates
- Security policy and vulnerability reporting process
- Project governance and roadmap documentation
- Performance testing with Locust (September 2025)
- Improved README with badges, quick start, and community links
- Enhanced contributing guidelines with detailed development setup
- Updated project structure for better maintainability
- Clarified development status in documentation

### Security
- Added security policy with vulnerability reporting process
- Enhanced authentication and authorization documentation
- Improved security configuration guidelines
21 changes: 21 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2025 Dataspace Cloud Software

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
101 changes: 68 additions & 33 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,19 @@
# Dataspace Cloud Software

This repository contains code related to the `dataspace` project, which is a system for sensor data
ingestion from various endpoints and a REST API to expose this data.
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

A comprehensive sensor data ingestion platform that collects, processes, and exposes IoT sensor data
through a modern microservices architecture. Built for scalability, reliability, and ease of use.

## ✨ Features

- **🌐 Multi-Protocol Support** - HTTP and TTN (The Things Network) data ingestion
- **🔐 Secure Authentication** - JWT-based auth with role-based access control
- **📊 Real-time Processing** - Kafka-powered message queuing and processing
- **📈 Monitoring & Analytics** - Prometheus metrics and Grafana dashboards
- **🐳 Container Ready** - Full Docker Compose deployment
- **📱 Modern Frontend** - React-based web interface
- **🔧 Developer Friendly** - Comprehensive APIs and documentation

## Architecture

Expand Down Expand Up @@ -94,16 +106,31 @@ Legend:
└── typos.toml
```

## Pre-requisites
## 🚀 Quick Start

```bash
# Clone the repository
git clone https://github.com/IRNAS/dataspace-cloud-software.git
cd dataspace-cloud-software

# Start the system
make dev-up

# Access the application
open http://localhost:3001
```

**Default Login:**

- **Admin:** `admin` / `password123`
- **User:** `user` / `user123`

The following tools are required to set up and run the system:
## 📋 Prerequisites

- [Docker](https://www.docker.com/get-started) (with Docker Compose)
- [Make](https://www.gnu.org/software/make/) (usually pre-installed on Unix-like systems)
- [Python 3.10](https://www.python.org/downloads/release/python-3100/) (for local development of
services)
- [Node.js and npm](https://nodejs.org/en/download/) (for local development of the frontend
application)
- [Python 3.10](https://www.python.org/downloads/release/python-3100/) (for local development)
- [Node.js and npm](https://nodejs.org/en/download/) (for frontend development)

## Deployment

Expand Down Expand Up @@ -198,6 +225,26 @@ With the `nginx` reverse proxy, most of the ports are not available externally,
access them, a port mapping needs to be added in the `docker-compose-prod.yml` file - see the
`prometheus` service as an example.

## 📚 Documentation

- **[📖 Documentation Hub](docs/README.md)** - Complete documentation index

## 🤝 Contributing

We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details.

- **[Security Policy](SECURITY.md)** - Security reporting

## 💬 Community

- **[Issues](https://github.com/IRNAS/dataspace-cloud-software/issues)** - Bug reports and feature
requests
- **[Security](SECURITY.md)** - Report security vulnerabilities privately

## 📄 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

### Configuration and Constants

Before deploying the system, you need to configure certain environment variables and constants. This
Expand All @@ -207,10 +254,11 @@ The following constants need to be defined in these folders:

#### common/constants.py

The constants found in the common/constants.py file are used throughout the application for
consistent configuration.
The constants found in the common/constants.py file are already configured with default values for
development. See the actual file for the current configuration:

```python
# Example from common/constants.py
import logging

LOG_LEVEL = logging.INFO
Expand All @@ -223,14 +271,14 @@ if test:
HOST = "geospatial-data-test"

DB_CONFIG = {
"dbname": "", # Database name
"user": "", # Database user
"password": "", # Database password
"host": "", # Internal database host
"port": , # Internal port
"dbname": "dataspace",
"user": "dataspace-geospatial-user",
"password": "password", # Change in production
"host": HOST,
"port": 13328 if localhost else 5432,
}

INGEST_API_KEY = "your-secret-api-key" # Replace with your actual key
INGEST_API_KEY = "your-secret-api-key" # Change before deployment
```

Detailed descriptions of each constant:
Expand All @@ -254,31 +302,18 @@ Detailed descriptions of each constant:
deployment or `13328` for Docker Compose - this is defined in the `docker-compose` files -
`ports` mapping in the `geospatial-data` service.

#### ingest_service/constants.py
#### Service-specific constants

The constants found in the ingest_service/constants.py file are specific to the Ingest Service.
Both `ingest_service/constants.py` and `processor_service/constants.py` contain Kafka configuration:

```python
import os

LOCALHOST = os.environ.get("LOCALHOST") == "true"

KAFKA_HOST = "localhost:9992" if LOCALHOST else "kafka:9993" # Kafka broker to connect to and produce messages to
# kafka:9993 is the service name defined in the docker-compose files
KAFKA_HOST = "localhost:9992" if LOCALHOST else "kafka:9993"
```

#### processor_service/constants.py

The constants found in the processor_service/constants.py file are specific to the Processor
Service.

```python
import os

LOCALHOST = os.environ.get("LOCALHOST") == "true"
KAFKA_HOST = "localhost:9992" if LOCALHOST else "kafka:9993" # Kafka broker to connect to and consume messages from
# kafka:9993 is the service name defined in the docker-compose files
```
These files are already configured with appropriate defaults for the development environment.

## Local Development

Expand Down
85 changes: 85 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Security Policy

## Security Best Practices

### For Users

1. **Keep Updated:** Always use the latest version of the software
2. **Secure Configuration:** Follow our
[deployment security guidelines](docs/development/DEPLOYMENT.md#security)
3. **Monitor Access:** Regularly review user access and permissions
4. **API Keys:** Use strong, unique API keys and rotate them regularly
5. **Network Security:** Use HTTPS in production and secure your network

### For Developers

1. **Dependencies:** Keep all dependencies updated
2. **Code Review:** All security-related changes require thorough review
3. **Input Validation:** Validate all user inputs
4. **Authentication:** Implement proper authentication and authorization
5. **Secrets Management:** Never commit secrets to version control

## Security Features

### Authentication & Authorization

- **JWT Tokens:** Secure token-based authentication
- **Role-Based Access:** Admin and user role separation
- **Password Security:** bcrypt password hashing with salt
- **Token Blacklist:** Immediate token invalidation capability

### Data Protection

- **Input Validation:** Pydantic model validation for all inputs
- **SQL Injection Prevention:** Parameterized queries and ORM usage
- **CORS Protection:** Configurable cross-origin resource sharing
- **Rate Limiting:** Built-in rate limiting for API endpoints

### Infrastructure Security

- **Docker Security:** Minimal base images and security scanning
- **Network Isolation:** Docker network segmentation
- **Secrets Management:** Environment variable-based configuration
- **Monitoring:** Prometheus metrics and Grafana dashboards

## Security Configuration

### Environment Variables

Ensure these security-related environment variables are properly configured:

```bash
# Authentication
JWT_SECRET_KEY=your-very-secure-secret-key
JWT_ALGORITHM=HS256
JWT_ACCESS_TOKEN_EXPIRE_MINUTES=1440

# Database
POSTGRES_PASSWORD=strong-database-password
POSTGRES_USER=secure-username

# API Security
INGEST_API_KEY=your-secure-api-key

# CORS
CORS_ORIGINS=https://yourdomain.com,https://app.yourdomain.com
```

### Production Security Checklist

- [ ] Change all default passwords
- [ ] Use strong, unique API keys
- [ ] Enable HTTPS with valid certificates
- [ ] Configure proper CORS origins
- [ ] Set up firewall rules
- [ ] Set up monitoring and alerting
- [ ] Regular security updates
- [ ] Penetration testing

## Security Resources

- [OWASP Top 10](https://owasp.org/www-project-top-ten/)
- [Python Security Best Practices](https://python-security.readthedocs.io/)
- [FastAPI Security](https://fastapi.tiangolo.com/tutorial/security/)
- [Docker Security](https://docs.docker.com/engine/security/)
- [PostgreSQL Security](https://www.postgresql.org/docs/current/security.html)
Loading
Loading