diff --git a/backend/plugins/gh-copilot/README.md b/backend/plugins/gh-copilot/README.md index 1c4aeda2481..4861ec48f66 100644 --- a/backend/plugins/gh-copilot/README.md +++ b/backend/plugins/gh-copilot/README.md @@ -18,7 +18,7 @@ limitations under the License. This plugin ingests GitHub Copilot **organization-level adoption metrics** (daily usage and seat assignments) and provides a Grafana dashboard for adoption trends. -It follows the same structure/patterns as other DevLake data-source plugins (notably `backend/plugins/q_dev`). +It follows the same structure/patterns as other DevLake data-source plugins (for example `backend/plugins/kiro`). ## What it collects diff --git a/backend/plugins/q_dev/Q_DEV_deploy_guide.md b/backend/plugins/q_dev/Q_DEV_deploy_guide.md deleted file mode 100644 index 5d97917546d..00000000000 --- a/backend/plugins/q_dev/Q_DEV_deploy_guide.md +++ /dev/null @@ -1,98 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - - -# DevLake Development Environment Deployment Guide - -## Environment Requirements -- Docker v19.03.10+ -- Golang v1.19+ -- GNU Make - - Mac (pre-installed) - - Windows: [Download](http://gnuwin32.sourceforge.net/packages/make.htm) - - Ubuntu: `sudo apt-get install build-essential libssl-dev` - -## How to Set Up the Development Environment -The following guide will explain how to run DevLake's frontend (config-ui) and backend in development mode. - -### Clone the Repository -Navigate to where you want to install this project and clone the repository: - -```bash -git clone https://github.com/apache/incubator-devlake.git -cd incubator-devlake -``` - -### Install Plugin Dependencies - -RefDiff plugin: -Install Go packages -```bash -cd backend -go get -cd .. -``` - -### Configure Environment File -Copy the example configuration file to a new local file: - -```bash -cp env.example .env -``` - -Update the following variables in the `.env` file: - -- `DB_URL`: Replace `mysql:3306` with `127.0.0.1:3306` -- `DISABLED_REMOTE_PLUGINS`: Set to `True` - -### Q Developer Plugin Configuration -The Q Developer plugin requires AWS credentials with access to both S3 and IAM Identity Center: - -**Required AWS Permissions:** -- S3: `s3:GetObject`, `s3:ListBucket` for the Q Developer data bucket -- Identity Center: `identitystore:DescribeUser` for user display name resolution - -**Required Configuration Fields:** -- AWS Access Key ID and Secret Access Key -- S3 bucket name and region -- IAM Identity Center Store ID (format: `d-xxxxxxxxxx`) -- IAM Identity Center region - -### Start MySQL and Grafana Containers - -Make sure the Docker daemon is running before this step. - -> Grafana needs to rebuild the image, then change the image in docker-compose.datasources.yml to `image: grafana:latest` - -```bash -docker-compose -f docker-compose-dev.yml up -d mysql grafana -``` - -### Run in Development Mode -Run devlake and config-ui in development mode in two separate terminals: - -```bash -# Install poetry, follow the guide: https://python-poetry.org/docs/#installation -# Run devlake, only using the q dev plugin here -DEVLAKE_PLUGINS=q_dev nohup make dev & -# Run config-ui -make configure-dev -``` - -For common errors, please refer to the troubleshooting documentation. - -Config UI runs on localhost:4000 \ No newline at end of file diff --git a/backend/plugins/q_dev/README.md b/backend/plugins/q_dev/README.md deleted file mode 100644 index 7d5cc4e7c33..00000000000 --- a/backend/plugins/q_dev/README.md +++ /dev/null @@ -1,106 +0,0 @@ - - -# Q Developer Plugin - -This plugin is used to retrieve AWS Q Developer usage data from AWS S3, process and analyze it, and resolve user display names through AWS IAM Identity Center. - -## Features - -- Retrieve CSV files from a specified prefix in AWS S3 -- Parse user usage data from CSV files -- Resolve user UUIDs to human-readable display names via AWS IAM Identity Center -- Aggregate data by user and calculate various metrics - -## Configuration - -Configuration items include: - -1. AWS Access Key ID -2. AWS Secret Key -3. AWS Region -4. S3 Bucket Name -5. Rate Limit (per hour) -6. IAM Identity Center Store ID -7. IAM Identity Center Region - -You can create a connection using the following curl command: -```bash -curl 'http://localhost:8080/plugins/q_dev/connections' \ ---header 'Content-Type: application/json' \ ---data-raw '{ - "name": "q_dev_connection", - "accessKeyId": "", - "secretAccessKey": "", - "region": "", - "bucket": "", - "identityStoreId": "", - "identityStoreRegion": "", - "rateLimitPerHour": 20000 -}' -``` -Please replace the following placeholders with actual values: -: Your AWS access key ID -: Your AWS secret access key -: The S3 bucket name you want to use -: The region where your S3 bucket is located -: Your IAM Identity Center Store ID (format: d-xxxxxxxxxx) -: The region where your Identity Center is deployed - -You can get all connections using the following curl command: -```bash -curl Get 'http://localhost:8080/plugins/q_dev/connections' -``` - -## Data Flow - -The plugin includes the following tasks: - -1. `collectQDevS3Files`: Collects file metadata information from S3, without downloading file content -2. `extractQDevS3Data`: Uses S3 file metadata to download CSV data, parse it into the database, and resolve user display names via Identity Center -3. `convertQDevUserMetrics`: Converts user data into aggregated metrics, calculating averages and totals - -## Data Tables - -- `_tool_q_dev_connections`: Stores AWS S3 connection information -- `_tool_q_dev_s3_file_meta`: Stores S3 file metadata -- `_tool_q_dev_user_data`: Stores user data parsed from CSV files -- `_tool_q_dev_user_metrics`: Stores aggregated user metrics - -Note: `_tool_q_dev_user_data` and `_tool_q_dev_user_metrics` tables now include `display_name` fields for human-readable user identification. - -## Data Collection Configuration -Steps to collect data: -1. On the Config UI page, select `Advanced Mode` on the left, click `Blueprints` -2. Create a new Blueprint -3. ![img.png](img.png) Click the gear icon on the right -4. Paste the following JSON configuration into `JSON Configuration`: - -```json -[ - [ - { - "plugin": "q_dev", - "subtasks": null, - "options": { - "connectionId": 5, - "s3Prefix": "" - } - } - ] -] -``` \ No newline at end of file diff --git a/backend/plugins/q_dev/api/blueprint_v200.go b/backend/plugins/q_dev/api/blueprint_v200.go deleted file mode 100644 index d7606a27564..00000000000 --- a/backend/plugins/q_dev/api/blueprint_v200.go +++ /dev/null @@ -1,100 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package api - -import ( - "github.com/apache/incubator-devlake/core/errors" - coreModels "github.com/apache/incubator-devlake/core/models" - "github.com/apache/incubator-devlake/core/plugin" - helper "github.com/apache/incubator-devlake/helpers/pluginhelper/api" - "github.com/apache/incubator-devlake/helpers/srvhelper" - "github.com/apache/incubator-devlake/plugins/q_dev/models" - "github.com/apache/incubator-devlake/plugins/q_dev/tasks" -) - -func MakeDataSourcePipelinePlanV200( - subtaskMetas []plugin.SubTaskMeta, - connectionId uint64, - bpScopes []*coreModels.BlueprintScope, -) (coreModels.PipelinePlan, []plugin.Scope, errors.Error) { - // load connection and scope from the db - connection, err := dsHelper.ConnSrv.FindByPk(connectionId) - if err != nil { - return nil, nil, err - } - scopeDetails, err := dsHelper.ScopeSrv.MapScopeDetails(connectionId, bpScopes) - if err != nil { - return nil, nil, err - } - - plan, err := makeDataSourcePipelinePlanV200(subtaskMetas, scopeDetails, connection) - if err != nil { - return nil, nil, err - } - scopes, err := makeScopesV200(scopeDetails, connection) - if err != nil { - return nil, nil, err - } - - return plan, scopes, nil -} - -func makeDataSourcePipelinePlanV200( - subtaskMetas []plugin.SubTaskMeta, - scopeDetails []*srvhelper.ScopeDetail[models.QDevS3Slice, srvhelper.NoScopeConfig], - connection *models.QDevConnection, -) (coreModels.PipelinePlan, errors.Error) { - plan := make(coreModels.PipelinePlan, len(scopeDetails)) - for i, scopeDetail := range scopeDetails { - s3Slice := scopeDetail.Scope - stage := plan[i] - if stage == nil { - stage = coreModels.PipelineStage{} - } - - // construct task options for q_dev - op := &tasks.QDevOptions{ - ConnectionId: s3Slice.ConnectionId, - S3Prefix: s3Slice.Prefix, - ScopeId: s3Slice.Id, - AccountId: s3Slice.AccountId, - BasePath: s3Slice.BasePath, - Year: s3Slice.Year, - Month: s3Slice.Month, - } - - // Pass empty entities array to enable all subtasks - task, err := helper.MakePipelinePlanTask("q_dev", subtaskMetas, []string{}, op) - if err != nil { - return nil, err - } - stage = append(stage, task) - plan[i] = stage - } - return plan, nil -} - -func makeScopesV200( - scopeDetails []*srvhelper.ScopeDetail[models.QDevS3Slice, srvhelper.NoScopeConfig], - connection *models.QDevConnection, -) ([]plugin.Scope, errors.Error) { - scopes := make([]plugin.Scope, 0) - // For Q Developer metrics, we don't need to create domain layer scopes - // The data is collected and stored directly in the tool layer - return scopes, nil -} diff --git a/backend/plugins/q_dev/api/connection.go b/backend/plugins/q_dev/api/connection.go deleted file mode 100644 index 1094d14efd6..00000000000 --- a/backend/plugins/q_dev/api/connection.go +++ /dev/null @@ -1,142 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package api - -import ( - "net/http" - - "github.com/apache/incubator-devlake/core/errors" - "github.com/apache/incubator-devlake/core/plugin" - "github.com/apache/incubator-devlake/helpers/pluginhelper/api" - "github.com/apache/incubator-devlake/plugins/q_dev/models" -) - -// 连接项目的CRUD API - -// PostConnections 创建新连接 (enhanced with Identity Store validation) -func PostConnections(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) { - // 创建连接 - connection := &models.QDevConnection{} - err := api.Decode(input.Body, connection, vld) - if err != nil { - return nil, err - } - - // 验证连接参数 (enhanced validation) - if err := validateConnection(connection); err != nil { - return nil, errors.BadInput.Wrap(err, "connection validation failed") - } - - // 保存到数据库 - err = connectionHelper.Create(connection, input) - if err != nil { - return nil, err - } - return &plugin.ApiResourceOutput{Body: connection.Sanitize(), Status: http.StatusOK}, nil -} - -// PatchConnection 更新现有连接 (enhanced with Identity Store validation) -func PatchConnection(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) { - connection := &models.QDevConnection{} - if err := connectionHelper.First(&connection, input.Params); err != nil { - return nil, err - } - if err := (&models.QDevConnection{}).MergeFromRequest(connection, input.Body); err != nil { - return nil, errors.Convert(err) - } - - // 验证更新后的连接参数 (enhanced validation) - if err := validateConnection(connection); err != nil { - return nil, errors.BadInput.Wrap(err, "connection validation failed") - } - - if err := connectionHelper.SaveWithCreateOrUpdate(connection); err != nil { - return nil, err - } - return &plugin.ApiResourceOutput{Body: connection.Sanitize(), Status: http.StatusOK}, nil -} - -// DeleteConnection 删除连接 -func DeleteConnection(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) { - conn := &models.QDevConnection{} - output, err := connectionHelper.Delete(conn, input) - if err != nil { - return output, err - } - output.Body = conn.Sanitize() - return output, nil -} - -// ListConnections 列出所有连接 -func ListConnections(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) { - var connections []models.QDevConnection - err := connectionHelper.List(&connections) - if err != nil { - return nil, err - } - // 敏感信息脱敏 - for i := 0; i < len(connections); i++ { - connections[i] = connections[i].Sanitize() - } - return &plugin.ApiResourceOutput{Body: connections}, nil -} - -// GetConnection 获取单个连接详情 -func GetConnection(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) { - connection := &models.QDevConnection{} - err := connectionHelper.First(connection, input.Params) - if err != nil { - return nil, err - } - return &plugin.ApiResourceOutput{Body: connection.Sanitize()}, err -} - -// validateConnection validates connection parameters including Identity Store fields -func validateConnection(connection *models.QDevConnection) error { - // Validate AWS credentials - if connection.AccessKeyId == "" { - return errors.Default.New("AccessKeyId is required") - } - if connection.SecretAccessKey == "" { - return errors.Default.New("SecretAccessKey is required") - } - if connection.Region == "" { - return errors.Default.New("Region is required") - } - if connection.Bucket == "" { - return errors.Default.New("Bucket is required") - } - - // Identity Store fields are optional, but must be provided together if used - if connection.IdentityStoreId == "" && connection.IdentityStoreRegion != "" { - return errors.Default.New("IdentityStoreRegion provided but IdentityStoreId is empty") - } - if connection.IdentityStoreId != "" && connection.IdentityStoreRegion == "" { - return errors.Default.New("IdentityStoreId provided but IdentityStoreRegion is empty") - } - - // Validate rate limit - if connection.RateLimitPerHour < 0 { - return errors.Default.New("RateLimitPerHour must be positive") - } - if connection.RateLimitPerHour == 0 { - connection.RateLimitPerHour = 20000 // Set default value - } - - return nil -} diff --git a/backend/plugins/q_dev/api/connection_test.go b/backend/plugins/q_dev/api/connection_test.go deleted file mode 100644 index 03a7e51cab3..00000000000 --- a/backend/plugins/q_dev/api/connection_test.go +++ /dev/null @@ -1,260 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package api - -import ( - "encoding/json" - "testing" - - "github.com/stretchr/testify/assert" - - "github.com/apache/incubator-devlake/plugins/q_dev/models" -) - -func TestValidateConnection_Success(t *testing.T) { - connection := &models.QDevConnection{ - QDevConn: models.QDevConn{ - AccessKeyId: "AKIAIOSFODNN7EXAMPLE", - SecretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", - Region: "us-east-1", - Bucket: "my-q-dev-bucket", - RateLimitPerHour: 20000, - IdentityStoreId: "d-1234567890", - IdentityStoreRegion: "us-west-2", - }, - } - - err := validateConnection(connection) - assert.NoError(t, err) -} - -func TestValidateConnection_MissingAccessKeyId(t *testing.T) { - connection := &models.QDevConnection{ - QDevConn: models.QDevConn{ - AccessKeyId: "", // Missing - SecretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", - Region: "us-east-1", - Bucket: "my-q-dev-bucket", - IdentityStoreId: "d-1234567890", - IdentityStoreRegion: "us-west-2", - }, - } - - err := validateConnection(connection) - assert.Error(t, err) - assert.Contains(t, err.Error(), "AccessKeyId is required") -} - -func TestValidateConnection_MissingSecretAccessKey(t *testing.T) { - connection := &models.QDevConnection{ - QDevConn: models.QDevConn{ - AccessKeyId: "AKIAIOSFODNN7EXAMPLE", - SecretAccessKey: "", // Missing - Region: "us-east-1", - Bucket: "my-q-dev-bucket", - IdentityStoreId: "d-1234567890", - IdentityStoreRegion: "us-west-2", - }, - } - - err := validateConnection(connection) - assert.Error(t, err) - assert.Contains(t, err.Error(), "SecretAccessKey is required") -} - -func TestValidateConnection_MissingRegion(t *testing.T) { - connection := &models.QDevConnection{ - QDevConn: models.QDevConn{ - AccessKeyId: "AKIAIOSFODNN7EXAMPLE", - SecretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", - Region: "", // Missing - Bucket: "my-q-dev-bucket", - IdentityStoreId: "d-1234567890", - IdentityStoreRegion: "us-west-2", - }, - } - - err := validateConnection(connection) - assert.Error(t, err) - assert.Contains(t, err.Error(), "Region is required") -} - -func TestValidateConnection_MissingBucket(t *testing.T) { - connection := &models.QDevConnection{ - QDevConn: models.QDevConn{ - AccessKeyId: "AKIAIOSFODNN7EXAMPLE", - SecretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", - Region: "us-east-1", - Bucket: "", // Missing - IdentityStoreId: "d-1234567890", - IdentityStoreRegion: "us-west-2", - }, - } - - err := validateConnection(connection) - assert.Error(t, err) - assert.Contains(t, err.Error(), "Bucket is required") -} - -func TestValidateConnection_EmptyIdentityStoreOk(t *testing.T) { - connection := &models.QDevConnection{ - QDevConn: models.QDevConn{ - AccessKeyId: "AKIAIOSFODNN7EXAMPLE", - SecretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", - Region: "us-east-1", - Bucket: "my-q-dev-bucket", - IdentityStoreId: "", - IdentityStoreRegion: "", - }, - } - - err := validateConnection(connection) - assert.NoError(t, err) -} - -func TestValidateConnection_IdentityStoreRegionWithoutId(t *testing.T) { - connection := &models.QDevConnection{ - QDevConn: models.QDevConn{ - AccessKeyId: "AKIAIOSFODNN7EXAMPLE", - SecretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", - Region: "us-east-1", - Bucket: "my-q-dev-bucket", - IdentityStoreId: "", - IdentityStoreRegion: "us-east-1", - }, - } - - err := validateConnection(connection) - assert.Error(t, err) - assert.Contains(t, err.Error(), "IdentityStoreRegion") -} - -func TestValidateConnection_IdentityStoreIdWithoutRegion(t *testing.T) { - connection := &models.QDevConnection{ - QDevConn: models.QDevConn{ - AccessKeyId: "AKIAIOSFODNN7EXAMPLE", - SecretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", - Region: "us-east-1", - Bucket: "my-q-dev-bucket", - IdentityStoreId: "d-1234567890", - IdentityStoreRegion: "", - }, - } - - err := validateConnection(connection) - assert.Error(t, err) - assert.Contains(t, err.Error(), "IdentityStoreId provided but IdentityStoreRegion is empty") -} - -func TestValidateConnection_InvalidRateLimit(t *testing.T) { - connection := &models.QDevConnection{ - QDevConn: models.QDevConn{ - AccessKeyId: "AKIAIOSFODNN7EXAMPLE", - SecretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", - Region: "us-east-1", - Bucket: "my-q-dev-bucket", - RateLimitPerHour: -1, // Invalid - IdentityStoreId: "d-1234567890", - IdentityStoreRegion: "us-west-2", - }, - } - - err := validateConnection(connection) - assert.Error(t, err) - assert.Contains(t, err.Error(), "RateLimitPerHour must be positive") -} - -func TestValidateConnection_DefaultRateLimit(t *testing.T) { - connection := &models.QDevConnection{ - QDevConn: models.QDevConn{ - AccessKeyId: "AKIAIOSFODNN7EXAMPLE", - SecretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", - Region: "us-east-1", - Bucket: "my-q-dev-bucket", - RateLimitPerHour: 0, // Should get default value - IdentityStoreId: "d-1234567890", - IdentityStoreRegion: "us-west-2", - }, - } - - err := validateConnection(connection) - assert.NoError(t, err) - assert.Equal(t, 20000, connection.RateLimitPerHour) // Should be set to default -} - -func TestConnectionRequestBody_Serialization(t *testing.T) { - // Test that the connection can be properly serialized/deserialized with new fields - original := &models.QDevConnection{ - QDevConn: models.QDevConn{ - AccessKeyId: "AKIAIOSFODNN7EXAMPLE", - SecretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", - Region: "us-east-1", - Bucket: "my-q-dev-bucket", - RateLimitPerHour: 20000, - IdentityStoreId: "d-1234567890", - IdentityStoreRegion: "us-west-2", - }, - } - - // Serialize to JSON - jsonData, err := json.Marshal(original) - assert.NoError(t, err) - - // Deserialize from JSON - var deserialized models.QDevConnection - err = json.Unmarshal(jsonData, &deserialized) - assert.NoError(t, err) - - // Verify all fields are preserved - assert.Equal(t, original.AccessKeyId, deserialized.AccessKeyId) - assert.Equal(t, original.SecretAccessKey, deserialized.SecretAccessKey) - assert.Equal(t, original.Region, deserialized.Region) - assert.Equal(t, original.Bucket, deserialized.Bucket) - assert.Equal(t, original.RateLimitPerHour, deserialized.RateLimitPerHour) - assert.Equal(t, original.IdentityStoreId, deserialized.IdentityStoreId) - assert.Equal(t, original.IdentityStoreRegion, deserialized.IdentityStoreRegion) -} - -func TestConnectionSanitization_PreservesIdentityStore(t *testing.T) { - connection := &models.QDevConnection{ - QDevConn: models.QDevConn{ - AccessKeyId: "AKIAIOSFODNN7EXAMPLE", - SecretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", - Region: "us-east-1", - Bucket: "my-q-dev-bucket", - RateLimitPerHour: 20000, - IdentityStoreId: "d-1234567890", - IdentityStoreRegion: "us-west-2", - }, - } - - sanitized := connection.Sanitize() - - // Secret should be sanitized - assert.NotEqual(t, "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", sanitized.SecretAccessKey) - - // Identity Store fields should be preserved - assert.Equal(t, "d-1234567890", sanitized.IdentityStoreId) - assert.Equal(t, "us-west-2", sanitized.IdentityStoreRegion) - - // Other fields should be preserved - assert.Equal(t, "AKIAIOSFODNN7EXAMPLE", sanitized.AccessKeyId) - assert.Equal(t, "us-east-1", sanitized.Region) - assert.Equal(t, "my-q-dev-bucket", sanitized.Bucket) - assert.Equal(t, 20000, sanitized.RateLimitPerHour) -} diff --git a/backend/plugins/q_dev/api/init.go b/backend/plugins/q_dev/api/init.go deleted file mode 100644 index 3bb67450b53..00000000000 --- a/backend/plugins/q_dev/api/init.go +++ /dev/null @@ -1,53 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package api - -import ( - "github.com/apache/incubator-devlake/core/context" - "github.com/apache/incubator-devlake/core/plugin" - "github.com/apache/incubator-devlake/helpers/pluginhelper/api" - "github.com/apache/incubator-devlake/helpers/srvhelper" - "github.com/apache/incubator-devlake/plugins/q_dev/models" - "github.com/go-playground/validator/v10" -) - -var vld *validator.Validate -var connectionHelper *api.ConnectionApiHelper -var basicRes context.BasicRes -var dsHelper *api.DsHelper[models.QDevConnection, models.QDevS3Slice, srvhelper.NoScopeConfig] - -func Init(br context.BasicRes, p plugin.PluginMeta) { - basicRes = br - vld = validator.New() - connectionHelper = api.NewConnectionHelper( - basicRes, - vld, - p.Name(), - ) - - dsHelper = api.NewDataSourceHelper[ - models.QDevConnection, models.QDevS3Slice, srvhelper.NoScopeConfig, - ]( - basicRes, - p.Name(), - []string{"prefix", "basePath", "name"}, - func(c models.QDevConnection) models.QDevConnection { return c.Sanitize() }, - func(s models.QDevS3Slice) models.QDevS3Slice { return s.Sanitize() }, - nil, - ) -} diff --git a/backend/plugins/q_dev/api/s3_slice_api.go b/backend/plugins/q_dev/api/s3_slice_api.go deleted file mode 100644 index 158aa4ef785..00000000000 --- a/backend/plugins/q_dev/api/s3_slice_api.go +++ /dev/null @@ -1,80 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package api - -import ( - "github.com/apache/incubator-devlake/core/errors" - "github.com/apache/incubator-devlake/core/plugin" - helper "github.com/apache/incubator-devlake/helpers/pluginhelper/api" - "github.com/apache/incubator-devlake/helpers/srvhelper" - "github.com/apache/incubator-devlake/plugins/q_dev/models" -) - -type PutScopesReqBody = helper.PutScopesReqBody[models.QDevS3Slice] -type ScopeDetail = srvhelper.ScopeDetail[models.QDevS3Slice, srvhelper.NoScopeConfig] - -// PutScopes create or update Q Developer scopes (S3 prefixes) -// @Summary create or update Q Developer scopes -// @Description Create or update Q Developer scopes -// @Tags plugins/q_dev -// @Accept application/json -// @Param connectionId path int true "connection ID" -// @Param scope body PutScopesReqBody true "json" -// @Success 200 {object} []models.QDevS3Slice -// @Failure 400 {object} shared.ApiBody "Bad Request" -// @Failure 500 {object} shared.ApiBody "Internal Error" -// @Router /plugins/q_dev/connections/{connectionId}/scopes [PUT] -func PutScopes(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) { - return dsHelper.ScopeApi.PutMultiple(input) -} - -// GetScopeList returns Q Developer scopes -// @Summary get Q Developer scopes -// @Description get Q Developer scopes -// @Tags plugins/q_dev -// @Param connectionId path int true "connection ID" -// @Param pageSize query int false "page size" -// @Param page query int false "page number" -// @Param blueprints query bool false "include blueprint references" -// @Success 200 {object} []ScopeDetail -// @Failure 400 {object} shared.ApiBody "Bad Request" -// @Failure 500 {object} shared.ApiBody "Internal Error" -// @Router /plugins/q_dev/connections/{connectionId}/scopes [GET] -func GetScopeList(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) { - return dsHelper.ScopeApi.GetPage(input) -} - -// GetScope returns a single scope record -func GetScope(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) { - return dsHelper.ScopeApi.GetScopeDetail(input) -} - -// PatchScope updates a scope record -func PatchScope(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) { - return dsHelper.ScopeApi.Patch(input) -} - -// DeleteScope removes a scope and optionally associated data. -func DeleteScope(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) { - return dsHelper.ScopeApi.Delete(input) -} - -// GetScopeLatestSyncState returns scope sync state info -func GetScopeLatestSyncState(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) { - return dsHelper.ScopeApi.GetScopeLatestSyncState(input) -} diff --git a/backend/plugins/q_dev/api/test_connection.go b/backend/plugins/q_dev/api/test_connection.go deleted file mode 100644 index b5cf35e2940..00000000000 --- a/backend/plugins/q_dev/api/test_connection.go +++ /dev/null @@ -1,67 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package api - -import ( - "github.com/apache/incubator-devlake/core/errors" - "github.com/apache/incubator-devlake/core/plugin" - "github.com/apache/incubator-devlake/helpers/pluginhelper/api" - "github.com/apache/incubator-devlake/plugins/q_dev/models" - "github.com/apache/incubator-devlake/plugins/q_dev/tasks" - - "net/http" -) - -// TestConnection 测试连接 -func TestConnection(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) { - // 解析连接参数 - var connection models.QDevConnection - err := api.Decode(input.Body, &connection, vld) - if err != nil { - return nil, err - } - - // 测试S3连接 - _, err = tasks.NewQDevS3Client(nil, &connection) - if err != nil { - return nil, err - } - - // 连接成功 - return &plugin.ApiResourceOutput{Status: http.StatusOK}, nil -} - -// TestExistingConnection 测试现有连接 -func TestExistingConnection(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) { - connection := &models.QDevConnection{} - err := connectionHelper.First(connection, input.Params) - if err != nil { - return nil, errors.BadInput.Wrap(err, "find connection from db") - } - if err := api.DecodeMapStruct(input.Body, connection, false); err != nil { - return nil, err - } - // 测试连接 - _, err = tasks.NewQDevS3Client(nil, connection) - if err != nil { - return nil, err - } - - // 连接成功 - return &plugin.ApiResourceOutput{Status: http.StatusOK}, nil -} diff --git a/backend/plugins/q_dev/img.png b/backend/plugins/q_dev/img.png deleted file mode 100644 index 16af8d1ad5c..00000000000 Binary files a/backend/plugins/q_dev/img.png and /dev/null differ diff --git a/backend/plugins/q_dev/impl/impl.go b/backend/plugins/q_dev/impl/impl.go deleted file mode 100644 index 3c6e1ed1644..00000000000 --- a/backend/plugins/q_dev/impl/impl.go +++ /dev/null @@ -1,216 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package impl - -import ( - "fmt" - - "github.com/apache/incubator-devlake/core/context" - "github.com/apache/incubator-devlake/core/dal" - "github.com/apache/incubator-devlake/core/errors" - coreModels "github.com/apache/incubator-devlake/core/models" - "github.com/apache/incubator-devlake/core/plugin" - helper "github.com/apache/incubator-devlake/helpers/pluginhelper/api" - "github.com/apache/incubator-devlake/plugins/q_dev/api" - "github.com/apache/incubator-devlake/plugins/q_dev/models" - "github.com/apache/incubator-devlake/plugins/q_dev/models/migrationscripts" - "github.com/apache/incubator-devlake/plugins/q_dev/tasks" -) - -var _ interface { - plugin.PluginMeta - plugin.PluginInit - plugin.PluginTask - plugin.PluginApi - plugin.PluginModel - plugin.PluginSource - plugin.PluginMigration - plugin.DataSourcePluginBlueprintV200 - plugin.CloseablePluginTask -} = (*QDev)(nil) - -type QDev struct{} - -func (p QDev) Init(basicRes context.BasicRes) errors.Error { - api.Init(basicRes, p) - return nil -} - -func (p QDev) GetTablesInfo() []dal.Tabler { - return []dal.Tabler{ - &models.QDevConnection{}, - &models.QDevUserData{}, - &models.QDevS3FileMeta{}, - &models.QDevS3Slice{}, - &models.QDevUserReport{}, - &models.QDevChatLog{}, - &models.QDevCompletionLog{}, - } -} - -func (p QDev) Description() string { - return "To collect and enrich data from AWS Q Developer usage metrics" -} - -func (p QDev) Name() string { - return "q_dev" -} - -func (p QDev) Connection() dal.Tabler { - return &models.QDevConnection{} -} - -func (p QDev) Scope() plugin.ToolLayerScope { - return &models.QDevS3Slice{} -} - -func (p QDev) ScopeConfig() dal.Tabler { - return nil -} - -func (p QDev) SubTaskMetas() []plugin.SubTaskMeta { - return []plugin.SubTaskMeta{ - tasks.CollectQDevS3FilesMeta, - tasks.ExtractQDevS3DataMeta, - tasks.ExtractQDevLoggingDataMeta, - } -} - -func (p QDev) PrepareTaskData(taskCtx plugin.TaskContext, options map[string]interface{}) (interface{}, errors.Error) { - var op tasks.QDevOptions - if err := helper.Decode(options, &op, nil); err != nil { - return nil, err - } - - connectionHelper := helper.NewConnectionHelper( - taskCtx, - nil, - p.Name(), - ) - connection := &models.QDevConnection{} - err := connectionHelper.FirstById(connection, op.ConnectionId) - if err != nil { - return nil, err - } - - // Create S3 client - s3Client, err := tasks.NewQDevS3Client(taskCtx, connection) - if err != nil { - return nil, err - } - - // Create Identity client (new) - identityClient, identityErr := tasks.NewQDevIdentityClient(connection) - if identityErr != nil { - taskCtx.GetLogger().Warn(identityErr, "Failed to create identity client, proceeding without user name resolution") - identityClient = nil - } - - // Resolve S3 prefixes to scan - var s3Prefixes []string - if op.AccountId != "" { - // New-style scope: construct both report paths using region from connection - region := connection.Region - timePart := fmt.Sprintf("%04d", op.Year) - if op.Month != nil { - timePart = fmt.Sprintf("%04d/%02d", op.Year, *op.Month) - } - // Kiro exports data to two well-known S3 prefixes: - // {basePath}/AWSLogs/{accountId}/KiroLogs/ — user report CSVs - // logging/AWSLogs/{accountId}/KiroLogs/ — interaction logs (JSON.gz) - // When basePath is empty, default to "user-report" for CSV data. - reportBase := op.BasePath - if reportBase == "" { - reportBase = "user-report" - } - csvBase := fmt.Sprintf("%s/AWSLogs/%s/KiroLogs", reportBase, op.AccountId) - logBase := fmt.Sprintf("logging/AWSLogs/%s/KiroLogs", op.AccountId) - s3Prefixes = []string{ - fmt.Sprintf("%s/by_user_analytic/%s/%s", csvBase, region, timePart), - fmt.Sprintf("%s/user_report/%s/%s", csvBase, region, timePart), - fmt.Sprintf("%s/GenerateAssistantResponse/%s/%s", logBase, region, timePart), - fmt.Sprintf("%s/GenerateCompletions/%s/%s", logBase, region, timePart), - } - } else { - // Legacy scope: use S3Prefix directly - s3Prefixes = []string{op.S3Prefix} - } - - return &tasks.QDevTaskData{ - Options: &op, - S3Client: s3Client, - IdentityClient: identityClient, - S3Prefixes: s3Prefixes, - }, nil -} - -func (p QDev) RootPkgPath() string { - return "github.com/apache/incubator-devlake/plugins/q_dev" -} - -func (p QDev) MigrationScripts() []plugin.MigrationScript { - return migrationscripts.All() -} - -func (p QDev) ApiResources() map[string]map[string]plugin.ApiResourceHandler { - return map[string]map[string]plugin.ApiResourceHandler{ - "test": { - "POST": api.TestConnection, - }, - "connections": { - "POST": api.PostConnections, - "GET": api.ListConnections, - }, - "connections/:connectionId": { - "PATCH": api.PatchConnection, - "DELETE": api.DeleteConnection, - "GET": api.GetConnection, - }, - "connections/:connectionId/test": { - "POST": api.TestExistingConnection, - }, - "connections/:connectionId/scopes": { - "GET": api.GetScopeList, - "PUT": api.PutScopes, - }, - "connections/:connectionId/scopes/:scopeId": { - "GET": api.GetScope, - "PATCH": api.PatchScope, - "DELETE": api.DeleteScope, - }, - "connections/:connectionId/scopes/:scopeId/latest-sync-state": { - "GET": api.GetScopeLatestSyncState, - }, - } -} - -func (p QDev) Close(taskCtx plugin.TaskContext) errors.Error { - data, ok := taskCtx.GetData().(*tasks.QDevTaskData) - if !ok { - return errors.Default.New(fmt.Sprintf("GetData failed when try to close %+v", taskCtx)) - } - data.S3Client.Close() - return nil -} - -func (p QDev) MakeDataSourcePipelinePlanV200( - connectionId uint64, - scopes []*coreModels.BlueprintScope, -) (coreModels.PipelinePlan, []plugin.Scope, errors.Error) { - return api.MakeDataSourcePipelinePlanV200(p.SubTaskMetas(), connectionId, scopes) -} diff --git a/backend/plugins/q_dev/impl/impl_test.go b/backend/plugins/q_dev/impl/impl_test.go deleted file mode 100644 index 7153617ab9a..00000000000 --- a/backend/plugins/q_dev/impl/impl_test.go +++ /dev/null @@ -1,125 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package impl - -import ( - "testing" - - "github.com/stretchr/testify/assert" - - "github.com/apache/incubator-devlake/plugins/q_dev/tasks" -) - -func TestQDev_BasicPluginMethods(t *testing.T) { - plugin := &QDev{} - - assert.Equal(t, "q_dev", plugin.Name()) - assert.Equal(t, "To collect and enrich data from AWS Q Developer usage metrics", plugin.Description()) - assert.Equal(t, "github.com/apache/incubator-devlake/plugins/q_dev", plugin.RootPkgPath()) - - // Test table info - tables := plugin.GetTablesInfo() - assert.Len(t, tables, 7) - - // Test subtask metas - subtasks := plugin.SubTaskMetas() - assert.Len(t, subtasks, 3) - - // Test API resources - apiResources := plugin.ApiResources() - assert.NotEmpty(t, apiResources) - assert.Contains(t, apiResources, "test") - assert.Contains(t, apiResources, "connections") -} - -func TestQDev_TaskDataStructure(t *testing.T) { - // Test that QDevTaskData has the expected structure (legacy mode) - taskData := &tasks.QDevTaskData{ - Options: &tasks.QDevOptions{ - ConnectionId: 1, - S3Prefix: "test/", - }, - S3Client: &tasks.QDevS3Client{ - Bucket: "test-bucket", - }, - IdentityClient: &tasks.QDevIdentityClient{ - StoreId: "d-1234567890", - Region: "us-west-2", - }, - S3Prefixes: []string{"test/"}, - } - - assert.NotNil(t, taskData.Options) - assert.NotNil(t, taskData.S3Client) - assert.NotNil(t, taskData.IdentityClient) - - assert.Equal(t, uint64(1), taskData.Options.ConnectionId) - assert.Equal(t, "test/", taskData.Options.S3Prefix) - assert.Equal(t, "test-bucket", taskData.S3Client.Bucket) - assert.Equal(t, "d-1234567890", taskData.IdentityClient.StoreId) - assert.Equal(t, "us-west-2", taskData.IdentityClient.Region) - assert.Equal(t, []string{"test/"}, taskData.S3Prefixes) -} - -func TestQDev_TaskDataWithAccountId(t *testing.T) { - // Test new-style scope with AccountId and multiple S3Prefixes - month := 1 - taskData := &tasks.QDevTaskData{ - Options: &tasks.QDevOptions{ - ConnectionId: 1, - AccountId: "034362076319", - BasePath: "user-report", - Year: 2026, - Month: &month, - }, - S3Client: &tasks.QDevS3Client{ - Bucket: "test-bucket", - }, - S3Prefixes: []string{ - "user-report/AWSLogs/034362076319/KiroLogs/by_user_analytic/us-east-1/2026/01", - "user-report/AWSLogs/034362076319/KiroLogs/user_report/us-east-1/2026/01", - }, - } - - assert.Equal(t, "034362076319", taskData.Options.AccountId) - assert.Equal(t, "user-report", taskData.Options.BasePath) - assert.Equal(t, 2026, taskData.Options.Year) - assert.Equal(t, &month, taskData.Options.Month) - assert.Len(t, taskData.S3Prefixes, 2) - assert.Contains(t, taskData.S3Prefixes[0], "by_user_analytic") - assert.Contains(t, taskData.S3Prefixes[1], "user_report") -} - -func TestQDev_TaskDataWithoutIdentityClient(t *testing.T) { - // Test that QDevTaskData works without IdentityClient - taskData := &tasks.QDevTaskData{ - Options: &tasks.QDevOptions{ - ConnectionId: 1, - }, - S3Client: &tasks.QDevS3Client{ - Bucket: "test-bucket", - }, - IdentityClient: nil, - S3Prefixes: []string{"some-prefix/"}, - } - - assert.NotNil(t, taskData.Options) - assert.NotNil(t, taskData.S3Client) - assert.Nil(t, taskData.IdentityClient) - assert.Len(t, taskData.S3Prefixes, 1) -} diff --git a/backend/plugins/q_dev/models/chat_log.go b/backend/plugins/q_dev/models/chat_log.go deleted file mode 100644 index 6b39bffa4de..00000000000 --- a/backend/plugins/q_dev/models/chat_log.go +++ /dev/null @@ -1,54 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package models - -import ( - "time" - - "github.com/apache/incubator-devlake/core/models/common" -) - -// QDevChatLog stores parsed data from GenerateAssistantResponse logging events -type QDevChatLog struct { - common.NoPKModel - ConnectionId uint64 `gorm:"primaryKey"` - ScopeId string `gorm:"primaryKey;type:varchar(255)" json:"scopeId"` - RequestId string `gorm:"primaryKey;type:varchar(255)" json:"requestId"` - UserId string `gorm:"index;type:varchar(255)" json:"userId"` - DisplayName string `gorm:"type:varchar(255)" json:"displayName"` - Timestamp time.Time `gorm:"index" json:"timestamp"` - ChatTriggerType string `gorm:"type:varchar(50)" json:"chatTriggerType"` - HasCustomization bool `json:"hasCustomization"` - ConversationId string `gorm:"type:varchar(255)" json:"conversationId"` - UtteranceId string `gorm:"type:varchar(255)" json:"utteranceId"` - ModelId string `gorm:"type:varchar(100)" json:"modelId"` - PromptLength int `json:"promptLength"` - ResponseLength int `json:"responseLength"` - OpenFileCount int `json:"openFileCount"` - ActiveFileName string `gorm:"type:varchar(512)" json:"activeFileName"` - ActiveFileExtension string `gorm:"type:varchar(50)" json:"activeFileExtension"` - HasSteering bool `json:"hasSteering"` - IsSpecMode bool `json:"isSpecMode"` - CodeReferenceCount int `json:"codeReferenceCount"` - WebLinkCount int `json:"webLinkCount"` - HasFollowupPrompts bool `json:"hasFollowupPrompts"` -} - -func (QDevChatLog) TableName() string { - return "_tool_q_dev_chat_log" -} diff --git a/backend/plugins/q_dev/models/completion_log.go b/backend/plugins/q_dev/models/completion_log.go deleted file mode 100644 index 00a1b471f13..00000000000 --- a/backend/plugins/q_dev/models/completion_log.go +++ /dev/null @@ -1,45 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package models - -import ( - "time" - - "github.com/apache/incubator-devlake/core/models/common" -) - -// QDevCompletionLog stores parsed data from GenerateCompletions logging events -type QDevCompletionLog struct { - common.NoPKModel - ConnectionId uint64 `gorm:"primaryKey"` - ScopeId string `gorm:"primaryKey;type:varchar(255)" json:"scopeId"` - RequestId string `gorm:"primaryKey;type:varchar(255)" json:"requestId"` - UserId string `gorm:"index;type:varchar(255)" json:"userId"` - DisplayName string `gorm:"type:varchar(255)" json:"displayName"` - Timestamp time.Time `gorm:"index" json:"timestamp"` - FileName string `gorm:"type:varchar(512)" json:"fileName"` - FileExtension string `gorm:"type:varchar(50)" json:"fileExtension"` - HasCustomization bool `json:"hasCustomization"` - CompletionsCount int `json:"completionsCount"` - LeftContextLength int `json:"leftContextLength"` - RightContextLength int `json:"rightContextLength"` -} - -func (QDevCompletionLog) TableName() string { - return "_tool_q_dev_completion_log" -} diff --git a/backend/plugins/q_dev/models/connection.go b/backend/plugins/q_dev/models/connection.go deleted file mode 100644 index 953e8dad756..00000000000 --- a/backend/plugins/q_dev/models/connection.go +++ /dev/null @@ -1,75 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package models - -import ( - "github.com/apache/incubator-devlake/core/utils" - helper "github.com/apache/incubator-devlake/helpers/pluginhelper/api" -) - -// QDevConn holds the essential information to connect to AWS S3 -type QDevConn struct { - // AccessKeyId for AWS - AccessKeyId string `mapstructure:"accessKeyId" json:"accessKeyId"` - // SecretAccessKey for AWS - SecretAccessKey string `mapstructure:"secretAccessKey" json:"secretAccessKey"` - // Region for AWS S3 - Region string `mapstructure:"region" json:"region"` - // Bucket for AWS S3 - Bucket string `mapstructure:"bucket" json:"bucket"` - // RateLimitPerHour limits the API requests sent to AWS - RateLimitPerHour int `mapstructure:"rateLimitPerHour" json:"rateLimitPerHour"` - - // Required fields for IAM Identity Center - // IdentityStoreId for AWS IAM Identity Center (required for user display names) - IdentityStoreId string `mapstructure:"identityStoreId" json:"identityStoreId"` - // IdentityStoreRegion for AWS IAM Identity Center (required, may differ from S3 region) - IdentityStoreRegion string `mapstructure:"identityStoreRegion" json:"identityStoreRegion"` -} - -func (conn *QDevConn) Sanitize() QDevConn { - conn.SecretAccessKey = utils.SanitizeString(conn.SecretAccessKey) - return *conn -} - -// QDevConnection holds QDevConn plus ID/Name for database storage -type QDevConnection struct { - helper.BaseConnection `mapstructure:",squash"` - QDevConn `mapstructure:",squash"` -} - -func (QDevConnection) TableName() string { - return "_tool_q_dev_connections" -} - -func (connection QDevConnection) Sanitize() QDevConnection { - connection.QDevConn = connection.QDevConn.Sanitize() - return connection -} - -func (connection *QDevConnection) MergeFromRequest(target *QDevConnection, body map[string]interface{}) error { - secretKey := target.SecretAccessKey - if err := helper.DecodeMapStruct(body, target, true); err != nil { - return err - } - modifiedSecretKey := target.SecretAccessKey - if modifiedSecretKey == "" || modifiedSecretKey == utils.SanitizeString(secretKey) { - target.SecretAccessKey = secretKey - } - return nil -} diff --git a/backend/plugins/q_dev/models/connection_test.go b/backend/plugins/q_dev/models/connection_test.go deleted file mode 100644 index 480564813df..00000000000 --- a/backend/plugins/q_dev/models/connection_test.go +++ /dev/null @@ -1,111 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package models - -import ( - "testing" - - "github.com/stretchr/testify/assert" -) - -func TestQDevConn_WithIdentityStore(t *testing.T) { - conn := QDevConn{ - AccessKeyId: "test-key", - SecretAccessKey: "test-secret", - Region: "us-east-1", - Bucket: "test-bucket", - RateLimitPerHour: 20000, - IdentityStoreId: "d-1234567890", - IdentityStoreRegion: "us-west-2", - } - - assert.Equal(t, "d-1234567890", conn.IdentityStoreId) - assert.Equal(t, "us-west-2", conn.IdentityStoreRegion) - assert.Equal(t, "us-east-1", conn.Region) // S3 region -} - -func TestQDevConn_RequiredFields(t *testing.T) { - // Test that all required fields are present - conn := QDevConn{ - AccessKeyId: "test-key", - SecretAccessKey: "test-secret", - Region: "us-east-1", - Bucket: "test-bucket", - RateLimitPerHour: 20000, - IdentityStoreId: "d-1234567890", - IdentityStoreRegion: "us-west-2", - } - - // All required fields should be non-empty - assert.NotEmpty(t, conn.AccessKeyId) - assert.NotEmpty(t, conn.SecretAccessKey) - assert.NotEmpty(t, conn.Region) - assert.NotEmpty(t, conn.Bucket) - assert.NotEmpty(t, conn.IdentityStoreId) - assert.NotEmpty(t, conn.IdentityStoreRegion) - assert.Greater(t, conn.RateLimitPerHour, 0) -} - -func TestQDevConn_Sanitize_PreservesIdentityStore(t *testing.T) { - conn := QDevConn{ - SecretAccessKey: "secret-key", - IdentityStoreId: "d-1234567890", - IdentityStoreRegion: "us-west-2", - } - - sanitized := conn.Sanitize() - assert.NotEqual(t, "secret-key", sanitized.SecretAccessKey) - assert.Equal(t, "d-1234567890", sanitized.IdentityStoreId) - assert.Equal(t, "us-west-2", sanitized.IdentityStoreRegion) -} - -func TestQDevConnection_WithIdentityStore(t *testing.T) { - connection := QDevConnection{ - QDevConn: QDevConn{ - AccessKeyId: "test-key", - SecretAccessKey: "test-secret", - Region: "us-east-1", - Bucket: "test-bucket", - IdentityStoreId: "d-1234567890", - IdentityStoreRegion: "us-west-2", - }, - } - - assert.Equal(t, "d-1234567890", connection.IdentityStoreId) - assert.Equal(t, "us-west-2", connection.IdentityStoreRegion) -} - -func TestQDevConnection_Sanitize_WithIdentityStore(t *testing.T) { - connection := QDevConnection{ - QDevConn: QDevConn{ - SecretAccessKey: "secret-key", - IdentityStoreId: "d-1234567890", - IdentityStoreRegion: "us-west-2", - }, - } - - sanitized := connection.Sanitize() - assert.NotEqual(t, "secret-key", sanitized.SecretAccessKey) - assert.Equal(t, "d-1234567890", sanitized.IdentityStoreId) - assert.Equal(t, "us-west-2", sanitized.IdentityStoreRegion) -} - -func TestQDevConnection_TableName(t *testing.T) { - connection := QDevConnection{} - assert.Equal(t, "_tool_q_dev_connections", connection.TableName()) -} diff --git a/backend/plugins/q_dev/models/migrationscripts/20250319_init.go b/backend/plugins/q_dev/models/migrationscripts/20250319_init.go deleted file mode 100644 index f73c50a9052..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/20250319_init.go +++ /dev/null @@ -1,45 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package migrationscripts - -import ( - "github.com/apache/incubator-devlake/core/context" - "github.com/apache/incubator-devlake/core/errors" - "github.com/apache/incubator-devlake/helpers/migrationhelper" - "github.com/apache/incubator-devlake/plugins/q_dev/models/migrationscripts/archived" -) - -type initTables struct{} - -func (*initTables) Name() string { - return "Init schema for Q Developer plugin" -} - -func (*initTables) Up(basicRes context.BasicRes) errors.Error { - return migrationhelper.AutoMigrateTables( - basicRes, - &archived.QDevConnection{}, - &archived.QDevUserData{}, - &archived.QDevUserMetrics{}, - &archived.QDevS3FileMeta{}, - ) -} - -func (*initTables) Version() uint64 { - return 20250319 -} diff --git a/backend/plugins/q_dev/models/migrationscripts/20250320_modify_file_meta.go b/backend/plugins/q_dev/models/migrationscripts/20250320_modify_file_meta.go deleted file mode 100644 index 83311cf077d..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/20250320_modify_file_meta.go +++ /dev/null @@ -1,111 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package migrationscripts - -import ( - "github.com/apache/incubator-devlake/core/context" - "github.com/apache/incubator-devlake/core/dal" - "github.com/apache/incubator-devlake/core/errors" -) - -type modifyFileMetaTable struct{} - -func (*modifyFileMetaTable) Name() string { - return "Modify QDevS3FileMeta table to allow NULL processed_time" -} - -func (*modifyFileMetaTable) Up(basicRes context.BasicRes) errors.Error { - db := basicRes.GetDal() - - // Target table and column - tableName := "_tool_q_dev_s3_file_meta" - columnName := "processed_time" - - // If column doesn't exist, no migration needed, idempotent - if !db.HasColumn(tableName, columnName) { - return nil - } - - // Read column metadata to check if already nullable, return idempotently if already nullable - var processedTimeNullable bool - { - cols, err := db.GetColumns(dal.DefaultTabler{Name: tableName}, func(cm dal.ColumnMeta) bool { - return cm.Name() == columnName - }) - if err != nil { - return errors.Default.Wrap(err, "failed to load column metadata for _tool_q_dev_s3_file_meta.processed_time") - } - if len(cols) == 0 { - // If column is not visible in metadata, treat as no processing needed - return nil - } - if nullable, ok := cols[0].Nullable(); ok { - processedTimeNullable = nullable - } - } - if processedTimeNullable { - return nil - } - - // Execute compatible SQL by dialect - switch db.Dialect() { - case "postgres": - // PostgreSQL makes column nullable via DROP NOT NULL, without changing data type - if err := db.Exec( - "ALTER TABLE ? ALTER COLUMN ? DROP NOT NULL", - dal.ClauseTable{Name: tableName}, - dal.ClauseColumn{Name: columnName}, - ); err != nil { - return errors.Default.Wrap(err, "failed to drop NOT NULL on processed_time for postgres") - } - return nil - case "mysql": - // MySQL requires MODIFY COLUMN with original type specification, preserve original type as much as possible - cols, err := db.GetColumns(dal.DefaultTabler{Name: tableName}, func(cm dal.ColumnMeta) bool { - return cm.Name() == columnName - }) - if err != nil { - return errors.Default.Wrap(err, "failed to load column metadata for mysql type preservation") - } - columnTypeSql := "DATETIME" - if len(cols) > 0 { - if ct, ok := cols[0].ColumnType(); ok && ct != "" { - columnTypeSql = ct - } else if dbt := cols[0].DatabaseTypeName(); dbt != "" { - // DatabaseTypeName may return DATETIME, TIMESTAMP etc - columnTypeSql = dbt - } - } - alterSql := "ALTER TABLE ? MODIFY COLUMN ? " + columnTypeSql + " NULL" - if err := db.Exec( - alterSql, - dal.ClauseTable{Name: tableName}, - dal.ClauseColumn{Name: columnName}, - ); err != nil { - return errors.Default.Wrap(err, "failed to modify processed_time to NULL for mysql") - } - return nil - default: - // Other dialects are not forced to migrate for now, return idempotently - return nil - } -} - -func (*modifyFileMetaTable) Version() uint64 { - return 20250320 -} diff --git a/backend/plugins/q_dev/models/migrationscripts/20250623_add_display_name_fields.go b/backend/plugins/q_dev/models/migrationscripts/20250623_add_display_name_fields.go deleted file mode 100644 index f0c772faf91..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/20250623_add_display_name_fields.go +++ /dev/null @@ -1,70 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package migrationscripts - -import ( - "github.com/apache/incubator-devlake/core/context" - "github.com/apache/incubator-devlake/core/errors" - "github.com/apache/incubator-devlake/core/plugin" - "github.com/apache/incubator-devlake/helpers/migrationhelper" -) - -var _ plugin.MigrationScript = (*addDisplayNameFields)(nil) - -type addDisplayNameFields struct{} - -type QDevConnection20250623 struct { - IdentityStoreId string `gorm:"type:VARCHAR(255)"` - IdentityStoreRegion string `gorm:"type:VARCHAR(255)"` -} - -func (QDevConnection20250623) TableName() string { - return "_tool_q_dev_connections" -} - -type QDevUserData20250623 struct { - DisplayName string `gorm:"type:VARCHAR(255)"` -} - -func (QDevUserData20250623) TableName() string { - return "_tool_q_dev_user_data" -} - -type QDevUserMetrics20250623 struct { - DisplayName string `gorm:"type:VARCHAR(255)"` -} - -func (QDevUserMetrics20250623) TableName() string { - return "_tool_q_dev_user_metrics" -} - -func (*addDisplayNameFields) Up(basicRes context.BasicRes) errors.Error { - return migrationhelper.AutoMigrateTables(basicRes, - &QDevConnection20250623{}, - &QDevUserData20250623{}, - &QDevUserMetrics20250623{}, - ) -} - -func (*addDisplayNameFields) Version() uint64 { - return 20250623000001 -} - -func (*addDisplayNameFields) Name() string { - return "add Identity Center fields to connections and display_name fields to user tables" -} diff --git a/backend/plugins/q_dev/models/migrationscripts/20250709_delete_user_metrics.go b/backend/plugins/q_dev/models/migrationscripts/20250709_delete_user_metrics.go deleted file mode 100644 index 4908559e339..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/20250709_delete_user_metrics.go +++ /dev/null @@ -1,46 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package migrationscripts - -import ( - "github.com/apache/incubator-devlake/core/context" - "github.com/apache/incubator-devlake/core/errors" - "github.com/apache/incubator-devlake/core/plugin" -) - -var _ plugin.MigrationScript = (*deleteUserMetrics)(nil) - -type deleteUserMetrics struct{} - -func (*deleteUserMetrics) Up(basicRes context.BasicRes) errors.Error { - db := basicRes.GetDal() - - // Drop the QDevUserMetrics table - // Ignore error if table doesn't exist - _ = db.Exec("DROP TABLE IF EXISTS _tool_q_dev_user_metrics") - - return nil -} - -func (*deleteUserMetrics) Version() uint64 { - return 20250709000001 -} - -func (*deleteUserMetrics) Name() string { - return "delete QDevUserMetrics table" -} diff --git a/backend/plugins/q_dev/models/migrationscripts/20250710_add_missing_metrics.go b/backend/plugins/q_dev/models/migrationscripts/20250710_add_missing_metrics.go deleted file mode 100644 index d2c6ebe391b..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/20250710_add_missing_metrics.go +++ /dev/null @@ -1,78 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package migrationscripts - -import ( - "github.com/apache/incubator-devlake/core/context" - "github.com/apache/incubator-devlake/core/errors" - "github.com/apache/incubator-devlake/core/plugin" -) - -var _ plugin.MigrationScript = (*addMissingMetrics)(nil) - -type addMissingMetrics struct{} - -func (*addMissingMetrics) Up(basicRes context.BasicRes) errors.Error { - db := basicRes.GetDal() - - // Add all missing metrics columns to _tool_q_dev_user_data table - // All columns are integer type with default value 0 - // Using snake_case column names to match GORM's default naming convention - _ = db.Exec(` - ALTER TABLE _tool_q_dev_user_data - ADD COLUMN chat_ai_code_lines INT DEFAULT 0, - ADD COLUMN chat_messages_interacted INT DEFAULT 0, - ADD COLUMN chat_messages_sent INT DEFAULT 0, - ADD COLUMN code_fix_acceptance_event_count INT DEFAULT 0, - ADD COLUMN code_fix_accepted_lines INT DEFAULT 0, - ADD COLUMN code_fix_generated_lines INT DEFAULT 0, - ADD COLUMN code_fix_generation_event_count INT DEFAULT 0, - ADD COLUMN code_review_failed_event_count INT DEFAULT 0, - ADD COLUMN dev_acceptance_event_count INT DEFAULT 0, - ADD COLUMN dev_accepted_lines INT DEFAULT 0, - ADD COLUMN dev_generated_lines INT DEFAULT 0, - ADD COLUMN dev_generation_event_count INT DEFAULT 0, - ADD COLUMN doc_generation_accepted_file_updates INT DEFAULT 0, - ADD COLUMN doc_generation_accepted_files_creations INT DEFAULT 0, - ADD COLUMN doc_generation_accepted_line_additions INT DEFAULT 0, - ADD COLUMN doc_generation_accepted_line_updates INT DEFAULT 0, - ADD COLUMN doc_generation_event_count INT DEFAULT 0, - ADD COLUMN doc_generation_rejected_file_creations INT DEFAULT 0, - ADD COLUMN doc_generation_rejected_file_updates INT DEFAULT 0, - ADD COLUMN doc_generation_rejected_line_additions INT DEFAULT 0, - ADD COLUMN doc_generation_rejected_line_updates INT DEFAULT 0, - ADD COLUMN test_generation_accepted_lines INT DEFAULT 0, - ADD COLUMN test_generation_accepted_tests INT DEFAULT 0, - ADD COLUMN test_generation_event_count INT DEFAULT 0, - ADD COLUMN test_generation_generated_lines INT DEFAULT 0, - ADD COLUMN test_generation_generated_tests INT DEFAULT 0, - ADD COLUMN transformation_event_count INT DEFAULT 0, - ADD COLUMN transformation_lines_generated INT DEFAULT 0, - ADD COLUMN transformation_lines_ingested INT DEFAULT 0 - `) - - return nil -} - -func (*addMissingMetrics) Version() uint64 { - return 20250710000001 -} - -func (*addMissingMetrics) Name() string { - return "add missing metrics columns to QDevUserData table" -} diff --git a/backend/plugins/q_dev/models/migrationscripts/20250926_add_s3_slice_table.go b/backend/plugins/q_dev/models/migrationscripts/20250926_add_s3_slice_table.go deleted file mode 100644 index 19fe467b1bf..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/20250926_add_s3_slice_table.go +++ /dev/null @@ -1,42 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package migrationscripts - -import ( - "github.com/apache/incubator-devlake/core/context" - "github.com/apache/incubator-devlake/core/errors" - "github.com/apache/incubator-devlake/helpers/migrationhelper" - "github.com/apache/incubator-devlake/plugins/q_dev/models/migrationscripts/archived" -) - -type addS3SliceTable struct{} - -func (*addS3SliceTable) Up(basicRes context.BasicRes) errors.Error { - return migrationhelper.AutoMigrateTables( - basicRes, - &archived.QDevS3Slice{}, - ) -} - -func (*addS3SliceTable) Version() uint64 { - return 20250926 -} - -func (*addS3SliceTable) Name() string { - return "Add S3 slice table for QDev plugin" -} diff --git a/backend/plugins/q_dev/models/migrationscripts/20251123_add_scope_config_id_to_s3_slice.go b/backend/plugins/q_dev/models/migrationscripts/20251123_add_scope_config_id_to_s3_slice.go deleted file mode 100644 index fdeb1058398..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/20251123_add_scope_config_id_to_s3_slice.go +++ /dev/null @@ -1,46 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package migrationscripts - -import ( - "github.com/apache/incubator-devlake/core/context" - "github.com/apache/incubator-devlake/core/errors" - "github.com/apache/incubator-devlake/helpers/migrationhelper" -) - -type addScopeConfigIdToS3Slice struct{} - -type QDevS3Slice20251123 struct { - ScopeConfigId uint64 `gorm:"type:BIGINT DEFAULT 0"` -} - -func (QDevS3Slice20251123) TableName() string { - return "_tool_q_dev_s3_slices" -} - -func (*addScopeConfigIdToS3Slice) Up(basicRes context.BasicRes) errors.Error { - return migrationhelper.AutoMigrateTables(basicRes, &QDevS3Slice20251123{}) -} - -func (*addScopeConfigIdToS3Slice) Version() uint64 { - return 20251123000001 -} - -func (*addScopeConfigIdToS3Slice) Name() string { - return "Add scope_config_id column to S3 slice table" -} diff --git a/backend/plugins/q_dev/models/migrationscripts/20251209_add_scope_id_fields.go b/backend/plugins/q_dev/models/migrationscripts/20251209_add_scope_id_fields.go deleted file mode 100644 index a4448b01248..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/20251209_add_scope_id_fields.go +++ /dev/null @@ -1,64 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package migrationscripts - -import ( - "github.com/apache/incubator-devlake/core/context" - "github.com/apache/incubator-devlake/core/dal" - "github.com/apache/incubator-devlake/core/errors" - "github.com/apache/incubator-devlake/core/plugin" -) - -var _ plugin.MigrationScript = (*addScopeIdFields)(nil) - -type addScopeIdFields struct{} - -func (*addScopeIdFields) Up(basicRes context.BasicRes) errors.Error { - db := basicRes.GetDal() - - // Add scope_id column to _tool_q_dev_user_data table - // This field links user data to QDevS3Slice scope, which can then be mapped to projects via project_mapping - if !db.HasColumn("_tool_q_dev_user_data", "scope_id") { - if err := db.AddColumn("_tool_q_dev_user_data", "scope_id", dal.Varchar); err != nil { - return errors.Default.Wrap(err, "failed to add scope_id to _tool_q_dev_user_data") - } - } - - // Add index on scope_id for better query performance - _ = db.Exec(`CREATE INDEX idx_q_dev_user_data_scope_id ON _tool_q_dev_user_data(scope_id)`) - - // Add scope_id column to _tool_q_dev_s3_file_meta table - if !db.HasColumn("_tool_q_dev_s3_file_meta", "scope_id") { - if err := db.AddColumn("_tool_q_dev_s3_file_meta", "scope_id", dal.Varchar); err != nil { - return errors.Default.Wrap(err, "failed to add scope_id to _tool_q_dev_s3_file_meta") - } - } - - // Add index on scope_id - _ = db.Exec(`CREATE INDEX idx_q_dev_s3_file_meta_scope_id ON _tool_q_dev_s3_file_meta(scope_id)`) - - return nil -} - -func (*addScopeIdFields) Version() uint64 { - return 20251209000001 -} - -func (*addScopeIdFields) Name() string { - return "add scope_id field to QDevUserData and QDevS3FileMeta for project association" -} diff --git a/backend/plugins/q_dev/models/migrationscripts/20260219_add_user_report_table.go b/backend/plugins/q_dev/models/migrationscripts/20260219_add_user_report_table.go deleted file mode 100644 index 5f38c7407bf..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/20260219_add_user_report_table.go +++ /dev/null @@ -1,42 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package migrationscripts - -import ( - "github.com/apache/incubator-devlake/core/context" - "github.com/apache/incubator-devlake/core/errors" - "github.com/apache/incubator-devlake/helpers/migrationhelper" - "github.com/apache/incubator-devlake/plugins/q_dev/models/migrationscripts/archived" -) - -type addUserReportTable struct{} - -func (*addUserReportTable) Up(basicRes context.BasicRes) errors.Error { - return migrationhelper.AutoMigrateTables( - basicRes, - &archived.QDevUserReport{}, - ) -} - -func (*addUserReportTable) Version() uint64 { - return 20260219000001 -} - -func (*addUserReportTable) Name() string { - return "Add user_report table for Kiro credits/subscription metrics" -} diff --git a/backend/plugins/q_dev/models/migrationscripts/20260220_add_account_id_to_s3_slice.go b/backend/plugins/q_dev/models/migrationscripts/20260220_add_account_id_to_s3_slice.go deleted file mode 100644 index f0b0b897fa6..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/20260220_add_account_id_to_s3_slice.go +++ /dev/null @@ -1,49 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package migrationscripts - -import ( - "github.com/apache/incubator-devlake/core/context" - "github.com/apache/incubator-devlake/core/dal" - "github.com/apache/incubator-devlake/core/errors" - "github.com/apache/incubator-devlake/core/plugin" -) - -var _ plugin.MigrationScript = (*addAccountIdToS3Slice)(nil) - -type addAccountIdToS3Slice struct{} - -func (*addAccountIdToS3Slice) Up(basicRes context.BasicRes) errors.Error { - db := basicRes.GetDal() - - if !db.HasColumn("_tool_q_dev_s3_slices", "account_id") { - if err := db.AddColumn("_tool_q_dev_s3_slices", "account_id", dal.Varchar); err != nil { - return errors.Default.Wrap(err, "failed to add account_id to _tool_q_dev_s3_slices") - } - } - - return nil -} - -func (*addAccountIdToS3Slice) Version() uint64 { - return 20260220000001 -} - -func (*addAccountIdToS3Slice) Name() string { - return "add account_id column to _tool_q_dev_s3_slices for auto-constructing S3 prefixes" -} diff --git a/backend/plugins/q_dev/models/migrationscripts/20260228_fix_dedup_user_tables.go b/backend/plugins/q_dev/models/migrationscripts/20260228_fix_dedup_user_tables.go deleted file mode 100644 index 94b7666a9a5..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/20260228_fix_dedup_user_tables.go +++ /dev/null @@ -1,60 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package migrationscripts - -import ( - "github.com/apache/incubator-devlake/core/context" - "github.com/apache/incubator-devlake/core/errors" - "github.com/apache/incubator-devlake/helpers/migrationhelper" - "github.com/apache/incubator-devlake/plugins/q_dev/models/migrationscripts/archived" -) - -type fixDedupUserTables struct{} - -func (*fixDedupUserTables) Up(basicRes context.BasicRes) errors.Error { - db := basicRes.GetDal() - - // Drop old tables that used auto-increment ID (which caused data duplication) - err := db.DropTables( - "_tool_q_dev_user_report", - "_tool_q_dev_user_data", - ) - if err != nil { - return errors.Default.Wrap(err, "failed to drop old user tables") - } - - // Recreate tables with composite primary keys for proper deduplication - err = migrationhelper.AutoMigrateTables( - basicRes, - &archived.QDevUserReportV2{}, - &archived.QDevUserDataV2{}, - ) - if err != nil { - return errors.Default.Wrap(err, "failed to recreate user tables") - } - - return nil -} - -func (*fixDedupUserTables) Version() uint64 { - return 20260228000001 -} - -func (*fixDedupUserTables) Name() string { - return "Rebuild user_report and user_data tables with composite primary keys to fix data duplication" -} diff --git a/backend/plugins/q_dev/models/migrationscripts/20260228_reset_s3_file_meta_processed.go b/backend/plugins/q_dev/models/migrationscripts/20260228_reset_s3_file_meta_processed.go deleted file mode 100644 index 7d0b8ba1f4f..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/20260228_reset_s3_file_meta_processed.go +++ /dev/null @@ -1,51 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package migrationscripts - -import ( - "github.com/apache/incubator-devlake/core/context" - "github.com/apache/incubator-devlake/core/dal" - "github.com/apache/incubator-devlake/core/errors" -) - -type resetS3FileMetaProcessed struct{} - -func (*resetS3FileMetaProcessed) Up(basicRes context.BasicRes) errors.Error { - db := basicRes.GetDal() - - // Reset processed flag so data will be re-extracted with the new - // dedup-safe composite-PK schema on next pipeline run - err := db.UpdateColumn( - "_tool_q_dev_s3_file_meta", - "processed", false, - dal.Where("1 = 1"), - ) - if err != nil { - return errors.Default.Wrap(err, "failed to reset s3_file_meta processed flag") - } - - return nil -} - -func (*resetS3FileMetaProcessed) Version() uint64 { - return 20260228000002 -} - -func (*resetS3FileMetaProcessed) Name() string { - return "Reset s3_file_meta processed flag to re-extract data with dedup-safe schema" -} diff --git a/backend/plugins/q_dev/models/migrationscripts/20260314_add_logging_tables.go b/backend/plugins/q_dev/models/migrationscripts/20260314_add_logging_tables.go deleted file mode 100644 index cbd5943ecd2..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/20260314_add_logging_tables.go +++ /dev/null @@ -1,43 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package migrationscripts - -import ( - "github.com/apache/incubator-devlake/core/context" - "github.com/apache/incubator-devlake/core/errors" - "github.com/apache/incubator-devlake/helpers/migrationhelper" - "github.com/apache/incubator-devlake/plugins/q_dev/models/migrationscripts/archived" -) - -type addLoggingTables struct{} - -func (*addLoggingTables) Up(basicRes context.BasicRes) errors.Error { - return migrationhelper.AutoMigrateTables( - basicRes, - &archived.QDevChatLog{}, - &archived.QDevCompletionLog{}, - ) -} - -func (*addLoggingTables) Version() uint64 { - return 20260314000001 -} - -func (*addLoggingTables) Name() string { - return "Add chat_log and completion_log tables for Kiro logging data" -} diff --git a/backend/plugins/q_dev/models/migrationscripts/20260319_add_logging_fields.go b/backend/plugins/q_dev/models/migrationscripts/20260319_add_logging_fields.go deleted file mode 100644 index f98c3d1066f..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/20260319_add_logging_fields.go +++ /dev/null @@ -1,45 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package migrationscripts - -import ( - "github.com/apache/incubator-devlake/core/context" - "github.com/apache/incubator-devlake/core/errors" - "github.com/apache/incubator-devlake/helpers/migrationhelper" - "github.com/apache/incubator-devlake/plugins/q_dev/models/migrationscripts/archived" -) - -var _ = (*addLoggingFields)(nil) - -type addLoggingFields struct{} - -func (*addLoggingFields) Up(basicRes context.BasicRes) errors.Error { - return migrationhelper.AutoMigrateTables( - basicRes, - &archived.QDevChatLog{}, - &archived.QDevCompletionLog{}, - ) -} - -func (*addLoggingFields) Version() uint64 { - return 20260319000001 -} - -func (*addLoggingFields) Name() string { - return "Add code_reference_count, web_link_count, has_followup_prompts to chat_log; left/right_context_length to completion_log" -} diff --git a/backend/plugins/q_dev/models/migrationscripts/archived/chat_log.go b/backend/plugins/q_dev/models/migrationscripts/archived/chat_log.go deleted file mode 100644 index ee7d10a1e87..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/archived/chat_log.go +++ /dev/null @@ -1,53 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package archived - -import ( - "time" - - "github.com/apache/incubator-devlake/core/models/migrationscripts/archived" -) - -type QDevChatLog struct { - archived.NoPKModel - ConnectionId uint64 `gorm:"primaryKey"` - ScopeId string `gorm:"primaryKey;type:varchar(255)" json:"scopeId"` - RequestId string `gorm:"primaryKey;type:varchar(255)" json:"requestId"` - UserId string `gorm:"index;type:varchar(255)" json:"userId"` - DisplayName string `gorm:"type:varchar(255)" json:"displayName"` - Timestamp time.Time `gorm:"index" json:"timestamp"` - ChatTriggerType string `gorm:"type:varchar(50)" json:"chatTriggerType"` - HasCustomization bool `json:"hasCustomization"` - ConversationId string `gorm:"type:varchar(255)" json:"conversationId"` - UtteranceId string `gorm:"type:varchar(255)" json:"utteranceId"` - ModelId string `gorm:"type:varchar(100)" json:"modelId"` - PromptLength int `json:"promptLength"` - ResponseLength int `json:"responseLength"` - OpenFileCount int `json:"openFileCount"` - ActiveFileName string `gorm:"type:varchar(512)" json:"activeFileName"` - ActiveFileExtension string `gorm:"type:varchar(50)" json:"activeFileExtension"` - HasSteering bool `json:"hasSteering"` - IsSpecMode bool `json:"isSpecMode"` - CodeReferenceCount int `json:"codeReferenceCount"` - WebLinkCount int `json:"webLinkCount"` - HasFollowupPrompts bool `json:"hasFollowupPrompts"` -} - -func (QDevChatLog) TableName() string { - return "_tool_q_dev_chat_log" -} diff --git a/backend/plugins/q_dev/models/migrationscripts/archived/completion_log.go b/backend/plugins/q_dev/models/migrationscripts/archived/completion_log.go deleted file mode 100644 index 4acff956978..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/archived/completion_log.go +++ /dev/null @@ -1,44 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package archived - -import ( - "time" - - "github.com/apache/incubator-devlake/core/models/migrationscripts/archived" -) - -type QDevCompletionLog struct { - archived.NoPKModel - ConnectionId uint64 `gorm:"primaryKey"` - ScopeId string `gorm:"primaryKey;type:varchar(255)" json:"scopeId"` - RequestId string `gorm:"primaryKey;type:varchar(255)" json:"requestId"` - UserId string `gorm:"index;type:varchar(255)" json:"userId"` - DisplayName string `gorm:"type:varchar(255)" json:"displayName"` - Timestamp time.Time `gorm:"index" json:"timestamp"` - FileName string `gorm:"type:varchar(512)" json:"fileName"` - FileExtension string `gorm:"type:varchar(50)" json:"fileExtension"` - HasCustomization bool `json:"hasCustomization"` - CompletionsCount int `json:"completionsCount"` - LeftContextLength int `json:"leftContextLength"` - RightContextLength int `json:"rightContextLength"` -} - -func (QDevCompletionLog) TableName() string { - return "_tool_q_dev_completion_log" -} diff --git a/backend/plugins/q_dev/models/migrationscripts/archived/connection.go b/backend/plugins/q_dev/models/migrationscripts/archived/connection.go deleted file mode 100644 index 5d03159ccab..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/archived/connection.go +++ /dev/null @@ -1,46 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package archived - -import ( - commonArchived "github.com/apache/incubator-devlake/core/models/migrationscripts/archived" -) - -// QDevConn holds the essential information to connect to AWS S3 -type QDevConn struct { - // AccessKeyId for AWS - AccessKeyId string `mapstructure:"accessKeyId" json:"accessKeyId"` - // SecretAccessKey for AWS - SecretAccessKey string `mapstructure:"secretAccessKey" json:"secretAccessKey"` - // Region for AWS - Region string `mapstructure:"region" json:"region"` - // Bucket for AWS S3 - Bucket string `mapstructure:"bucket" json:"bucket"` - // RateLimitPerHour limits the API requests sent to AWS - RateLimitPerHour int `mapstructure:"rateLimitPerHour" json:"rateLimitPerHour"` -} - -// QDevConnection holds QDevConn plus ID/Name for database storage -type QDevConnection struct { - commonArchived.BaseConnection `mapstructure:",squash"` - QDevConn `mapstructure:",squash"` -} - -func (QDevConnection) TableName() string { - return "_tool_q_dev_connections" -} diff --git a/backend/plugins/q_dev/models/migrationscripts/archived/s3_file_meta.go b/backend/plugins/q_dev/models/migrationscripts/archived/s3_file_meta.go deleted file mode 100644 index a648563a5d4..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/archived/s3_file_meta.go +++ /dev/null @@ -1,38 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package archived - -import ( - "time" - - "github.com/apache/incubator-devlake/core/models/migrationscripts/archived" -) - -// QDevS3FileMeta 存储S3文件的元数据信息 -type QDevS3FileMeta struct { - archived.NoPKModel - ConnectionId uint64 `gorm:"primaryKey"` - FileName string `gorm:"primaryKey;type:varchar(255)"` - S3Path string `gorm:"type:varchar(512)" json:"s3Path"` - Processed bool `gorm:"default:false"` - ProcessedTime *time.Time `gorm:"default:null"` -} - -func (QDevS3FileMeta) TableName() string { - return "_tool_q_dev_s3_file_meta" -} diff --git a/backend/plugins/q_dev/models/migrationscripts/archived/s3_slice.go b/backend/plugins/q_dev/models/migrationscripts/archived/s3_slice.go deleted file mode 100644 index 77c7dec0db2..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/archived/s3_slice.go +++ /dev/null @@ -1,36 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package archived - -import ( - "github.com/apache/incubator-devlake/core/models/migrationscripts/archived" -) - -type QDevS3Slice struct { - archived.NoPKModel - ConnectionId uint64 `gorm:"primaryKey"` - Id string `gorm:"primaryKey;type:varchar(512)"` - Prefix string `gorm:"type:varchar(512);not null"` - BasePath string `gorm:"type:varchar(512)"` - Year int `gorm:"not null"` - Month *int -} - -func (QDevS3Slice) TableName() string { - return "_tool_q_dev_s3_slices" -} diff --git a/backend/plugins/q_dev/models/migrationscripts/archived/user_data.go b/backend/plugins/q_dev/models/migrationscripts/archived/user_data.go deleted file mode 100644 index 00e6a0a4416..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/archived/user_data.go +++ /dev/null @@ -1,51 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package archived - -import ( - "time" - - "github.com/apache/incubator-devlake/core/models/migrationscripts/archived" -) - -// QDevUserData 存储从CSV中提取的原始数据 -type QDevUserData struct { - archived.Model - ConnectionId uint64 `gorm:"primaryKey"` - UserId string `gorm:"index" json:"userId"` - Date time.Time `gorm:"index" json:"date"` - CodeReview_FindingsCount int - CodeReview_SucceededEventCount int - InlineChat_AcceptanceEventCount int - InlineChat_AcceptedLineAdditions int - InlineChat_AcceptedLineDeletions int - InlineChat_DismissalEventCount int - InlineChat_DismissedLineAdditions int - InlineChat_DismissedLineDeletions int - InlineChat_RejectedLineAdditions int - InlineChat_RejectedLineDeletions int - InlineChat_RejectionEventCount int - InlineChat_TotalEventCount int - Inline_AICodeLines int - Inline_AcceptanceCount int - Inline_SuggestionsCount int -} - -func (QDevUserData) TableName() string { - return "_tool_q_dev_user_data" -} diff --git a/backend/plugins/q_dev/models/migrationscripts/archived/user_data_v2.go b/backend/plugins/q_dev/models/migrationscripts/archived/user_data_v2.go deleted file mode 100644 index 8d5db8496a3..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/archived/user_data_v2.go +++ /dev/null @@ -1,81 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package archived - -import ( - "time" - - "github.com/apache/incubator-devlake/core/models/migrationscripts/archived" -) - -type QDevUserDataV2 struct { - archived.NoPKModel - ConnectionId uint64 `gorm:"primaryKey"` - ScopeId string `gorm:"primaryKey;type:varchar(255)" json:"scopeId"` - UserId string `gorm:"primaryKey;type:varchar(255)" json:"userId"` - Date time.Time `gorm:"primaryKey;type:date" json:"date"` - DisplayName string `gorm:"type:varchar(255)" json:"displayName"` - CodeReview_FindingsCount int - CodeReview_SucceededEventCount int - InlineChat_AcceptanceEventCount int - InlineChat_AcceptedLineAdditions int - InlineChat_AcceptedLineDeletions int - InlineChat_DismissalEventCount int - InlineChat_DismissedLineAdditions int - InlineChat_DismissedLineDeletions int - InlineChat_RejectedLineAdditions int - InlineChat_RejectedLineDeletions int - InlineChat_RejectionEventCount int - InlineChat_TotalEventCount int - Inline_AICodeLines int - Inline_AcceptanceCount int - Inline_SuggestionsCount int - Chat_AICodeLines int - Chat_MessagesInteracted int - Chat_MessagesSent int - CodeFix_AcceptanceEventCount int - CodeFix_AcceptedLines int - CodeFix_GeneratedLines int - CodeFix_GenerationEventCount int - CodeReview_FailedEventCount int - Dev_AcceptanceEventCount int - Dev_AcceptedLines int - Dev_GeneratedLines int - Dev_GenerationEventCount int - DocGeneration_AcceptedFileUpdates int - DocGeneration_AcceptedFilesCreations int - DocGeneration_AcceptedLineAdditions int - DocGeneration_AcceptedLineUpdates int - DocGeneration_EventCount int - DocGeneration_RejectedFileCreations int - DocGeneration_RejectedFileUpdates int - DocGeneration_RejectedLineAdditions int - DocGeneration_RejectedLineUpdates int - TestGeneration_AcceptedLines int - TestGeneration_AcceptedTests int - TestGeneration_EventCount int - TestGeneration_GeneratedLines int - TestGeneration_GeneratedTests int - Transformation_EventCount int - Transformation_LinesGenerated int - Transformation_LinesIngested int -} - -func (QDevUserDataV2) TableName() string { - return "_tool_q_dev_user_data" -} diff --git a/backend/plugins/q_dev/models/migrationscripts/archived/user_metrics.go b/backend/plugins/q_dev/models/migrationscripts/archived/user_metrics.go deleted file mode 100644 index be948feea3f..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/archived/user_metrics.go +++ /dev/null @@ -1,67 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package archived - -import ( - "time" - - "github.com/apache/incubator-devlake/core/models/migrationscripts/archived" -) - -// QDevUserMetrics 存储按用户聚合的指标数据 -type QDevUserMetrics struct { - archived.NoPKModel - ConnectionId uint64 `gorm:"primaryKey"` - UserId string `gorm:"primaryKey"` - FirstDate time.Time - LastDate time.Time - TotalDays int - - // 聚合指标 - TotalCodeReview_FindingsCount int - TotalCodeReview_SucceededEventCount int - TotalInlineChat_AcceptanceEventCount int - TotalInlineChat_AcceptedLineAdditions int - TotalInlineChat_AcceptedLineDeletions int - TotalInlineChat_DismissalEventCount int - TotalInlineChat_DismissedLineAdditions int - TotalInlineChat_DismissedLineDeletions int - TotalInlineChat_RejectedLineAdditions int - TotalInlineChat_RejectedLineDeletions int - TotalInlineChat_RejectionEventCount int - TotalInlineChat_TotalEventCount int - TotalInline_AICodeLines int - TotalInline_AcceptanceCount int - TotalInline_SuggestionsCount int - - // 平均指标 - AvgCodeReview_FindingsCount float64 - AvgCodeReview_SucceededEventCount float64 - AvgInlineChat_AcceptanceEventCount float64 - AvgInlineChat_TotalEventCount float64 - AvgInline_AICodeLines float64 - AvgInline_AcceptanceCount float64 - AvgInline_SuggestionsCount float64 - - // 接受率指标 - AcceptanceRate float64 -} - -func (QDevUserMetrics) TableName() string { - return "_tool_q_dev_user_metrics" -} diff --git a/backend/plugins/q_dev/models/migrationscripts/archived/user_report.go b/backend/plugins/q_dev/models/migrationscripts/archived/user_report.go deleted file mode 100644 index 53bef49b153..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/archived/user_report.go +++ /dev/null @@ -1,46 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package archived - -import ( - "time" - - "github.com/apache/incubator-devlake/core/models/migrationscripts/archived" -) - -type QDevUserReport struct { - archived.Model - ConnectionId uint64 `gorm:"primaryKey"` - UserId string `gorm:"index" json:"userId"` - Date time.Time `gorm:"index" json:"date"` - DisplayName string `gorm:"type:varchar(255)" json:"displayName"` - ScopeId string `gorm:"index;type:varchar(255)" json:"scopeId"` - ClientType string `gorm:"type:varchar(50)" json:"clientType"` - SubscriptionTier string `gorm:"type:varchar(50)" json:"subscriptionTier"` - ProfileId string `gorm:"type:varchar(512)" json:"profileId"` - ChatConversations int `json:"chatConversations"` - CreditsUsed float64 `json:"creditsUsed"` - OverageCap float64 `json:"overageCap"` - OverageCreditsUsed float64 `json:"overageCreditsUsed"` - OverageEnabled bool `json:"overageEnabled"` - TotalMessages int `json:"totalMessages"` -} - -func (QDevUserReport) TableName() string { - return "_tool_q_dev_user_report" -} diff --git a/backend/plugins/q_dev/models/migrationscripts/archived/user_report_v2.go b/backend/plugins/q_dev/models/migrationscripts/archived/user_report_v2.go deleted file mode 100644 index 7045874851c..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/archived/user_report_v2.go +++ /dev/null @@ -1,46 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package archived - -import ( - "time" - - "github.com/apache/incubator-devlake/core/models/migrationscripts/archived" -) - -type QDevUserReportV2 struct { - archived.NoPKModel - ConnectionId uint64 `gorm:"primaryKey"` - ScopeId string `gorm:"primaryKey;type:varchar(255)" json:"scopeId"` - UserId string `gorm:"primaryKey;type:varchar(255)" json:"userId"` - Date time.Time `gorm:"primaryKey;type:date" json:"date"` - ClientType string `gorm:"primaryKey;type:varchar(50)" json:"clientType"` - DisplayName string `gorm:"type:varchar(255)" json:"displayName"` - SubscriptionTier string `gorm:"type:varchar(50)" json:"subscriptionTier"` - ProfileId string `gorm:"type:varchar(512)" json:"profileId"` - ChatConversations int `json:"chatConversations"` - CreditsUsed float64 `json:"creditsUsed"` - OverageCap float64 `json:"overageCap"` - OverageCreditsUsed float64 `json:"overageCreditsUsed"` - OverageEnabled bool `json:"overageEnabled"` - TotalMessages int `json:"totalMessages"` -} - -func (QDevUserReportV2) TableName() string { - return "_tool_q_dev_user_report" -} diff --git a/backend/plugins/q_dev/models/migrationscripts/register.go b/backend/plugins/q_dev/models/migrationscripts/register.go deleted file mode 100644 index 8b5de0bcc16..00000000000 --- a/backend/plugins/q_dev/models/migrationscripts/register.go +++ /dev/null @@ -1,41 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package migrationscripts - -import ( - "github.com/apache/incubator-devlake/core/plugin" -) - -// All return all migration scripts -func All() []plugin.MigrationScript { - return []plugin.MigrationScript{ - new(initTables), - new(modifyFileMetaTable), - new(addDisplayNameFields), - new(addMissingMetrics), - new(addS3SliceTable), - new(addScopeConfigIdToS3Slice), - new(addScopeIdFields), - new(addUserReportTable), - new(addAccountIdToS3Slice), - new(fixDedupUserTables), - new(resetS3FileMetaProcessed), - new(addLoggingTables), - new(addLoggingFields), - } -} diff --git a/backend/plugins/q_dev/models/s3_file_meta.go b/backend/plugins/q_dev/models/s3_file_meta.go deleted file mode 100644 index 09f7480ad80..00000000000 --- a/backend/plugins/q_dev/models/s3_file_meta.go +++ /dev/null @@ -1,39 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package models - -import ( - "time" - - "github.com/apache/incubator-devlake/core/models/common" -) - -// QDevS3FileMeta 存储S3文件的元数据信息 -type QDevS3FileMeta struct { - common.NoPKModel - ConnectionId uint64 `gorm:"primaryKey"` - FileName string `gorm:"primaryKey;type:varchar(255)"` - S3Path string `gorm:"type:varchar(512)" json:"s3Path"` - ScopeId string `gorm:"type:varchar(255);index" json:"scopeId"` - Processed bool `gorm:"default:false"` - ProcessedTime *time.Time `gorm:"default:null"` -} - -func (QDevS3FileMeta) TableName() string { - return "_tool_q_dev_s3_file_meta" -} diff --git a/backend/plugins/q_dev/models/s3_slice.go b/backend/plugins/q_dev/models/s3_slice.go deleted file mode 100644 index 19ecd6920bd..00000000000 --- a/backend/plugins/q_dev/models/s3_slice.go +++ /dev/null @@ -1,278 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package models - -import ( - "fmt" - "strconv" - "strings" - - "github.com/apache/incubator-devlake/core/models/common" - "github.com/apache/incubator-devlake/core/plugin" - "gorm.io/gorm" -) - -// QDevS3Slice describes a time-sliced S3 prefix to collect from. -type QDevS3Slice struct { - common.Scope `mapstructure:",squash"` - Id string `json:"id" mapstructure:"id" gorm:"primaryKey;type:varchar(512)"` - Prefix string `json:"prefix" mapstructure:"prefix" gorm:"type:varchar(512);not null"` - BasePath string `json:"basePath" mapstructure:"basePath" gorm:"type:varchar(512)"` - AccountId string `json:"accountId,omitempty" mapstructure:"accountId" gorm:"type:varchar(255)"` - Year int `json:"year" mapstructure:"year" gorm:"not null"` - Month *int `json:"month,omitempty" mapstructure:"month"` - - Name string `json:"name" mapstructure:"name" gorm:"-"` - FullName string `json:"fullName" mapstructure:"fullName" gorm:"-"` -} - -func (QDevS3Slice) TableName() string { - return "_tool_q_dev_s3_slices" -} - -// BeforeSave ensures derived fields stay in sync before persisting. -func (s *QDevS3Slice) BeforeSave(_ *gorm.DB) error { - return s.normalize(true) -} - -// AfterFind fills derived fields for API responses. -func (s *QDevS3Slice) AfterFind(_ *gorm.DB) error { - return s.normalize(false) -} - -// normalize trims inputs, derives prefix/id/name fields, and optionally validates. -func (s *QDevS3Slice) normalize(strict bool) error { - if s == nil { - return nil - } - - s.BasePath = cleanPath(s.BasePath) - s.AccountId = strings.TrimSpace(s.AccountId) - s.Prefix = cleanPath(selectNonEmpty(s.Prefix, s.Id)) - - if s.Year <= 0 { - if err := s.deriveYearAndMonthFromPrefix(); err != nil && strict { - return err - } - } - - if s.Year <= 0 { - if strict { - return fmt.Errorf("year is required for QDev S3 slice") - } - } - - if s.Month != nil { - if *s.Month < 1 || *s.Month > 12 { - return fmt.Errorf("month must be between 1 and 12") - } - } - - if s.AccountId != "" { - // New-style scope: construct a logical identifier from component parts - s.Prefix = buildPrefixWithAccount(s.BasePath, s.AccountId, s.Year, s.Month) - } else { - // Legacy scope: derive prefix from basePath + year + month - if s.Prefix == "" { - s.Prefix = buildPrefix(s.BasePath, s.Year, s.Month) - } - - prefix := buildPrefix(s.BasePath, s.Year, s.Month) - if prefix != "" { - s.Prefix = prefix - } - } - - if s.Id == "" { - if s.AccountId != "" { - // Use URL-safe ID: account_year or account_year_month - if s.Month != nil { - s.Id = fmt.Sprintf("%s_%04d_%02d", s.AccountId, s.Year, *s.Month) - } else { - s.Id = fmt.Sprintf("%s_%04d", s.AccountId, s.Year) - } - } else { - s.Id = s.Prefix - } - } - - if s.AccountId != "" { - if s.Month != nil { - s.Name = fmt.Sprintf("%s %04d-%02d", s.AccountId, s.Year, *s.Month) - } else if s.Year > 0 { - s.Name = fmt.Sprintf("%s %04d", s.AccountId, s.Year) - } - } else { - if s.Month != nil { - s.Name = fmt.Sprintf("%04d-%02d", s.Year, *s.Month) - } else if s.Year > 0 { - s.Name = fmt.Sprintf("%04d", s.Year) - } - } - - if s.FullName == "" { - s.FullName = s.Prefix - } - - return nil -} - -func (s *QDevS3Slice) deriveYearAndMonthFromPrefix() error { - if s == nil { - return nil - } - segments := splitPath(s.Prefix) - if len(segments) == 0 { - return fmt.Errorf("prefix is empty") - } - last := segments[len(segments)-1] - if len(last) == 2 { - if month, err := strconv.Atoi(last); err == nil { - s.Month = ptr(month) - if len(segments) >= 2 { - yearSegment := segments[len(segments)-2] - year, yearErr := strconv.Atoi(yearSegment) - if yearErr != nil { - return yearErr - } - s.Year = year - base := segments[:len(segments)-2] - s.BasePath = strings.Join(base, "/") - return nil - } - } - } - if year, err := strconv.Atoi(last); err == nil { - s.Year = year - base := segments[:len(segments)-1] - s.BasePath = strings.Join(base, "/") - s.Month = nil - return nil - } - return fmt.Errorf("unable to derive year/month from prefix %q", s.Prefix) -} - -func (s QDevS3Slice) ScopeId() string { - return s.Id -} - -func (s QDevS3Slice) ScopeName() string { - if s.Name != "" { - return s.Name - } - if s.AccountId != "" { - if s.Month != nil { - return fmt.Sprintf("%s %04d-%02d", s.AccountId, s.Year, *s.Month) - } - if s.Year > 0 { - return fmt.Sprintf("%s %04d", s.AccountId, s.Year) - } - } - if s.Month != nil { - return fmt.Sprintf("%04d-%02d", s.Year, *s.Month) - } - if s.Year > 0 { - return fmt.Sprintf("%04d", s.Year) - } - return s.Prefix -} - -func (s QDevS3Slice) ScopeFullName() string { - if s.FullName != "" { - return s.FullName - } - return s.Prefix -} - -func (s QDevS3Slice) ScopeParams() interface{} { - return &QDevS3SliceParams{ - ConnectionId: s.ConnectionId, - Prefix: s.Prefix, - } -} - -// Sanitize returns a copy ready for JSON serialization. -func (s QDevS3Slice) Sanitize() QDevS3Slice { - _ = s.normalize(false) - return s -} - -type QDevS3SliceParams struct { - ConnectionId uint64 `json:"connectionId"` - Prefix string `json:"prefix"` -} - -var _ plugin.ToolLayerScope = (*QDevS3Slice)(nil) - -func buildPrefixWithAccount(basePath string, accountId string, year int, month *int) string { - parts := splitPath(basePath) - if accountId != "" { - parts = append(parts, accountId) - } - if year > 0 { - parts = append(parts, fmt.Sprintf("%04d", year)) - } - if month != nil { - parts = append(parts, fmt.Sprintf("%02d", *month)) - } - return strings.Join(parts, "/") -} - -func buildPrefix(basePath string, year int, month *int) string { - parts := splitPath(basePath) - if year > 0 { - parts = append(parts, fmt.Sprintf("%04d", year)) - } - if month != nil { - parts = append(parts, fmt.Sprintf("%02d", *month)) - } - return strings.Join(parts, "/") -} - -func splitPath(value string) []string { - if value == "" { - return nil - } - chunks := strings.Split(value, "/") - result := make([]string, 0, len(chunks)) - for _, chunk := range chunks { - trimmed := strings.TrimSpace(chunk) - if trimmed == "" { - continue - } - result = append(result, trimmed) - } - return result -} - -func cleanPath(value string) string { - return strings.Join(splitPath(value), "/") -} - -func selectNonEmpty(values ...string) string { - for _, v := range values { - if strings.TrimSpace(v) != "" { - return strings.TrimSpace(v) - } - } - return "" -} - -func ptr[T any](value T) *T { - return &value -} diff --git a/backend/plugins/q_dev/models/user_data.go b/backend/plugins/q_dev/models/user_data.go deleted file mode 100644 index 3d59f965ade..00000000000 --- a/backend/plugins/q_dev/models/user_data.go +++ /dev/null @@ -1,83 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package models - -import ( - "time" - - "github.com/apache/incubator-devlake/core/models/common" -) - -// QDevUserData 存储从CSV中提取的原始数据 -type QDevUserData struct { - common.NoPKModel - ConnectionId uint64 `gorm:"primaryKey"` - ScopeId string `gorm:"primaryKey;type:varchar(255)" json:"scopeId"` - UserId string `gorm:"primaryKey;type:varchar(255)" json:"userId"` - Date time.Time `gorm:"primaryKey;type:date" json:"date"` - DisplayName string `gorm:"type:varchar(255)" json:"displayName"` - - CodeReview_FindingsCount int - CodeReview_SucceededEventCount int - InlineChat_AcceptanceEventCount int - InlineChat_AcceptedLineAdditions int - InlineChat_AcceptedLineDeletions int - InlineChat_DismissalEventCount int - InlineChat_DismissedLineAdditions int - InlineChat_DismissedLineDeletions int - InlineChat_RejectedLineAdditions int - InlineChat_RejectedLineDeletions int - InlineChat_RejectionEventCount int - InlineChat_TotalEventCount int - Inline_AICodeLines int - Inline_AcceptanceCount int - Inline_SuggestionsCount int - Chat_AICodeLines int - Chat_MessagesInteracted int - Chat_MessagesSent int - CodeFix_AcceptanceEventCount int - CodeFix_AcceptedLines int - CodeFix_GeneratedLines int - CodeFix_GenerationEventCount int - CodeReview_FailedEventCount int - Dev_AcceptanceEventCount int - Dev_AcceptedLines int - Dev_GeneratedLines int - Dev_GenerationEventCount int - DocGeneration_AcceptedFileUpdates int - DocGeneration_AcceptedFilesCreations int - DocGeneration_AcceptedLineAdditions int - DocGeneration_AcceptedLineUpdates int - DocGeneration_EventCount int - DocGeneration_RejectedFileCreations int - DocGeneration_RejectedFileUpdates int - DocGeneration_RejectedLineAdditions int - DocGeneration_RejectedLineUpdates int - TestGeneration_AcceptedLines int - TestGeneration_AcceptedTests int - TestGeneration_EventCount int - TestGeneration_GeneratedLines int - TestGeneration_GeneratedTests int - Transformation_EventCount int - Transformation_LinesGenerated int - Transformation_LinesIngested int -} - -func (QDevUserData) TableName() string { - return "_tool_q_dev_user_data" -} diff --git a/backend/plugins/q_dev/models/user_data_test.go b/backend/plugins/q_dev/models/user_data_test.go deleted file mode 100644 index 74d5dfcfeea..00000000000 --- a/backend/plugins/q_dev/models/user_data_test.go +++ /dev/null @@ -1,137 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package models - -import ( - "testing" - "time" - - "github.com/stretchr/testify/assert" -) - -func TestQDevUserDataAllMetrics(t *testing.T) { - // Create a test user data object with all metrics - userData := &QDevUserData{ - ConnectionId: 1, - UserId: "test-user-id", - Date: time.Now(), - DisplayName: "Test User", - - // Set values for existing metrics - CodeReview_FindingsCount: 10, - CodeReview_SucceededEventCount: 11, - InlineChat_AcceptanceEventCount: 12, - InlineChat_AcceptedLineAdditions: 13, - InlineChat_AcceptedLineDeletions: 14, - InlineChat_DismissalEventCount: 15, - InlineChat_DismissedLineAdditions: 16, - InlineChat_DismissedLineDeletions: 17, - InlineChat_RejectedLineAdditions: 18, - InlineChat_RejectedLineDeletions: 19, - InlineChat_RejectionEventCount: 20, - InlineChat_TotalEventCount: 21, - Inline_AICodeLines: 22, - Inline_AcceptanceCount: 23, - Inline_SuggestionsCount: 24, - - // Set values for new metrics - Chat_AICodeLines: 25, - Chat_MessagesInteracted: 26, - Chat_MessagesSent: 27, - CodeFix_AcceptanceEventCount: 28, - CodeFix_AcceptedLines: 29, - CodeFix_GeneratedLines: 30, - CodeFix_GenerationEventCount: 31, - CodeReview_FailedEventCount: 32, - Dev_AcceptanceEventCount: 33, - Dev_AcceptedLines: 34, - Dev_GeneratedLines: 35, - Dev_GenerationEventCount: 36, - DocGeneration_AcceptedFileUpdates: 37, - DocGeneration_AcceptedFilesCreations: 38, - DocGeneration_AcceptedLineAdditions: 39, - DocGeneration_AcceptedLineUpdates: 40, - DocGeneration_EventCount: 41, - DocGeneration_RejectedFileCreations: 42, - DocGeneration_RejectedFileUpdates: 43, - DocGeneration_RejectedLineAdditions: 44, - DocGeneration_RejectedLineUpdates: 45, - TestGeneration_AcceptedLines: 46, - TestGeneration_AcceptedTests: 47, - TestGeneration_EventCount: 48, - TestGeneration_GeneratedLines: 49, - TestGeneration_GeneratedTests: 50, - Transformation_EventCount: 51, - Transformation_LinesGenerated: 52, - Transformation_LinesIngested: 53, - } - - // Verify that all metrics are accessible - // Existing metrics - assert.Equal(t, 10, userData.CodeReview_FindingsCount) - assert.Equal(t, 11, userData.CodeReview_SucceededEventCount) - assert.Equal(t, 12, userData.InlineChat_AcceptanceEventCount) - assert.Equal(t, 13, userData.InlineChat_AcceptedLineAdditions) - assert.Equal(t, 14, userData.InlineChat_AcceptedLineDeletions) - assert.Equal(t, 15, userData.InlineChat_DismissalEventCount) - assert.Equal(t, 16, userData.InlineChat_DismissedLineAdditions) - assert.Equal(t, 17, userData.InlineChat_DismissedLineDeletions) - assert.Equal(t, 18, userData.InlineChat_RejectedLineAdditions) - assert.Equal(t, 19, userData.InlineChat_RejectedLineDeletions) - assert.Equal(t, 20, userData.InlineChat_RejectionEventCount) - assert.Equal(t, 21, userData.InlineChat_TotalEventCount) - assert.Equal(t, 22, userData.Inline_AICodeLines) - assert.Equal(t, 23, userData.Inline_AcceptanceCount) - assert.Equal(t, 24, userData.Inline_SuggestionsCount) - - // New metrics - assert.Equal(t, 25, userData.Chat_AICodeLines) - assert.Equal(t, 26, userData.Chat_MessagesInteracted) - assert.Equal(t, 27, userData.Chat_MessagesSent) - assert.Equal(t, 28, userData.CodeFix_AcceptanceEventCount) - assert.Equal(t, 29, userData.CodeFix_AcceptedLines) - assert.Equal(t, 30, userData.CodeFix_GeneratedLines) - assert.Equal(t, 31, userData.CodeFix_GenerationEventCount) - assert.Equal(t, 32, userData.CodeReview_FailedEventCount) - assert.Equal(t, 33, userData.Dev_AcceptanceEventCount) - assert.Equal(t, 34, userData.Dev_AcceptedLines) - assert.Equal(t, 35, userData.Dev_GeneratedLines) - assert.Equal(t, 36, userData.Dev_GenerationEventCount) - assert.Equal(t, 37, userData.DocGeneration_AcceptedFileUpdates) - assert.Equal(t, 38, userData.DocGeneration_AcceptedFilesCreations) - assert.Equal(t, 39, userData.DocGeneration_AcceptedLineAdditions) - assert.Equal(t, 40, userData.DocGeneration_AcceptedLineUpdates) - assert.Equal(t, 41, userData.DocGeneration_EventCount) - assert.Equal(t, 42, userData.DocGeneration_RejectedFileCreations) - assert.Equal(t, 43, userData.DocGeneration_RejectedFileUpdates) - assert.Equal(t, 44, userData.DocGeneration_RejectedLineAdditions) - assert.Equal(t, 45, userData.DocGeneration_RejectedLineUpdates) - assert.Equal(t, 46, userData.TestGeneration_AcceptedLines) - assert.Equal(t, 47, userData.TestGeneration_AcceptedTests) - assert.Equal(t, 48, userData.TestGeneration_EventCount) - assert.Equal(t, 49, userData.TestGeneration_GeneratedLines) - assert.Equal(t, 50, userData.TestGeneration_GeneratedTests) - assert.Equal(t, 51, userData.Transformation_EventCount) - assert.Equal(t, 52, userData.Transformation_LinesGenerated) - assert.Equal(t, 53, userData.Transformation_LinesIngested) -} - -func TestQDevUserDataTableName(t *testing.T) { - userData := &QDevUserData{} - assert.Equal(t, "_tool_q_dev_user_data", userData.TableName()) -} diff --git a/backend/plugins/q_dev/models/user_report.go b/backend/plugins/q_dev/models/user_report.go deleted file mode 100644 index 17c4ac07daa..00000000000 --- a/backend/plugins/q_dev/models/user_report.go +++ /dev/null @@ -1,46 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package models - -import ( - "time" - - "github.com/apache/incubator-devlake/core/models/common" -) - -type QDevUserReport struct { - common.NoPKModel - ConnectionId uint64 `gorm:"primaryKey"` - ScopeId string `gorm:"primaryKey;type:varchar(255)" json:"scopeId"` - UserId string `gorm:"primaryKey;type:varchar(255)" json:"userId"` - Date time.Time `gorm:"primaryKey;type:date" json:"date"` - ClientType string `gorm:"primaryKey;type:varchar(50)" json:"clientType"` - DisplayName string `gorm:"type:varchar(255)" json:"displayName"` - SubscriptionTier string `gorm:"type:varchar(50)" json:"subscriptionTier"` - ProfileId string `gorm:"type:varchar(512)" json:"profileId"` - ChatConversations int `json:"chatConversations"` - CreditsUsed float64 `json:"creditsUsed"` - OverageCap float64 `json:"overageCap"` - OverageCreditsUsed float64 `json:"overageCreditsUsed"` - OverageEnabled bool `json:"overageEnabled"` - TotalMessages int `json:"totalMessages"` -} - -func (QDevUserReport) TableName() string { - return "_tool_q_dev_user_report" -} diff --git a/backend/plugins/q_dev/q_dev.go b/backend/plugins/q_dev/q_dev.go deleted file mode 100644 index 1c01f20a763..00000000000 --- a/backend/plugins/q_dev/q_dev.go +++ /dev/null @@ -1,43 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package main - -import ( - "github.com/apache/incubator-devlake/core/runner" - "github.com/apache/incubator-devlake/plugins/q_dev/impl" - "github.com/spf13/cobra" -) - -var PluginEntry impl.QDev - -// standalone mode for debugging -func main() { - cmd := &cobra.Command{Use: "q_dev"} - connectionId := cmd.Flags().Uint64P("connectionId", "c", 0, "q_dev connection id") - s3Prefix := cmd.Flags().StringP("s3Prefix", "p", "", "s3 bucket prefix for q_dev data") - timeAfter := cmd.Flags().StringP("timeAfter", "a", "", "collect data that are created after specified time, ie 2006-01-02T15:04:05Z") - - _ = cmd.MarkFlagRequired("connectionId") - cmd.Run = func(cmd *cobra.Command, args []string) { - runner.DirectRun(cmd, args, PluginEntry, map[string]interface{}{ - "connectionId": *connectionId, - "s3Prefix": *s3Prefix, - }, *timeAfter) - } - runner.RunCmd(cmd) -} diff --git a/backend/plugins/q_dev/tasks/identity_client.go b/backend/plugins/q_dev/tasks/identity_client.go deleted file mode 100644 index 855ce4ebdc0..00000000000 --- a/backend/plugins/q_dev/tasks/identity_client.go +++ /dev/null @@ -1,96 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package tasks - -import ( - "github.com/aws/aws-sdk-go/aws" - "github.com/aws/aws-sdk-go/aws/credentials" - "github.com/aws/aws-sdk-go/aws/session" - "github.com/aws/aws-sdk-go/service/identitystore" - - "github.com/apache/incubator-devlake/plugins/q_dev/models" -) - -// IdentityStoreAPI interface for AWS Identity Store operations -// This allows for easier testing with mocks -type IdentityStoreAPI interface { - DescribeUser(input *identitystore.DescribeUserInput) (*identitystore.DescribeUserOutput, error) -} - -// QDevIdentityClient wraps AWS Identity Store client for user display name resolution -type QDevIdentityClient struct { - IdentityStore IdentityStoreAPI - StoreId string - Region string -} - -// NewQDevIdentityClient creates a new Identity Store client for the given connection -// Returns nil if Identity Store is not configured (empty ID or region) -func NewQDevIdentityClient(connection *models.QDevConnection) (*QDevIdentityClient, error) { - // Return nil if Identity Store is not configured - if connection.IdentityStoreId == "" || connection.IdentityStoreRegion == "" { - return nil, nil - } - - // Create AWS session with Identity Store region and credentials - sess, err := session.NewSession(&aws.Config{ - Region: aws.String(connection.IdentityStoreRegion), - Credentials: credentials.NewStaticCredentials( - connection.AccessKeyId, - connection.SecretAccessKey, - "", // No session token - ), - }) - if err != nil { - return nil, err - } - - return &QDevIdentityClient{ - IdentityStore: identitystore.New(sess), - StoreId: connection.IdentityStoreId, - Region: connection.IdentityStoreRegion, - }, nil -} - -// ResolveUserDisplayName resolves a user ID to a human-readable display name -// Returns the display name if found, otherwise returns the original userId as fallback -func (client *QDevIdentityClient) ResolveUserDisplayName(userId string) (string, error) { - // Check if client or IdentityStore is nil - if client == nil || client.IdentityStore == nil { - return userId, nil - } - - input := &identitystore.DescribeUserInput{ - IdentityStoreId: aws.String(client.StoreId), - UserId: aws.String(userId), - } - - result, err := client.IdentityStore.DescribeUser(input) - if err != nil { - // Return userId as fallback on error, but still return the error for logging - return userId, err - } - - // Check if DisplayName exists and is not empty - if result.DisplayName != nil && *result.DisplayName != "" { - return *result.DisplayName, nil - } - - // Fallback to userId if no display name available - return userId, nil -} diff --git a/backend/plugins/q_dev/tasks/identity_client_test.go b/backend/plugins/q_dev/tasks/identity_client_test.go deleted file mode 100644 index 933457ddc44..00000000000 --- a/backend/plugins/q_dev/tasks/identity_client_test.go +++ /dev/null @@ -1,195 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package tasks - -import ( - "errors" - "testing" - - "github.com/aws/aws-sdk-go/service/identitystore" - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/mock" - - "github.com/apache/incubator-devlake/plugins/q_dev/models" -) - -// Mock IdentityStore interface for testing -type MockIdentityStoreAPI struct { - mock.Mock -} - -func (m *MockIdentityStoreAPI) DescribeUser(input *identitystore.DescribeUserInput) (*identitystore.DescribeUserOutput, error) { - args := m.Called(input) - if args.Get(0) == nil { - return nil, args.Error(1) - } - return args.Get(0).(*identitystore.DescribeUserOutput), args.Error(1) -} - -func TestNewQDevIdentityClient_Success(t *testing.T) { - connection := &models.QDevConnection{ - QDevConn: models.QDevConn{ - AccessKeyId: "test-key", - SecretAccessKey: "test-secret", - IdentityStoreId: "d-1234567890", - IdentityStoreRegion: "us-west-2", - }, - } - - client, err := NewQDevIdentityClient(connection) - assert.NoError(t, err) - assert.NotNil(t, client) - assert.Equal(t, "d-1234567890", client.StoreId) - assert.Equal(t, "us-west-2", client.Region) -} - -func TestNewQDevIdentityClient_EmptyIdentityStoreId(t *testing.T) { - connection := &models.QDevConnection{ - QDevConn: models.QDevConn{ - AccessKeyId: "test-key", - SecretAccessKey: "test-secret", - IdentityStoreId: "", // Empty identity store ID - IdentityStoreRegion: "us-west-2", - }, - } - - client, err := NewQDevIdentityClient(connection) - assert.NoError(t, err) - assert.Nil(t, client) // Should return nil when no identity store configured -} - -func TestNewQDevIdentityClient_EmptyIdentityStoreRegion(t *testing.T) { - connection := &models.QDevConnection{ - QDevConn: models.QDevConn{ - AccessKeyId: "test-key", - SecretAccessKey: "test-secret", - IdentityStoreId: "d-1234567890", - IdentityStoreRegion: "", // Empty identity store region - }, - } - - client, err := NewQDevIdentityClient(connection) - assert.NoError(t, err) - assert.Nil(t, client) // Should return nil when no region configured -} - -func TestQDevIdentityClient_ResolveUserDisplayName_Success(t *testing.T) { - mockAPI := &MockIdentityStoreAPI{} - client := &QDevIdentityClient{ - IdentityStore: mockAPI, - StoreId: "d-1234567890", - Region: "us-west-2", - } - - displayName := "John Doe" - mockAPI.On("DescribeUser", mock.AnythingOfType("*identitystore.DescribeUserInput")).Return( - &identitystore.DescribeUserOutput{ - DisplayName: &displayName, - }, nil) - - result, err := client.ResolveUserDisplayName("user-123") - assert.NoError(t, err) - assert.Equal(t, "John Doe", result) - - mockAPI.AssertExpectations(t) -} - -func TestQDevIdentityClient_ResolveUserDisplayName_NoDisplayName(t *testing.T) { - mockAPI := &MockIdentityStoreAPI{} - client := &QDevIdentityClient{ - IdentityStore: mockAPI, - StoreId: "d-1234567890", - Region: "us-west-2", - } - - // Return output with nil DisplayName - mockAPI.On("DescribeUser", mock.AnythingOfType("*identitystore.DescribeUserInput")).Return( - &identitystore.DescribeUserOutput{ - DisplayName: nil, - }, nil) - - result, err := client.ResolveUserDisplayName("user-123") - assert.NoError(t, err) - assert.Equal(t, "user-123", result) // Should fallback to UUID - - mockAPI.AssertExpectations(t) -} - -func TestQDevIdentityClient_ResolveUserDisplayName_EmptyDisplayName(t *testing.T) { - mockAPI := &MockIdentityStoreAPI{} - client := &QDevIdentityClient{ - IdentityStore: mockAPI, - StoreId: "d-1234567890", - Region: "us-west-2", - } - - emptyName := "" - mockAPI.On("DescribeUser", mock.AnythingOfType("*identitystore.DescribeUserInput")).Return( - &identitystore.DescribeUserOutput{ - DisplayName: &emptyName, - }, nil) - - result, err := client.ResolveUserDisplayName("user-123") - assert.NoError(t, err) - assert.Equal(t, "user-123", result) // Should fallback to UUID when empty - - mockAPI.AssertExpectations(t) -} - -func TestQDevIdentityClient_ResolveUserDisplayName_APIError(t *testing.T) { - mockAPI := &MockIdentityStoreAPI{} - client := &QDevIdentityClient{ - IdentityStore: mockAPI, - StoreId: "d-1234567890", - Region: "us-west-2", - } - - mockAPI.On("DescribeUser", mock.AnythingOfType("*identitystore.DescribeUserInput")).Return( - nil, errors.New("user not found")) - - result, err := client.ResolveUserDisplayName("user-123") - assert.Error(t, err) - assert.Equal(t, "user-123", result) // Should fallback to UUID on error - assert.Contains(t, err.Error(), "user not found") - - mockAPI.AssertExpectations(t) -} - -func TestQDevIdentityClient_ResolveUserDisplayName_InputValidation(t *testing.T) { - mockAPI := &MockIdentityStoreAPI{} - client := &QDevIdentityClient{ - IdentityStore: mockAPI, - StoreId: "d-1234567890", - Region: "us-west-2", - } - - displayName := "Jane Smith" - mockAPI.On("DescribeUser", mock.MatchedBy(func(input *identitystore.DescribeUserInput) bool { - // Verify the input parameters are correctly set - return *input.IdentityStoreId == "d-1234567890" && *input.UserId == "test-user-456" - })).Return( - &identitystore.DescribeUserOutput{ - DisplayName: &displayName, - }, nil) - - result, err := client.ResolveUserDisplayName("test-user-456") - assert.NoError(t, err) - assert.Equal(t, "Jane Smith", result) - - mockAPI.AssertExpectations(t) -} diff --git a/backend/plugins/q_dev/tasks/s3_client.go b/backend/plugins/q_dev/tasks/s3_client.go deleted file mode 100644 index a45b5022a8d..00000000000 --- a/backend/plugins/q_dev/tasks/s3_client.go +++ /dev/null @@ -1,47 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package tasks - -import ( - "github.com/apache/incubator-devlake/core/errors" - "github.com/apache/incubator-devlake/core/plugin" - "github.com/apache/incubator-devlake/plugins/q_dev/models" - "github.com/aws/aws-sdk-go/aws" - "github.com/aws/aws-sdk-go/aws/credentials" - "github.com/aws/aws-sdk-go/aws/session" - "github.com/aws/aws-sdk-go/service/s3" -) - -func NewQDevS3Client(taskCtx plugin.TaskContext, connection *models.QDevConnection) (*QDevS3Client, errors.Error) { - // 创建AWS session - sess, err := session.NewSession(&aws.Config{ - Region: aws.String(connection.Region), - Credentials: credentials.NewStaticCredentials(connection.AccessKeyId, connection.SecretAccessKey, ""), - }) - if err != nil { - return nil, errors.Convert(err) - } - - // 创建S3服务客户端 - s3Client := s3.New(sess) - - return &QDevS3Client{ - S3: s3Client, - Bucket: connection.Bucket, - }, nil -} diff --git a/backend/plugins/q_dev/tasks/s3_data_extractor.go b/backend/plugins/q_dev/tasks/s3_data_extractor.go deleted file mode 100644 index da29bca07fe..00000000000 --- a/backend/plugins/q_dev/tasks/s3_data_extractor.go +++ /dev/null @@ -1,445 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package tasks - -import ( - "encoding/csv" - "fmt" - "io" - "strconv" - "strings" - "time" - - "github.com/apache/incubator-devlake/core/dal" - "github.com/apache/incubator-devlake/core/errors" - "github.com/apache/incubator-devlake/core/plugin" - "github.com/apache/incubator-devlake/plugins/q_dev/models" - "github.com/aws/aws-sdk-go/aws" - "github.com/aws/aws-sdk-go/service/s3" -) - -var _ plugin.SubTaskEntryPoint = ExtractQDevS3Data - -// ExtractQDevS3Data 从S3下载CSV数据并解析 -func ExtractQDevS3Data(taskCtx plugin.SubTaskContext) errors.Error { - data := taskCtx.GetData().(*QDevTaskData) - db := taskCtx.GetDal() - - // 查询未处理的CSV文件元数据（排除.json.gz日志文件） - cursor, err := db.Cursor( - dal.From(&models.QDevS3FileMeta{}), - dal.Where("connection_id = ? AND processed = ? AND file_name LIKE ?", - data.Options.ConnectionId, false, "%.csv"), - ) - if err != nil { - return errors.Default.Wrap(err, "failed to get file metadata cursor") - } - defer cursor.Close() - - taskCtx.SetProgress(0, -1) - - // 处理每个文件 - for cursor.Next() { - fileMeta := &models.QDevS3FileMeta{} - err = db.Fetch(cursor, fileMeta) - if err != nil { - return errors.Default.Wrap(err, "failed to fetch file metadata") - } - - // 获取文件内容 - getInput := &s3.GetObjectInput{ - Bucket: aws.String(data.S3Client.Bucket), - Key: aws.String(fileMeta.S3Path), - } - - getResult, err := data.S3Client.S3.GetObject(getInput) - if err != nil { - return errors.Convert(err) - } - - // Use a transaction to process the file and update its status - tx := db.Begin() - csvErr := processCSVData(taskCtx, tx, getResult.Body, fileMeta) - if csvErr != nil { - if rollbackErr := tx.Rollback(); rollbackErr != nil { - taskCtx.GetLogger().Error(rollbackErr, "failed to rollback transaction") - } - return errors.Default.Wrap(csvErr, fmt.Sprintf("failed to process CSV file %s", fileMeta.FileName)) - } - - // Update file processing status within the same transaction - fileMeta.Processed = true - now := time.Now() - fileMeta.ProcessedTime = &now - err = tx.Update(fileMeta) - if err != nil { - if rollbackErr := tx.Rollback(); rollbackErr != nil { - taskCtx.GetLogger().Error(rollbackErr, "failed to rollback transaction") - } - return errors.Default.Wrap(err, "failed to update file metadata") - } - - // Commit the transaction - err = tx.Commit() - if err != nil { - return errors.Default.Wrap(err, "failed to commit transaction") - } - - taskCtx.IncProgress(1) - } - - return nil -} - -// 处理CSV文件 -func processCSVData(taskCtx plugin.SubTaskContext, db dal.Dal, reader io.ReadCloser, fileMeta *models.QDevS3FileMeta) errors.Error { - defer reader.Close() - - // Get task data to access Identity Client - data := taskCtx.GetData().(*QDevTaskData) - - csvReader := csv.NewReader(reader) - // 使用默认的逗号分隔符，不需要设置 Comma - csvReader.LazyQuotes = true // 允许非标准引号处理 - csvReader.FieldsPerRecord = -1 // 允许每行字段数不同 - - // 读取标头 - headers, err := csvReader.Read() - taskCtx.GetLogger().Debug("CSV headers: %+v", headers) - if err != nil { - return errors.Convert(err) - } - - // Auto-detect CSV format from headers - isNewFormat := detectUserReportFormat(headers) - if isNewFormat { - taskCtx.GetLogger().Debug("Detected new user_report CSV format") - } else { - taskCtx.GetLogger().Debug("Detected old by_user_analytic CSV format") - } - - // 逐行读取数据 - for { - record, err := csvReader.Read() - if err == io.EOF { - break - } - if err != nil { - return errors.Convert(err) - } - - if isNewFormat { - reportData, err := createUserReportData(taskCtx.GetLogger(), headers, record, fileMeta, data.IdentityClient) - if err != nil { - return errors.Default.Wrap(err, "failed to create user report data") - } - err = db.CreateOrUpdate(reportData) - if err != nil { - return errors.Default.Wrap(err, "failed to save user report data") - } - } else { - // 创建用户数据对象 (updated to include display name resolution) - userData, err := createUserDataWithDisplayName(taskCtx.GetLogger(), headers, record, fileMeta, data.IdentityClient) - if err != nil { - return errors.Default.Wrap(err, "failed to create user data") - } - - err = db.CreateOrUpdate(userData) - if err != nil { - return errors.Default.Wrap(err, "failed to save user data") - } - } - } - - return nil -} - -// detectUserReportFormat checks CSV headers to determine if this is the new user_report format -func detectUserReportFormat(headers []string) bool { - for _, h := range headers { - trimmed := strings.TrimSpace(h) - if trimmed == "Client_Type" || trimmed == "Credits_Used" { - return true - } - } - return false -} - -// createUserReportData creates a QDevUserReport from a new-format CSV record -func createUserReportData(logger interface { - Debug(format string, a ...interface{}) -}, headers []string, record []string, fileMeta *models.QDevS3FileMeta, identityClient UserDisplayNameResolver) (*models.QDevUserReport, errors.Error) { - report := &models.QDevUserReport{ - ConnectionId: fileMeta.ConnectionId, - ScopeId: fileMeta.ScopeId, - } - - // Build field map - fieldMap := make(map[string]string) - for i, header := range headers { - if i < len(record) { - logger.Debug("Mapping header[%d]: '%s' -> '%s'", i, header, record[i]) - fieldMap[header] = record[i] - trimmedHeader := strings.TrimSpace(header) - if trimmedHeader != header { - logger.Debug("Also adding trimmed header: '%s'", trimmedHeader) - fieldMap[trimmedHeader] = record[i] - } - } - } - - // UserId (normalize to strip "d-{directoryId}." prefix if present) - report.UserId = normalizeUserId(getStringField(fieldMap, "UserId")) - if report.UserId == "" { - return nil, errors.Default.New("UserId not found in CSV record") - } - - // DisplayName - report.DisplayName = resolveDisplayName(logger, report.UserId, identityClient) - - // Date - dateStr := getStringField(fieldMap, "Date") - if dateStr == "" { - return nil, errors.Default.New("Date not found in CSV record") - } - var err error - report.Date, err = parseDate(dateStr) - if err != nil { - return nil, errors.Default.Wrap(err, "failed to parse date") - } - - // String fields - report.ClientType = getStringField(fieldMap, "Client_Type") - report.SubscriptionTier = getStringField(fieldMap, "Subscription_Tier") - report.ProfileId = getStringField(fieldMap, "ProfileId") - - // Numeric fields - report.ChatConversations = parseInt(fieldMap, "Chat_Conversations") - report.CreditsUsed = parseFloat(fieldMap, "Credits_Used") - report.OverageCap = parseFloat(fieldMap, "Overage_Cap") - report.OverageCreditsUsed = parseFloat(fieldMap, "Overage_Credits_Used") - report.OverageEnabled = parseBool(fieldMap, "Overage_Enabled") - report.TotalMessages = parseInt(fieldMap, "Total_Messages") - - return report, nil -} - -// getStringField returns the string value for a field, or empty string if not found -func getStringField(fieldMap map[string]string, field string) string { - value, ok := fieldMap[field] - if !ok { - return "" - } - return value -} - -// parseFloat extracts a float64 from the field map, returning 0 if missing or invalid -func parseFloat(fieldMap map[string]string, field string) float64 { - value, ok := fieldMap[field] - if !ok { - return 0 - } - f, err := strconv.ParseFloat(strings.TrimSpace(value), 64) - if err != nil { - return 0 - } - return f -} - -// parseBool extracts a boolean from the field map, returning false if missing or invalid -func parseBool(fieldMap map[string]string, field string) bool { - value, ok := fieldMap[field] - if !ok { - return false - } - trimmed := strings.TrimSpace(strings.ToLower(value)) - return trimmed == "true" || trimmed == "1" || trimmed == "yes" -} - -// UserDisplayNameResolver interface for resolving user display names -type UserDisplayNameResolver interface { - ResolveUserDisplayName(userId string) (string, error) -} - -// 从CSV记录创建用户数据对象 (enhanced with display name resolution) -func createUserDataWithDisplayName(logger interface { - Debug(format string, a ...interface{}) -}, headers []string, record []string, fileMeta *models.QDevS3FileMeta, identityClient UserDisplayNameResolver) (*models.QDevUserData, errors.Error) { - userData := &models.QDevUserData{ - ConnectionId: fileMeta.ConnectionId, - ScopeId: fileMeta.ScopeId, - } - - // 创建字段映射 - fieldMap := make(map[string]string) - for i, header := range headers { - if i < len(record) { - logger.Debug("Mapping header[%d]: '%s' -> '%s'", i, header, record[i]) - fieldMap[header] = record[i] - // 同时添加去除空格的版本 - trimmedHeader := strings.TrimSpace(header) - if trimmedHeader != header { - logger.Debug("Also adding trimmed header: '%s'", trimmedHeader) - fieldMap[trimmedHeader] = record[i] - } - } - } - - // 设置必要字段 - var err error - var ok bool - - // 设置UserId (normalize to strip "d-{directoryId}." prefix if present) - rawUserId, ok := fieldMap["UserId"] - if !ok { - return nil, errors.Default.New("UserId not found in CSV record") - } - userData.UserId = normalizeUserId(rawUserId) - - // 设置DisplayName (new functionality) - userData.DisplayName = resolveDisplayName(logger, userData.UserId, identityClient) - - // 设置Date - dateStr, ok := fieldMap["Date"] - if !ok { - return nil, errors.Default.New("Date not found in CSV record") - } - - userData.Date, err = parseDate(dateStr) - if err != nil { - return nil, errors.Default.Wrap(err, "failed to parse date") - } - - // 设置所有指标字段 - userData.CodeReview_FindingsCount = parseInt(fieldMap, "CodeReview_FindingsCount") - userData.CodeReview_SucceededEventCount = parseInt(fieldMap, "CodeReview_SucceededEventCount") - userData.InlineChat_AcceptanceEventCount = parseInt(fieldMap, "InlineChat_AcceptanceEventCount") - userData.InlineChat_AcceptedLineAdditions = parseInt(fieldMap, "InlineChat_AcceptedLineAdditions") - userData.InlineChat_AcceptedLineDeletions = parseInt(fieldMap, "InlineChat_AcceptedLineDeletions") - userData.InlineChat_DismissalEventCount = parseInt(fieldMap, "InlineChat_DismissalEventCount") - userData.InlineChat_DismissedLineAdditions = parseInt(fieldMap, "InlineChat_DismissedLineAdditions") - userData.InlineChat_DismissedLineDeletions = parseInt(fieldMap, "InlineChat_DismissedLineDeletions") - userData.InlineChat_RejectedLineAdditions = parseInt(fieldMap, "InlineChat_RejectedLineAdditions") - userData.InlineChat_RejectedLineDeletions = parseInt(fieldMap, "InlineChat_RejectedLineDeletions") - userData.InlineChat_RejectionEventCount = parseInt(fieldMap, "InlineChat_RejectionEventCount") - userData.InlineChat_TotalEventCount = parseInt(fieldMap, "InlineChat_TotalEventCount") - userData.Inline_AICodeLines = parseInt(fieldMap, "Inline_AICodeLines") - userData.Inline_AcceptanceCount = parseInt(fieldMap, "Inline_AcceptanceCount") - userData.Inline_SuggestionsCount = parseInt(fieldMap, "Inline_SuggestionsCount") - userData.Chat_AICodeLines = parseInt(fieldMap, "Chat_AICodeLines") - userData.Chat_MessagesInteracted = parseInt(fieldMap, "Chat_MessagesInteracted") - userData.Chat_MessagesSent = parseInt(fieldMap, "Chat_MessagesSent") - userData.CodeFix_AcceptanceEventCount = parseInt(fieldMap, "CodeFix_AcceptanceEventCount") - userData.CodeFix_AcceptedLines = parseInt(fieldMap, "CodeFix_AcceptedLines") - userData.CodeFix_GeneratedLines = parseInt(fieldMap, "CodeFix_GeneratedLines") - userData.CodeFix_GenerationEventCount = parseInt(fieldMap, "CodeFix_GenerationEventCount") - userData.CodeReview_FailedEventCount = parseInt(fieldMap, "CodeReview_FailedEventCount") - userData.Dev_AcceptanceEventCount = parseInt(fieldMap, "Dev_AcceptanceEventCount") - userData.Dev_AcceptedLines = parseInt(fieldMap, "Dev_AcceptedLines") - userData.Dev_GeneratedLines = parseInt(fieldMap, "Dev_GeneratedLines") - userData.Dev_GenerationEventCount = parseInt(fieldMap, "Dev_GenerationEventCount") - userData.DocGeneration_AcceptedFileUpdates = parseInt(fieldMap, "DocGeneration_AcceptedFileUpdates") - userData.DocGeneration_AcceptedFilesCreations = parseInt(fieldMap, "DocGeneration_AcceptedFilesCreations") - userData.DocGeneration_AcceptedLineAdditions = parseInt(fieldMap, "DocGeneration_AcceptedLineAdditions") - userData.DocGeneration_AcceptedLineUpdates = parseInt(fieldMap, "DocGeneration_AcceptedLineUpdates") - userData.DocGeneration_EventCount = parseInt(fieldMap, "DocGeneration_EventCount") - userData.DocGeneration_RejectedFileCreations = parseInt(fieldMap, "DocGeneration_RejectedFileCreations") - userData.DocGeneration_RejectedFileUpdates = parseInt(fieldMap, "DocGeneration_RejectedFileUpdates") - userData.DocGeneration_RejectedLineAdditions = parseInt(fieldMap, "DocGeneration_RejectedLineAdditions") - userData.DocGeneration_RejectedLineUpdates = parseInt(fieldMap, "DocGeneration_RejectedLineUpdates") - userData.TestGeneration_AcceptedLines = parseInt(fieldMap, "TestGeneration_AcceptedLines") - userData.TestGeneration_AcceptedTests = parseInt(fieldMap, "TestGeneration_AcceptedTests") - userData.TestGeneration_EventCount = parseInt(fieldMap, "TestGeneration_EventCount") - userData.TestGeneration_GeneratedLines = parseInt(fieldMap, "TestGeneration_GeneratedLines") - userData.TestGeneration_GeneratedTests = parseInt(fieldMap, "TestGeneration_GeneratedTests") - userData.Transformation_EventCount = parseInt(fieldMap, "Transformation_EventCount") - userData.Transformation_LinesGenerated = parseInt(fieldMap, "Transformation_LinesGenerated") - userData.Transformation_LinesIngested = parseInt(fieldMap, "Transformation_LinesIngested") - - return userData, nil -} - -// resolveDisplayName resolves user ID to display name using Identity Client -func resolveDisplayName(logger interface { - Debug(format string, a ...interface{}) -}, userId string, identityClient UserDisplayNameResolver) string { - // If no identity client available, use userId as fallback - if identityClient == nil { - return userId - } - - // Try to resolve display name - displayName, err := identityClient.ResolveUserDisplayName(userId) - if err != nil { - // Log error but continue with userId as fallback - logger.Debug("Failed to resolve display name for user %s: %v", userId, err) - return userId - } - - // If display name is empty, use userId as fallback - if displayName == "" { - return userId - } - - return displayName -} - -// 解析日期 -func parseDate(dateStr string) (time.Time, errors.Error) { - // 尝试常见的日期格式 - formats := []string{ - "2006-01-02", - "2006/01/02", - "01/02/2006", - "01-02-2006", - time.RFC3339, - } - - for _, format := range formats { - date, err := time.Parse(format, dateStr) - if err == nil { - return date, nil - } - } - - return time.Time{}, errors.Default.New(fmt.Sprintf("failed to parse date: %s", dateStr)) -} - -// 解析整数 -func parseInt(fieldMap map[string]string, field string) int { - value, ok := fieldMap[field] - if !ok { - return 0 - } - - intValue, err := strconv.Atoi(value) - if err != nil { - return 0 - } - - return intValue -} - -var ExtractQDevS3DataMeta = plugin.SubTaskMeta{ - Name: "extractQDevS3Data", - EntryPoint: ExtractQDevS3Data, - EnabledByDefault: true, - Description: "Extract data from S3 CSV files and save to database", - DomainTypes: []string{plugin.DOMAIN_TYPE_CROSS}, - Dependencies: []*plugin.SubTaskMeta{&CollectQDevS3FilesMeta}, -} diff --git a/backend/plugins/q_dev/tasks/s3_data_extractor_test.go b/backend/plugins/q_dev/tasks/s3_data_extractor_test.go deleted file mode 100644 index 0a5f808ebb1..00000000000 --- a/backend/plugins/q_dev/tasks/s3_data_extractor_test.go +++ /dev/null @@ -1,604 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package tasks - -import ( - "testing" - "time" - - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/mock" - - "github.com/apache/incubator-devlake/plugins/q_dev/models" -) - -// Mock Identity Client for testing -type MockIdentityClient struct { - mock.Mock -} - -func (m *MockIdentityClient) ResolveUserDisplayName(userId string) (string, error) { - args := m.Called(userId) - return args.String(0), args.Error(1) -} - -// Ensure MockIdentityClient implements UserDisplayNameResolver -var _ UserDisplayNameResolver = (*MockIdentityClient)(nil) - -// MockLogger is a mock implementation of the logger interface for testing -type MockLogger struct { - mock.Mock -} - -func (m *MockLogger) Debug(format string, args ...interface{}) { - m.Called(format, args) -} - -func TestCreateUserDataWithDisplayName_Success(t *testing.T) { - headers := []string{"UserId", "Date", "CodeReview_FindingsCount", "Inline_AcceptanceCount"} - record := []string{"user-123", "2025-06-23", "5", "10"} - fileMeta := &models.QDevS3FileMeta{ - ConnectionId: 1, - } - - mockIdentityClient := &MockIdentityClient{} - mockIdentityClient.On("ResolveUserDisplayName", "user-123").Return("John Doe", nil) - - mockLogger := &MockLogger{} - // Add expectations for Debug calls - mockLogger.On("Debug", "Mapping header[%d]: '%s' -> '%s'", mock.Anything).Return() - mockLogger.On("Debug", "Also adding trimmed header: '%s'", mock.Anything).Return() - - userData, err := createUserDataWithDisplayName(mockLogger, headers, record, fileMeta, mockIdentityClient) - - assert.NoError(t, err) - assert.NotNil(t, userData) - assert.Equal(t, "user-123", userData.UserId) - assert.Equal(t, "John Doe", userData.DisplayName) - assert.Equal(t, uint64(1), userData.ConnectionId) - assert.Equal(t, 5, userData.CodeReview_FindingsCount) - assert.Equal(t, 10, userData.Inline_AcceptanceCount) - - mockIdentityClient.AssertExpectations(t) -} - -func TestCreateUserDataWithDisplayName_FallbackToUUID(t *testing.T) { - headers := []string{"UserId", "Date"} - record := []string{"user-456", "2025-06-23"} - fileMeta := &models.QDevS3FileMeta{ - ConnectionId: 1, - } - - mockIdentityClient := &MockIdentityClient{} - mockIdentityClient.On("ResolveUserDisplayName", "user-456").Return("user-456", assert.AnError) - - mockLogger := &MockLogger{} - // Add expectations for Debug calls - mockLogger.On("Debug", "Mapping header[%d]: '%s' -> '%s'", mock.Anything).Return() - mockLogger.On("Debug", "Also adding trimmed header: '%s'", mock.Anything).Return() - mockLogger.On("Debug", "Failed to resolve display name for user %s: %v", mock.Anything).Return() - - userData, err := createUserDataWithDisplayName(mockLogger, headers, record, fileMeta, mockIdentityClient) - - assert.NoError(t, err) - assert.NotNil(t, userData) - assert.Equal(t, "user-456", userData.UserId) - assert.Equal(t, "user-456", userData.DisplayName) // Should fallback to UUID - - mockIdentityClient.AssertExpectations(t) -} - -func TestCreateUserDataWithDisplayName_NoIdentityClient(t *testing.T) { - headers := []string{"UserId", "Date"} - record := []string{"user-789", "2025-06-23"} - fileMeta := &models.QDevS3FileMeta{ - ConnectionId: 1, - } - - mockLogger := &MockLogger{} - // Add expectations for Debug calls - mockLogger.On("Debug", "Mapping header[%d]: '%s' -> '%s'", mock.Anything).Return() - mockLogger.On("Debug", "Also adding trimmed header: '%s'", mock.Anything).Return() - - userData, err := createUserDataWithDisplayName(mockLogger, headers, record, fileMeta, nil) - - assert.NoError(t, err) - assert.NotNil(t, userData) - assert.Equal(t, "user-789", userData.UserId) - assert.Equal(t, "user-789", userData.DisplayName) // Should use UUID when no client -} - -func TestCreateUserDataWithDisplayName_EmptyDisplayName(t *testing.T) { - headers := []string{"UserId", "Date"} - record := []string{"user-empty", "2025-06-23"} - fileMeta := &models.QDevS3FileMeta{ - ConnectionId: 1, - } - - mockIdentityClient := &MockIdentityClient{} - mockIdentityClient.On("ResolveUserDisplayName", "user-empty").Return("", nil) - - mockLogger := &MockLogger{} - // Add expectations for Debug calls - mockLogger.On("Debug", "Mapping header[%d]: '%s' -> '%s'", mock.Anything).Return() - mockLogger.On("Debug", "Also adding trimmed header: '%s'", mock.Anything).Return() - - userData, err := createUserDataWithDisplayName(mockLogger, headers, record, fileMeta, mockIdentityClient) - - assert.NoError(t, err) - assert.NotNil(t, userData) - assert.Equal(t, "user-empty", userData.UserId) - assert.Equal(t, "user-empty", userData.DisplayName) // Should fallback when empty - - mockIdentityClient.AssertExpectations(t) -} - -func TestCreateUserDataWithDisplayName_AllExistingMetrics(t *testing.T) { - headers := []string{ - "UserId", "Date", "CodeReview_FindingsCount", "CodeReview_SucceededEventCount", - "InlineChat_AcceptanceEventCount", "InlineChat_AcceptedLineAdditions", - "InlineChat_AcceptedLineDeletions", "InlineChat_DismissalEventCount", - "InlineChat_DismissedLineAdditions", "InlineChat_DismissedLineDeletions", - "InlineChat_RejectedLineAdditions", "InlineChat_RejectedLineDeletions", - "InlineChat_RejectionEventCount", "InlineChat_TotalEventCount", - "Inline_AICodeLines", "Inline_AcceptanceCount", "Inline_SuggestionsCount", - } - record := []string{ - "test-user", "2025-06-23", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15", - } - fileMeta := &models.QDevS3FileMeta{ - ConnectionId: 123, - } - - mockIdentityClient := &MockIdentityClient{} - mockIdentityClient.On("ResolveUserDisplayName", "test-user").Return("Test User", nil) - - mockLogger := &MockLogger{} - // Add expectations for Debug calls - mockLogger.On("Debug", "Mapping header[%d]: '%s' -> '%s'", mock.Anything).Return() - mockLogger.On("Debug", "Also adding trimmed header: '%s'", mock.Anything).Return() - - userData, err := createUserDataWithDisplayName(mockLogger, headers, record, fileMeta, mockIdentityClient) - - assert.NoError(t, err) - assert.NotNil(t, userData) - - // Verify basic fields - assert.Equal(t, "test-user", userData.UserId) - assert.Equal(t, "Test User", userData.DisplayName) - assert.Equal(t, uint64(123), userData.ConnectionId) - - // Verify date parsing - expectedDate, _ := time.Parse("2006-01-02", "2025-06-23") - assert.Equal(t, expectedDate, userData.Date) - - // Verify all existing metric fields - assert.Equal(t, 1, userData.CodeReview_FindingsCount) - assert.Equal(t, 2, userData.CodeReview_SucceededEventCount) - assert.Equal(t, 3, userData.InlineChat_AcceptanceEventCount) - assert.Equal(t, 4, userData.InlineChat_AcceptedLineAdditions) - assert.Equal(t, 5, userData.InlineChat_AcceptedLineDeletions) - assert.Equal(t, 6, userData.InlineChat_DismissalEventCount) - assert.Equal(t, 7, userData.InlineChat_DismissedLineAdditions) - assert.Equal(t, 8, userData.InlineChat_DismissedLineDeletions) - assert.Equal(t, 9, userData.InlineChat_RejectedLineAdditions) - assert.Equal(t, 10, userData.InlineChat_RejectedLineDeletions) - assert.Equal(t, 11, userData.InlineChat_RejectionEventCount) - assert.Equal(t, 12, userData.InlineChat_TotalEventCount) - assert.Equal(t, 13, userData.Inline_AICodeLines) - assert.Equal(t, 14, userData.Inline_AcceptanceCount) - assert.Equal(t, 15, userData.Inline_SuggestionsCount) - - mockIdentityClient.AssertExpectations(t) -} - -func TestCreateUserDataWithDisplayName_AllNewMetrics(t *testing.T) { - headers := []string{ - "UserId", "Date", - "Chat_AICodeLines", "Chat_MessagesInteracted", "Chat_MessagesSent", - "CodeFix_AcceptanceEventCount", "CodeFix_AcceptedLines", "CodeFix_GeneratedLines", "CodeFix_GenerationEventCount", - "CodeReview_FailedEventCount", - "Dev_AcceptanceEventCount", "Dev_AcceptedLines", "Dev_GeneratedLines", "Dev_GenerationEventCount", - "DocGeneration_AcceptedFileUpdates", "DocGeneration_AcceptedFilesCreations", "DocGeneration_AcceptedLineAdditions", - "DocGeneration_AcceptedLineUpdates", "DocGeneration_EventCount", "DocGeneration_RejectedFileCreations", - "DocGeneration_RejectedFileUpdates", "DocGeneration_RejectedLineAdditions", "DocGeneration_RejectedLineUpdates", - "TestGeneration_AcceptedLines", "TestGeneration_AcceptedTests", "TestGeneration_EventCount", - "TestGeneration_GeneratedLines", "TestGeneration_GeneratedTests", - "Transformation_EventCount", "Transformation_LinesGenerated", "Transformation_LinesIngested", - } - - record := []string{ - "test-user", "2025-06-23", - "101", "102", "103", "104", "105", "106", "107", "108", "109", "110", - "111", "112", "113", "114", "115", "116", "117", "118", "119", "120", - "121", "122", "123", "124", "125", "126", "127", "128", "129", - } - - fileMeta := &models.QDevS3FileMeta{ - ConnectionId: 123, - } - - mockIdentityClient := &MockIdentityClient{} - mockIdentityClient.On("ResolveUserDisplayName", "test-user").Return("Test User", nil) - - mockLogger := &MockLogger{} - // Add expectations for Debug calls - mockLogger.On("Debug", "Mapping header[%d]: '%s' -> '%s'", mock.Anything).Return() - mockLogger.On("Debug", "Also adding trimmed header: '%s'", mock.Anything).Return() - - userData, err := createUserDataWithDisplayName(mockLogger, headers, record, fileMeta, mockIdentityClient) - - assert.NoError(t, err) - assert.NotNil(t, userData) - - // Verify basic fields - assert.Equal(t, "test-user", userData.UserId) - assert.Equal(t, "Test User", userData.DisplayName) - - // Verify all new metric fields - assert.Equal(t, 101, userData.Chat_AICodeLines) - assert.Equal(t, 102, userData.Chat_MessagesInteracted) - assert.Equal(t, 103, userData.Chat_MessagesSent) - assert.Equal(t, 104, userData.CodeFix_AcceptanceEventCount) - assert.Equal(t, 105, userData.CodeFix_AcceptedLines) - assert.Equal(t, 106, userData.CodeFix_GeneratedLines) - assert.Equal(t, 107, userData.CodeFix_GenerationEventCount) - assert.Equal(t, 108, userData.CodeReview_FailedEventCount) - assert.Equal(t, 109, userData.Dev_AcceptanceEventCount) - assert.Equal(t, 110, userData.Dev_AcceptedLines) - assert.Equal(t, 111, userData.Dev_GeneratedLines) - assert.Equal(t, 112, userData.Dev_GenerationEventCount) - assert.Equal(t, 113, userData.DocGeneration_AcceptedFileUpdates) - assert.Equal(t, 114, userData.DocGeneration_AcceptedFilesCreations) - assert.Equal(t, 115, userData.DocGeneration_AcceptedLineAdditions) - assert.Equal(t, 116, userData.DocGeneration_AcceptedLineUpdates) - assert.Equal(t, 117, userData.DocGeneration_EventCount) - assert.Equal(t, 118, userData.DocGeneration_RejectedFileCreations) - assert.Equal(t, 119, userData.DocGeneration_RejectedFileUpdates) - assert.Equal(t, 120, userData.DocGeneration_RejectedLineAdditions) - assert.Equal(t, 121, userData.DocGeneration_RejectedLineUpdates) - assert.Equal(t, 122, userData.TestGeneration_AcceptedLines) - assert.Equal(t, 123, userData.TestGeneration_AcceptedTests) - assert.Equal(t, 124, userData.TestGeneration_EventCount) - assert.Equal(t, 125, userData.TestGeneration_GeneratedLines) - assert.Equal(t, 126, userData.TestGeneration_GeneratedTests) - assert.Equal(t, 127, userData.Transformation_EventCount) - assert.Equal(t, 128, userData.Transformation_LinesGenerated) - assert.Equal(t, 129, userData.Transformation_LinesIngested) - - mockIdentityClient.AssertExpectations(t) -} - -func TestCreateUserDataWithDisplayName_MissingMetrics(t *testing.T) { - // Only provide a few metrics in the CSV - headers := []string{"UserId", "Date", "CodeReview_FindingsCount", "Chat_AICodeLines"} - record := []string{"test-user", "2025-06-23", "42", "99"} - - fileMeta := &models.QDevS3FileMeta{ - ConnectionId: 123, - } - - mockIdentityClient := &MockIdentityClient{} - mockIdentityClient.On("ResolveUserDisplayName", "test-user").Return("Test User", nil) - - mockLogger := &MockLogger{} - // Add expectations for Debug calls - mockLogger.On("Debug", "Mapping header[%d]: '%s' -> '%s'", mock.Anything).Return() - mockLogger.On("Debug", "Also adding trimmed header: '%s'", mock.Anything).Return() - - userData, err := createUserDataWithDisplayName(mockLogger, headers, record, fileMeta, mockIdentityClient) - - assert.NoError(t, err) - assert.NotNil(t, userData) - - // Verify provided metrics are set correctly - assert.Equal(t, 42, userData.CodeReview_FindingsCount) - assert.Equal(t, 99, userData.Chat_AICodeLines) - - // Verify missing metrics are set to 0 - assert.Equal(t, 0, userData.CodeReview_SucceededEventCount) - assert.Equal(t, 0, userData.InlineChat_AcceptanceEventCount) - assert.Equal(t, 0, userData.Chat_MessagesInteracted) - assert.Equal(t, 0, userData.TestGeneration_AcceptedTests) - assert.Equal(t, 0, userData.Transformation_LinesIngested) - - mockIdentityClient.AssertExpectations(t) -} - -func TestCreateUserDataWithDisplayName_InvalidMetricValues(t *testing.T) { - headers := []string{ - "UserId", "Date", "CodeReview_FindingsCount", "Chat_AICodeLines", - "InlineChat_AcceptanceEventCount", "TestGeneration_AcceptedTests", - } - record := []string{"test-user", "2025-06-23", "42", "not-a-number", "abc", ""} - - fileMeta := &models.QDevS3FileMeta{ - ConnectionId: 123, - } - - mockIdentityClient := &MockIdentityClient{} - mockIdentityClient.On("ResolveUserDisplayName", "test-user").Return("Test User", nil) - - mockLogger := &MockLogger{} - // Add expectations for Debug calls - mockLogger.On("Debug", "Mapping header[%d]: '%s' -> '%s'", mock.Anything).Return() - mockLogger.On("Debug", "Also adding trimmed header: '%s'", mock.Anything).Return() - - userData, err := createUserDataWithDisplayName(mockLogger, headers, record, fileMeta, mockIdentityClient) - - assert.NoError(t, err) - assert.NotNil(t, userData) - - // Verify valid metric is set correctly - assert.Equal(t, 42, userData.CodeReview_FindingsCount) - - // Verify invalid metrics are set to 0 - assert.Equal(t, 0, userData.Chat_AICodeLines) - assert.Equal(t, 0, userData.InlineChat_AcceptanceEventCount) - assert.Equal(t, 0, userData.TestGeneration_AcceptedTests) - - mockIdentityClient.AssertExpectations(t) -} - -func TestCreateUserDataWithDisplayName_MissingUserId(t *testing.T) { - headers := []string{"Date", "CodeReview_FindingsCount"} - record := []string{"2025-06-23", "5"} - fileMeta := &models.QDevS3FileMeta{ - ConnectionId: 1, - } - - mockLogger := &MockLogger{} - // Add expectations for Debug calls - mockLogger.On("Debug", "Mapping header[%d]: '%s' -> '%s'", mock.Anything).Return() - mockLogger.On("Debug", "Also adding trimmed header: '%s'", mock.Anything).Return() - - userData, err := createUserDataWithDisplayName(mockLogger, headers, record, fileMeta, nil) - - assert.Error(t, err) - assert.Nil(t, userData) - assert.Contains(t, err.Error(), "UserId not found") -} - -func TestCreateUserDataWithDisplayName_MissingDate(t *testing.T) { - headers := []string{"UserId", "CodeReview_FindingsCount"} - record := []string{"user-123", "5"} - fileMeta := &models.QDevS3FileMeta{ - ConnectionId: 1, - } - - mockLogger := &MockLogger{} - // Add expectations for Debug calls - mockLogger.On("Debug", "Mapping header[%d]: '%s' -> '%s'", mock.Anything).Return() - mockLogger.On("Debug", "Also adding trimmed header: '%s'", mock.Anything).Return() - - userData, err := createUserDataWithDisplayName(mockLogger, headers, record, fileMeta, nil) - - assert.Error(t, err) - assert.Nil(t, userData) - assert.Contains(t, err.Error(), "Date not found") -} - -func TestParseDate(t *testing.T) { - testCases := []struct { - dateStr string - expectedDate time.Time - expectError bool - }{ - {"2025-07-10", time.Date(2025, 7, 10, 0, 0, 0, 0, time.UTC), false}, - {"2025/07/10", time.Date(2025, 7, 10, 0, 0, 0, 0, time.UTC), false}, - {"07/10/2025", time.Date(2025, 7, 10, 0, 0, 0, 0, time.UTC), false}, - {"07-10-2025", time.Date(2025, 7, 10, 0, 0, 0, 0, time.UTC), false}, - {"2025-07-10T15:04:05Z", time.Date(2025, 7, 10, 15, 4, 5, 0, time.UTC), false}, - {"invalid-date", time.Time{}, true}, - } - - for _, tc := range testCases { - date, err := parseDate(tc.dateStr) - - if tc.expectError { - assert.Error(t, err) - } else { - assert.NoError(t, err) - assert.Equal(t, tc.expectedDate, date) - } - } -} - -func TestDetectUserReportFormat(t *testing.T) { - // New format: contains Client_Type - assert.True(t, detectUserReportFormat([]string{"UserId", "Date", "Client_Type", "Credits_Used"})) - // New format: contains Credits_Used - assert.True(t, detectUserReportFormat([]string{"UserId", "Date", "Credits_Used", "Total_Messages"})) - // Old format: code-level metrics - assert.False(t, detectUserReportFormat([]string{"UserId", "Date", "Chat_AICodeLines", "Inline_AICodeLines"})) - // Old format: no new-format indicators - assert.False(t, detectUserReportFormat([]string{"UserId", "Date", "CodeReview_FindingsCount"})) - // Empty headers - assert.False(t, detectUserReportFormat([]string{})) - // Whitespace-padded header still detected - assert.True(t, detectUserReportFormat([]string{"UserId", " Client_Type ", "Date"})) -} - -func TestCreateUserReportData_Success(t *testing.T) { - headers := []string{ - "UserId", "Date", "Client_Type", "Subscription_Tier", "ProfileId", - "Chat_Conversations", "Credits_Used", "Overage_Cap", "Overage_Credits_Used", - "Overage_Enabled", "Total_Messages", - } - record := []string{ - "user-abc", "2026-01-15", "KIRO_IDE", "Pro", "profile-xyz", - "12", "45.5", "100.0", "5.25", - "true", "87", - } - fileMeta := &models.QDevS3FileMeta{ - ConnectionId: 1, - ScopeId: "scope-1", - } - - mockIdentityClient := &MockIdentityClient{} - mockIdentityClient.On("ResolveUserDisplayName", "user-abc").Return("Alice Bob", nil) - - mockLogger := &MockLogger{} - mockLogger.On("Debug", mock.Anything, mock.Anything).Return() - - report, err := createUserReportData(mockLogger, headers, record, fileMeta, mockIdentityClient) - - assert.NoError(t, err) - assert.NotNil(t, report) - assert.Equal(t, "user-abc", report.UserId) - assert.Equal(t, "Alice Bob", report.DisplayName) - assert.Equal(t, uint64(1), report.ConnectionId) - assert.Equal(t, "scope-1", report.ScopeId) - assert.Equal(t, "KIRO_IDE", report.ClientType) - assert.Equal(t, "Pro", report.SubscriptionTier) - assert.Equal(t, "profile-xyz", report.ProfileId) - assert.Equal(t, 12, report.ChatConversations) - assert.Equal(t, 45.5, report.CreditsUsed) - assert.Equal(t, 100.0, report.OverageCap) - assert.Equal(t, 5.25, report.OverageCreditsUsed) - assert.True(t, report.OverageEnabled) - assert.Equal(t, 87, report.TotalMessages) - - expectedDate, _ := time.Parse("2006-01-02", "2026-01-15") - assert.Equal(t, expectedDate, report.Date) - - mockIdentityClient.AssertExpectations(t) -} - -func TestCreateUserReportData_MissingUserId(t *testing.T) { - headers := []string{"Date", "Client_Type", "Credits_Used"} - record := []string{"2026-01-15", "KIRO_IDE", "10.0"} - fileMeta := &models.QDevS3FileMeta{ConnectionId: 1} - - mockLogger := &MockLogger{} - mockLogger.On("Debug", mock.Anything, mock.Anything).Return() - - report, err := createUserReportData(mockLogger, headers, record, fileMeta, nil) - - assert.Error(t, err) - assert.Nil(t, report) - assert.Contains(t, err.Error(), "UserId not found") -} - -func TestCreateUserReportData_MissingDate(t *testing.T) { - headers := []string{"UserId", "Client_Type", "Credits_Used"} - record := []string{"user-abc", "KIRO_IDE", "10.0"} - fileMeta := &models.QDevS3FileMeta{ConnectionId: 1} - - mockLogger := &MockLogger{} - mockLogger.On("Debug", mock.Anything, mock.Anything).Return() - - report, err := createUserReportData(mockLogger, headers, record, fileMeta, nil) - - assert.Error(t, err) - assert.Nil(t, report) - assert.Contains(t, err.Error(), "Date not found") -} - -func TestCreateUserReportData_OverageDisabled(t *testing.T) { - headers := []string{"UserId", "Date", "Overage_Enabled", "Credits_Used"} - record := []string{"user-abc", "2026-01-15", "false", "10.0"} - fileMeta := &models.QDevS3FileMeta{ConnectionId: 1} - - mockLogger := &MockLogger{} - mockLogger.On("Debug", mock.Anything, mock.Anything).Return() - - report, err := createUserReportData(mockLogger, headers, record, fileMeta, nil) - - assert.NoError(t, err) - assert.False(t, report.OverageEnabled) -} - -func TestCreateUserReportData_InvalidNumericValues(t *testing.T) { - headers := []string{"UserId", "Date", "Credits_Used", "Chat_Conversations", "Total_Messages"} - record := []string{"user-abc", "2026-01-15", "not-a-float", "not-an-int", ""} - fileMeta := &models.QDevS3FileMeta{ConnectionId: 1} - - mockLogger := &MockLogger{} - mockLogger.On("Debug", mock.Anything, mock.Anything).Return() - - report, err := createUserReportData(mockLogger, headers, record, fileMeta, nil) - - assert.NoError(t, err) - assert.Equal(t, float64(0), report.CreditsUsed) - assert.Equal(t, 0, report.ChatConversations) - assert.Equal(t, 0, report.TotalMessages) -} - -func TestParseFloat(t *testing.T) { - fieldMap := map[string]string{ - "ValidFloat": "3.14", - "ZeroFloat": "0", - "NegativeFloat": "-2.5", - "IntegerValue": "42", - "InvalidFloat": "not-a-number", - "EmptyString": "", - "Whitespace": " 1.5 ", - } - - assert.Equal(t, 3.14, parseFloat(fieldMap, "ValidFloat")) - assert.Equal(t, float64(0), parseFloat(fieldMap, "ZeroFloat")) - assert.Equal(t, -2.5, parseFloat(fieldMap, "NegativeFloat")) - assert.Equal(t, float64(42), parseFloat(fieldMap, "IntegerValue")) - assert.Equal(t, float64(0), parseFloat(fieldMap, "InvalidFloat")) - assert.Equal(t, float64(0), parseFloat(fieldMap, "EmptyString")) - assert.Equal(t, 1.5, parseFloat(fieldMap, "Whitespace")) - assert.Equal(t, float64(0), parseFloat(fieldMap, "NonExistentField")) -} - -func TestParseBool(t *testing.T) { - fieldMap := map[string]string{ - "TrueValue": "true", - "TrueUpper": "True", - "TrueOne": "1", - "TrueYes": "yes", - "FalseValue": "false", - "FalseZero": "0", - "EmptyString": "", - "InvalidBool": "maybe", - "WhitespaceVal": " true ", - } - - assert.True(t, parseBool(fieldMap, "TrueValue")) - assert.True(t, parseBool(fieldMap, "TrueUpper")) - assert.True(t, parseBool(fieldMap, "TrueOne")) - assert.True(t, parseBool(fieldMap, "TrueYes")) - assert.False(t, parseBool(fieldMap, "FalseValue")) - assert.False(t, parseBool(fieldMap, "FalseZero")) - assert.False(t, parseBool(fieldMap, "EmptyString")) - assert.False(t, parseBool(fieldMap, "InvalidBool")) - assert.True(t, parseBool(fieldMap, "WhitespaceVal")) - assert.False(t, parseBool(fieldMap, "NonExistentField")) -} - -func TestParseInt(t *testing.T) { - fieldMap := map[string]string{ - "ValidInt": "42", - "ZeroInt": "0", - "NegativeInt": "-10", - "InvalidInt": "not-a-number", - "EmptyString": "", - } - - assert.Equal(t, 42, parseInt(fieldMap, "ValidInt")) - assert.Equal(t, 0, parseInt(fieldMap, "ZeroInt")) - assert.Equal(t, -10, parseInt(fieldMap, "NegativeInt")) - assert.Equal(t, 0, parseInt(fieldMap, "InvalidInt")) - assert.Equal(t, 0, parseInt(fieldMap, "EmptyString")) - assert.Equal(t, 0, parseInt(fieldMap, "NonExistentField")) -} diff --git a/backend/plugins/q_dev/tasks/s3_file_collector.go b/backend/plugins/q_dev/tasks/s3_file_collector.go deleted file mode 100644 index 1ab4f8f0aa7..00000000000 --- a/backend/plugins/q_dev/tasks/s3_file_collector.go +++ /dev/null @@ -1,117 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package tasks - -import ( - "strings" - - "github.com/apache/incubator-devlake/core/dal" - "github.com/apache/incubator-devlake/core/errors" - "github.com/apache/incubator-devlake/core/plugin" - "github.com/apache/incubator-devlake/plugins/q_dev/models" - "github.com/aws/aws-sdk-go/aws" - "github.com/aws/aws-sdk-go/service/s3" -) - -var _ plugin.SubTaskEntryPoint = CollectQDevS3Files - -// CollectQDevS3Files collects S3 file metadata -func CollectQDevS3Files(taskCtx plugin.SubTaskContext) errors.Error { - data := taskCtx.GetData().(*QDevTaskData) - db := taskCtx.GetDal() - - taskCtx.SetProgress(0, -1) - - for _, rawPrefix := range data.S3Prefixes { - prefix := rawPrefix - if prefix != "" && !strings.HasSuffix(prefix, "/") { - prefix = prefix + "/" - } - - taskCtx.GetLogger().Info("Scanning S3 prefix: %s", prefix) - - var continuationToken *string - for { - input := &s3.ListObjectsV2Input{ - Bucket: aws.String(data.S3Client.Bucket), - Prefix: aws.String(prefix), - ContinuationToken: continuationToken, - } - - result, err := data.S3Client.S3.ListObjectsV2(input) - if err != nil { - return errors.Convert(err) - } - - for _, object := range result.Contents { - // Only process CSV and JSON.gz files - if !strings.HasSuffix(*object.Key, ".csv") && !strings.HasSuffix(*object.Key, ".json.gz") { - taskCtx.GetLogger().Debug("Skipping unsupported file: %s", *object.Key) - continue - } - - // Check if this file already exists in our database - existingFile := &models.QDevS3FileMeta{} - err = db.First(existingFile, dal.Where("connection_id = ? AND s3_path = ?", - data.Options.ConnectionId, *object.Key)) - - if err == nil { - if existingFile.Processed { - taskCtx.GetLogger().Debug("Skipping already processed file: %s", *object.Key) - continue - } - taskCtx.GetLogger().Debug("Found existing unprocessed file: %s", *object.Key) - continue - } else if !db.IsErrorNotFound(err) { - return errors.Default.Wrap(err, "failed to query existing file metadata") - } - - fileMeta := &models.QDevS3FileMeta{ - ConnectionId: data.Options.ConnectionId, - FileName: *object.Key, - S3Path: *object.Key, - ScopeId: data.Options.ScopeId, - Processed: false, - } - - err = db.Create(fileMeta) - if err != nil { - return errors.Default.Wrap(err, "failed to create file metadata") - } - - taskCtx.IncProgress(1) - } - - if !*result.IsTruncated { - break - } - - continuationToken = result.NextContinuationToken - } - } - - return nil -} - -var CollectQDevS3FilesMeta = plugin.SubTaskMeta{ - Name: "collectQDevS3Files", - EntryPoint: CollectQDevS3Files, - EnabledByDefault: true, - Description: "Collect S3 file metadata from AWS S3 bucket", - DomainTypes: []string{plugin.DOMAIN_TYPE_CROSS}, -} diff --git a/backend/plugins/q_dev/tasks/s3_logging_extractor.go b/backend/plugins/q_dev/tasks/s3_logging_extractor.go deleted file mode 100644 index 3fa771789a1..00000000000 --- a/backend/plugins/q_dev/tasks/s3_logging_extractor.go +++ /dev/null @@ -1,450 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package tasks - -import ( - "compress/gzip" - "encoding/json" - "path/filepath" - "strings" - "sync" - "time" - - "github.com/apache/incubator-devlake/core/dal" - "github.com/apache/incubator-devlake/core/errors" - "github.com/apache/incubator-devlake/core/plugin" - "github.com/apache/incubator-devlake/plugins/q_dev/models" - "github.com/aws/aws-sdk-go/aws" - "github.com/aws/aws-sdk-go/service/s3" -) - -var _ plugin.SubTaskEntryPoint = ExtractQDevLoggingData - -const ( - loggingBatchSize = 50 // number of files to process per DB transaction - s3DownloadWorkers = 10 // parallel S3 download goroutines - s3DownloadChanSize = 20 // buffered channel size for download results -) - -// downloadResult holds the parsed records from one S3 file -type downloadResult struct { - FileMeta *models.QDevS3FileMeta - ChatLogs []*models.QDevChatLog - CompLogs []*models.QDevCompletionLog - Err error -} - -// ExtractQDevLoggingData extracts logging data from S3 JSON.gz files -func ExtractQDevLoggingData(taskCtx plugin.SubTaskContext) errors.Error { - data := taskCtx.GetData().(*QDevTaskData) - db := taskCtx.GetDal() - - cursor, err := db.Cursor( - dal.From(&models.QDevS3FileMeta{}), - dal.Where("connection_id = ? AND processed = ? AND file_name LIKE ?", - data.Options.ConnectionId, false, "%.json.gz"), - ) - if err != nil { - return errors.Default.Wrap(err, "failed to get logging file metadata cursor") - } - defer cursor.Close() - - // Collect all file metas first - var fileMetas []*models.QDevS3FileMeta - for cursor.Next() { - fm := &models.QDevS3FileMeta{} - if err := db.Fetch(cursor, fm); err != nil { - return errors.Default.Wrap(err, "failed to fetch file metadata") - } - fileMetas = append(fileMetas, fm) - } - - if len(fileMetas) == 0 { - return nil - } - - taskCtx.SetProgress(0, len(fileMetas)) - taskCtx.GetLogger().Info("Processing %d logging files with %d workers", len(fileMetas), s3DownloadWorkers) - - // Display name cache to avoid repeated IAM calls - displayNameCache := &sync.Map{} - - // Process in batches - for batchStart := 0; batchStart < len(fileMetas); batchStart += loggingBatchSize { - batchEnd := batchStart + loggingBatchSize - if batchEnd > len(fileMetas) { - batchEnd = len(fileMetas) - } - batch := fileMetas[batchStart:batchEnd] - - // Parallel download and parse - results := parallelDownloadAndParse(taskCtx, data, batch, displayNameCache) - - // Check for download errors - for _, r := range results { - if r.Err != nil { - return errors.Default.Wrap(errors.Convert(r.Err), - "failed to download/parse "+r.FileMeta.FileName) - } - } - - // Batch write to DB in a single transaction - tx := db.Begin() - var txErr errors.Error - for _, r := range results { - for _, chatLog := range r.ChatLogs { - if txErr = tx.CreateOrUpdate(chatLog); txErr != nil { - break - } - } - if txErr != nil { - break - } - for _, compLog := range r.CompLogs { - if txErr = tx.CreateOrUpdate(compLog); txErr != nil { - break - } - } - if txErr != nil { - break - } - r.FileMeta.Processed = true - now := time.Now() - r.FileMeta.ProcessedTime = &now - if txErr = tx.Update(r.FileMeta); txErr != nil { - break - } - } - if txErr != nil { - if rbErr := tx.Rollback(); rbErr != nil { - taskCtx.GetLogger().Error(rbErr, "failed to rollback transaction") - } - return errors.Default.Wrap(txErr, "failed to write logging batch") - } - if err := tx.Commit(); err != nil { - return errors.Default.Wrap(err, "failed to commit batch") - } - - taskCtx.IncProgress(len(batch)) - } - - return nil -} - -// parallelDownloadAndParse downloads and parses S3 files concurrently -func parallelDownloadAndParse( - taskCtx plugin.SubTaskContext, - data *QDevTaskData, - fileMetas []*models.QDevS3FileMeta, - displayNameCache *sync.Map, -) []downloadResult { - results := make([]downloadResult, len(fileMetas)) - jobs := make(chan int, s3DownloadChanSize) - var wg sync.WaitGroup - - // Start workers - for w := 0; w < s3DownloadWorkers; w++ { - wg.Add(1) - go func() { - defer wg.Done() - for idx := range jobs { - fm := fileMetas[idx] - result := downloadAndParseFile(taskCtx, data, fm, displayNameCache) - results[idx] = result - } - }() - } - - // Send jobs - for i := range fileMetas { - jobs <- i - } - close(jobs) - wg.Wait() - - return results -} - -// downloadAndParseFile downloads one S3 file and parses it into model records -func downloadAndParseFile( - taskCtx plugin.SubTaskContext, - data *QDevTaskData, - fileMeta *models.QDevS3FileMeta, - displayNameCache *sync.Map, -) downloadResult { - result := downloadResult{FileMeta: fileMeta} - - getResult, err := data.S3Client.S3.GetObject(&s3.GetObjectInput{ - Bucket: aws.String(data.S3Client.Bucket), - Key: aws.String(fileMeta.S3Path), - }) - if err != nil { - result.Err = err - return result - } - defer getResult.Body.Close() - - gzReader, err := gzip.NewReader(getResult.Body) - if err != nil { - result.Err = err - return result - } - defer gzReader.Close() - - var logFile loggingFile - if err := json.NewDecoder(gzReader).Decode(&logFile); err != nil { - result.Err = err - return result - } - - isChatLog := strings.Contains(fileMeta.S3Path, "GenerateAssistantResponse") - - for _, rawRecord := range logFile.Records { - if isChatLog { - chatLog, err := parseChatRecord(rawRecord, fileMeta, data.IdentityClient, displayNameCache) - if err != nil { - result.Err = err - return result - } - if chatLog != nil { - result.ChatLogs = append(result.ChatLogs, chatLog) - } - } else { - compLog, err := parseCompletionRecord(rawRecord, fileMeta, data.IdentityClient, displayNameCache) - if err != nil { - result.Err = err - return result - } - if compLog != nil { - result.CompLogs = append(result.CompLogs, compLog) - } - } - } - - return result -} - -// cachedResolveDisplayName resolves display name with caching -func cachedResolveDisplayName(userId string, identityClient UserDisplayNameResolver, cache *sync.Map) string { - if v, ok := cache.Load(userId); ok { - return v.(string) - } - if identityClient == nil { - cache.Store(userId, userId) - return userId - } - displayName, err := identityClient.ResolveUserDisplayName(userId) - if err != nil || displayName == "" { - cache.Store(userId, userId) - return userId - } - cache.Store(userId, displayName) - return displayName -} - -// JSON structures for logging data - -type loggingFile struct { - Records []json.RawMessage `json:"records"` -} - -type chatLogRecord struct { - Request *chatLogRequest `json:"generateAssistantResponseEventRequest"` - Response *chatLogResponse `json:"generateAssistantResponseEventResponse"` -} - -type chatLogRequest struct { - UserID string `json:"userId"` - Timestamp string `json:"timeStamp"` - ChatTriggerType string `json:"chatTriggerType"` - CustomizationArn *string `json:"customizationArn"` - ModelID string `json:"modelId"` - Prompt string `json:"prompt"` -} - -type chatLogResponse struct { - RequestID string `json:"requestId"` - AssistantResponse string `json:"assistantResponse"` - FollowupPrompts string `json:"followupPrompts"` - MessageMetadata struct { - ConversationID *string `json:"conversationId"` - UtteranceID *string `json:"utteranceId"` - } `json:"messageMetadata"` - CodeReferenceEvents []json.RawMessage `json:"codeReferenceEvents"` - SupplementaryWebLinksEvent []json.RawMessage `json:"supplementaryWebLinksEvent"` -} - -type completionLogRecord struct { - Request *completionLogRequest `json:"generateCompletionsEventRequest"` - Response *completionLogResponse `json:"generateCompletionsEventResponse"` -} - -type completionLogRequest struct { - UserID string `json:"userId"` - Timestamp string `json:"timeStamp"` - FileName string `json:"fileName"` - CustomizationArn *string `json:"customizationArn"` - LeftContext string `json:"leftContext"` - RightContext string `json:"rightContext"` -} - -type completionLogResponse struct { - RequestID string `json:"requestId"` - Completions []json.RawMessage `json:"completions"` -} - -func parseChatRecord(raw json.RawMessage, fileMeta *models.QDevS3FileMeta, identityClient UserDisplayNameResolver, cache *sync.Map) (*models.QDevChatLog, error) { - var record chatLogRecord - if err := json.Unmarshal(raw, &record); err != nil { - return nil, err - } - - if record.Request == nil || record.Response == nil { - return nil, nil - } - - ts, err := time.Parse(time.RFC3339Nano, record.Request.Timestamp) - if err != nil { - ts = time.Now() - } - - userId := normalizeUserId(record.Request.UserID) - chatLog := &models.QDevChatLog{ - ConnectionId: fileMeta.ConnectionId, - ScopeId: fileMeta.ScopeId, - RequestId: record.Response.RequestID, - UserId: userId, - DisplayName: cachedResolveDisplayName(userId, identityClient, cache), - Timestamp: ts, - ChatTriggerType: record.Request.ChatTriggerType, - HasCustomization: record.Request.CustomizationArn != nil && *record.Request.CustomizationArn != "", - ModelId: record.Request.ModelID, - PromptLength: len(record.Request.Prompt), - ResponseLength: len(record.Response.AssistantResponse), - } - - // Parse structured info from prompt - prompt := record.Request.Prompt - chatLog.OpenFileCount = countOpenFiles(prompt) - chatLog.ActiveFileName, chatLog.ActiveFileExtension = parseActiveFile(prompt) - chatLog.HasSteering = strings.Contains(prompt, ".kiro/steering") - chatLog.IsSpecMode = strings.Contains(prompt, "implicit-rules") - - if record.Response.MessageMetadata.ConversationID != nil { - chatLog.ConversationId = *record.Response.MessageMetadata.ConversationID - } - if record.Response.MessageMetadata.UtteranceID != nil { - chatLog.UtteranceId = *record.Response.MessageMetadata.UtteranceID - } - - // New fields from docs: codeReferenceEvents, supplementaryWebLinksEvent, followupPrompts - chatLog.CodeReferenceCount = len(record.Response.CodeReferenceEvents) - chatLog.WebLinkCount = len(record.Response.SupplementaryWebLinksEvent) - chatLog.HasFollowupPrompts = record.Response.FollowupPrompts != "" - - return chatLog, nil -} - -// countOpenFiles counts tags within block -func countOpenFiles(prompt string) int { - start := strings.Index(prompt, "") - if start == -1 { - return 0 - } - end := strings.Index(prompt, "") - if end == -1 { - return 0 - } - block := prompt[start:end] - return strings.Count(block, "") - if start == -1 { - return "", "" - } - end := strings.Index(prompt[start:], "") - if end == -1 { - return "", "" - } - block := prompt[start : start+end] - nameStart := strings.Index(block, "name=\"") - if nameStart == -1 { - return "", "" - } - nameStart += len("name=\"") - nameEnd := strings.Index(block[nameStart:], "\"") - if nameEnd == -1 { - return "", "" - } - fileName := block[nameStart : nameStart+nameEnd] - ext := filepath.Ext(fileName) - return fileName, ext -} - -func parseCompletionRecord(raw json.RawMessage, fileMeta *models.QDevS3FileMeta, identityClient UserDisplayNameResolver, cache *sync.Map) (*models.QDevCompletionLog, error) { - var record completionLogRecord - if err := json.Unmarshal(raw, &record); err != nil { - return nil, err - } - - if record.Request == nil || record.Response == nil { - return nil, nil - } - - ts, err := time.Parse(time.RFC3339Nano, record.Request.Timestamp) - if err != nil { - ts = time.Now() - } - - userId := normalizeUserId(record.Request.UserID) - return &models.QDevCompletionLog{ - ConnectionId: fileMeta.ConnectionId, - ScopeId: fileMeta.ScopeId, - RequestId: record.Response.RequestID, - UserId: userId, - DisplayName: cachedResolveDisplayName(userId, identityClient, cache), - Timestamp: ts, - FileName: record.Request.FileName, - FileExtension: filepath.Ext(record.Request.FileName), - HasCustomization: record.Request.CustomizationArn != nil && *record.Request.CustomizationArn != "", - CompletionsCount: len(record.Response.Completions), - LeftContextLength: len(record.Request.LeftContext), - RightContextLength: len(record.Request.RightContext), - }, nil -} - -// normalizeUserId strips the "d-{directoryId}." prefix from Identity Center user IDs -// so that logging user IDs match the short UUID format used in user-report CSVs. -func normalizeUserId(userId string) string { - if idx := strings.LastIndex(userId, "."); idx != -1 && strings.HasPrefix(userId, "d-") { - return userId[idx+1:] - } - return userId -} - -var ExtractQDevLoggingDataMeta = plugin.SubTaskMeta{ - Name: "extractQDevLoggingData", - EntryPoint: ExtractQDevLoggingData, - EnabledByDefault: true, - Description: "Extract logging data from S3 JSON.gz files (chat and completion events)", - DomainTypes: []string{plugin.DOMAIN_TYPE_CROSS}, - Dependencies: []*plugin.SubTaskMeta{&CollectQDevS3FilesMeta}, -} diff --git a/backend/plugins/q_dev/tasks/task_data.go b/backend/plugins/q_dev/tasks/task_data.go deleted file mode 100644 index 3fd3c65848d..00000000000 --- a/backend/plugins/q_dev/tasks/task_data.go +++ /dev/null @@ -1,52 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package tasks - -import ( - "github.com/aws/aws-sdk-go/service/s3" -) - -type QDevApiParams struct { - ConnectionId uint64 `json:"connectionId"` -} - -type QDevOptions struct { - ConnectionId uint64 `json:"connectionId"` - S3Prefix string `json:"s3Prefix"` - ScopeId string `json:"scopeId"` - AccountId string `json:"accountId"` - BasePath string `json:"basePath"` - Year int `json:"year"` - Month *int `json:"month"` -} - -type QDevTaskData struct { - Options *QDevOptions - S3Client *QDevS3Client - IdentityClient *QDevIdentityClient - S3Prefixes []string -} - -type QDevS3Client struct { - S3 *s3.S3 - Bucket string -} - -func (client *QDevS3Client) Close() { - // S3客户端不需要特别关闭操作 -} diff --git a/backend/plugins/q_dev/tasks/task_data_test.go b/backend/plugins/q_dev/tasks/task_data_test.go deleted file mode 100644 index 757f27428de..00000000000 --- a/backend/plugins/q_dev/tasks/task_data_test.go +++ /dev/null @@ -1,142 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package tasks - -import ( - "testing" - - "github.com/aws/aws-sdk-go/service/s3" - "github.com/stretchr/testify/assert" -) - -func TestQDevTaskData_WithIdentityClient(t *testing.T) { - taskData := &QDevTaskData{ - Options: &QDevOptions{ - ConnectionId: 1, - S3Prefix: "test-prefix/", - }, - S3Client: &QDevS3Client{ - S3: &s3.S3{}, - Bucket: "test-bucket", - }, - IdentityClient: &QDevIdentityClient{ - StoreId: "d-1234567890", - Region: "us-west-2", - }, - S3Prefixes: []string{"test-prefix/"}, - } - - assert.NotNil(t, taskData.IdentityClient) - assert.Equal(t, "d-1234567890", taskData.IdentityClient.StoreId) - assert.Equal(t, "us-west-2", taskData.IdentityClient.Region) - assert.NotNil(t, taskData.S3Client) - assert.NotNil(t, taskData.Options) - assert.Equal(t, []string{"test-prefix/"}, taskData.S3Prefixes) -} - -func TestQDevTaskData_WithoutIdentityClient(t *testing.T) { - taskData := &QDevTaskData{ - Options: &QDevOptions{ - ConnectionId: 1, - S3Prefix: "test-prefix/", - }, - S3Client: &QDevS3Client{ - S3: &s3.S3{}, - Bucket: "test-bucket", - }, - IdentityClient: nil, // No identity client configured - } - - assert.Nil(t, taskData.IdentityClient) - assert.NotNil(t, taskData.S3Client) - assert.NotNil(t, taskData.Options) - assert.Equal(t, uint64(1), taskData.Options.ConnectionId) - assert.Equal(t, "test-prefix/", taskData.Options.S3Prefix) -} - -func TestQDevTaskData_AllFields(t *testing.T) { - month := 3 - options := &QDevOptions{ - ConnectionId: 123, - S3Prefix: "data/q-dev/", - AccountId: "034362076319", - BasePath: "user-report", - Year: 2026, - Month: &month, - } - - s3Client := &QDevS3Client{ - S3: &s3.S3{}, - Bucket: "my-data-bucket", - } - - identityClient := &QDevIdentityClient{ - StoreId: "d-9876543210", - Region: "eu-west-1", - } - - taskData := &QDevTaskData{ - Options: options, - S3Client: s3Client, - IdentityClient: identityClient, - S3Prefixes: []string{ - "user-report/AWSLogs/034362076319/KiroLogs/by_user_analytic/us-east-1/2026/03", - "user-report/AWSLogs/034362076319/KiroLogs/user_report/us-east-1/2026/03", - }, - } - - // Verify all fields are properly set - assert.Equal(t, options, taskData.Options) - assert.Equal(t, s3Client, taskData.S3Client) - assert.Equal(t, identityClient, taskData.IdentityClient) - - // Verify nested field access - assert.Equal(t, uint64(123), taskData.Options.ConnectionId) - assert.Equal(t, "data/q-dev/", taskData.Options.S3Prefix) - assert.Equal(t, "034362076319", taskData.Options.AccountId) - assert.Equal(t, "user-report", taskData.Options.BasePath) - assert.Equal(t, 2026, taskData.Options.Year) - assert.Equal(t, &month, taskData.Options.Month) - assert.Equal(t, "my-data-bucket", taskData.S3Client.Bucket) - assert.Equal(t, "d-9876543210", taskData.IdentityClient.StoreId) - assert.Equal(t, "eu-west-1", taskData.IdentityClient.Region) - assert.Len(t, taskData.S3Prefixes, 2) -} - -func TestQDevTaskData_EmptyStruct(t *testing.T) { - taskData := &QDevTaskData{} - - assert.Nil(t, taskData.Options) - assert.Nil(t, taskData.S3Client) - assert.Nil(t, taskData.IdentityClient) -} - -func TestQDevTaskData_PartialInitialization(t *testing.T) { - taskData := &QDevTaskData{ - Options: &QDevOptions{ - ConnectionId: 456, - }, - // S3Client and IdentityClient intentionally nil - } - - assert.NotNil(t, taskData.Options) - assert.Equal(t, uint64(456), taskData.Options.ConnectionId) - assert.Equal(t, "", taskData.Options.S3Prefix) // Default empty string - assert.Nil(t, taskData.S3Client) - assert.Nil(t, taskData.IdentityClient) -} diff --git a/backend/plugins/schema_e2e/migration_schema_test.go b/backend/plugins/schema_e2e/migration_schema_test.go index 5cdf115f960..80b3975200c 100644 --- a/backend/plugins/schema_e2e/migration_schema_test.go +++ b/backend/plugins/schema_e2e/migration_schema_test.go @@ -76,7 +76,6 @@ import ( opsgenie "github.com/apache/incubator-devlake/plugins/opsgenie/impl" org "github.com/apache/incubator-devlake/plugins/org/impl" pagerduty "github.com/apache/incubator-devlake/plugins/pagerduty/impl" - q_dev "github.com/apache/incubator-devlake/plugins/q_dev/impl" refdiff "github.com/apache/incubator-devlake/plugins/refdiff/impl" rootly "github.com/apache/incubator-devlake/plugins/rootly/impl" slack "github.com/apache/incubator-devlake/plugins/slack/impl" @@ -128,7 +127,6 @@ func allGoPlugins() []plugin.PluginMeta { opsgenie.Opsgenie{}, org.Org{}, pagerduty.PagerDuty{}, - q_dev.QDev{}, refdiff.RefDiff{}, rootly.Rootly{}, slack.Slack{}, diff --git a/backend/plugins/table_info_test.go b/backend/plugins/table_info_test.go index afe5fa1a31d..f4182fe9d89 100644 --- a/backend/plugins/table_info_test.go +++ b/backend/plugins/table_info_test.go @@ -52,7 +52,6 @@ import ( opsgenie "github.com/apache/incubator-devlake/plugins/opsgenie/impl" org "github.com/apache/incubator-devlake/plugins/org/impl" pagerduty "github.com/apache/incubator-devlake/plugins/pagerduty/impl" - q_dev "github.com/apache/incubator-devlake/plugins/q_dev/impl" refdiff "github.com/apache/incubator-devlake/plugins/refdiff/impl" rootly "github.com/apache/incubator-devlake/plugins/rootly/impl" slack "github.com/apache/incubator-devlake/plugins/slack/impl" @@ -115,7 +114,6 @@ func Test_GetPluginTablesInfo(t *testing.T) { checker.FeedIn("opsgenie/models", opsgenie.Opsgenie{}.GetTablesInfo) checker.FeedIn("linker/models", linker.Linker{}.GetTablesInfo) checker.FeedIn("issue_trace/models", issueTrace.IssueTrace{}.GetTablesInfo) - checker.FeedIn("q_dev/models", q_dev.QDev{}.GetTablesInfo) checker.FeedIn("kiro/models", kiro.Kiro{}.GetTablesInfo) checker.FeedIn("gh-copilot/models", copilot.GhCopilot{}.GetTablesInfo) err := checker.Verify() diff --git a/config-ui/src/plugins/register/index.ts b/config-ui/src/plugins/register/index.ts index 441cda5cb2f..64a719523e4 100644 --- a/config-ui/src/plugins/register/index.ts +++ b/config-ui/src/plugins/register/index.ts @@ -42,7 +42,6 @@ import { TAPDConfig } from './tapd'; import { WebhookConfig } from './webhook'; import { ZenTaoConfig } from './zentao'; import { OpsgenieConfig } from './opsgenie'; -import { QDevConfig } from './q-dev'; import { TeambitionConfig } from './teambition'; import { TestmoConfig } from './testmo'; import { SlackConfig } from './slack/config'; @@ -71,7 +70,6 @@ export const pluginConfigs: IPluginConfig[] = [ RootlyConfig, SlackConfig, TempoConfig, - QDevConfig, SonarQubeConfig, TAPDConfig, TestmoConfig, diff --git a/config-ui/src/plugins/register/q-dev/assets/icon.svg b/config-ui/src/plugins/register/q-dev/assets/icon.svg deleted file mode 100644 index 503114f140a..00000000000 --- a/config-ui/src/plugins/register/q-dev/assets/icon.svg +++ /dev/null @@ -1,62 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/config-ui/src/plugins/register/q-dev/config.tsx b/config-ui/src/plugins/register/q-dev/config.tsx deleted file mode 100644 index 6516e74e044..00000000000 --- a/config-ui/src/plugins/register/q-dev/config.tsx +++ /dev/null @@ -1,96 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -import { IPluginConfig } from '@/types'; - -import Icon from './assets/icon.svg?react'; -import { AwsCredentials, IdentityCenterConfig, S3Config } from './connection-fields'; -import { QDevDataScope } from './data-scope'; - -export const QDevConfig: IPluginConfig = { - plugin: 'q_dev', - name: 'Q Developer', - icon: ({ color }) => , - sort: 12, - connection: { - docLink: 'https://devlake.apache.org/docs/UserManual/plugins/qdev', - initialValues: { - name: '', - authType: 'access_key', - accessKeyId: '', - secretAccessKey: '', - region: 'us-east-1', - bucket: '', - identityStoreId: '', - identityStoreRegion: '', - rateLimitPerHour: 20000, - }, - fields: [ - 'name', - ({ type, initialValues, values, setValues, setErrors }: any) => ( - - ), - ({ initialValues, values, setValues, setErrors }: any) => ( - - ), - ({ initialValues, values, setValues, setErrors }: any) => ( - - ), - 'proxy', - { - key: 'rateLimitPerHour', - subLabel: 'Set a fixed hourly rate limit if you need to throttle collection speed (default 20,000).', - defaultValue: 20000, - }, - ], - }, - dataScope: { - title: 'S3 Prefixes', - render: ({ connectionId, disabledItems, selectedItems, onChangeSelectedItems }) => ( - - ), - }, - scopeConfig: { - entities: ['CROSS'], - transformation: {}, - }, -}; diff --git a/config-ui/src/plugins/register/q-dev/connection-fields/aws-credentials.tsx b/config-ui/src/plugins/register/q-dev/connection-fields/aws-credentials.tsx deleted file mode 100644 index ea7c9dd40b5..00000000000 --- a/config-ui/src/plugins/register/q-dev/connection-fields/aws-credentials.tsx +++ /dev/null @@ -1,235 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -import { ChangeEvent, useEffect, useMemo, useRef } from 'react'; -import { Input, Radio } from 'antd'; - -import { Block } from '@/components'; - -interface Props { - type: 'create' | 'update'; - initialValues: any; - values: any; - setValues: (values: any) => void; - setErrors: (errors: any) => void; -} - -const ACCESS_KEY_PATTERN = /^[A-Z0-9]{16,32}$/; -const REGION_PATTERN = /^[a-z]{2}-[a-z]+-\d$/; - -const syncError = ( - key: string, - error: string, - setErrors: (errors: any) => void, - ref: React.MutableRefObject, -) => { - if (ref.current !== error) { - ref.current = error; - setErrors({ [key]: error }); - } -}; - -export const AwsCredentials = ({ type, initialValues, values, setValues, setErrors }: Props) => { - const isUpdate = type === 'update'; - - const authType = values.authType ?? 'access_key'; - const accessKeyId = values.accessKeyId ?? ''; - const secretAccessKey = values.secretAccessKey ?? ''; - const region = values.region ?? ''; - - const isAccessKeyAuth = authType === 'access_key'; - - useEffect(() => { - if (values.authType === undefined) { - setValues({ authType: initialValues.authType ?? 'access_key' }); - } - }, [initialValues.authType, values.authType, setValues]); - - useEffect(() => { - if (values.accessKeyId === undefined) { - setValues({ accessKeyId: initialValues.accessKeyId ?? '' }); - } - }, [initialValues.accessKeyId, values.accessKeyId, setValues]); - - useEffect(() => { - if (values.secretAccessKey === undefined) { - setValues({ secretAccessKey: type === 'create' ? (initialValues.secretAccessKey ?? '') : '' }); - } - }, [type, initialValues.secretAccessKey, values.secretAccessKey, setValues]); - - useEffect(() => { - if (values.region === undefined) { - setValues({ region: initialValues.region ?? 'us-east-1' }); - } - }, [initialValues.region, values.region, setValues]); - - const accessKeyError = useMemo(() => { - if (!isAccessKeyAuth) return ''; // Not required for IAM role auth - if (!accessKeyId) { - return isUpdate ? '' : 'AWS Access Key ID is required'; - } - if (!ACCESS_KEY_PATTERN.test(accessKeyId)) { - return 'AWS Access Key ID must contain 16-32 uppercase letters or digits'; - } - return ''; - }, [accessKeyId, isUpdate, isAccessKeyAuth]); - - const secretKeyError = useMemo(() => { - if (!isAccessKeyAuth) return ''; // Not required for IAM role auth - if (!secretAccessKey) { - return isUpdate ? '' : 'AWS Secret Access Key is required'; - } - if (secretAccessKey && secretAccessKey.length < 40) { - return 'AWS Secret Access Key looks too short'; - } - return ''; - }, [secretAccessKey, isUpdate, isAccessKeyAuth]); - - const regionError = useMemo(() => { - if (!region) { - return 'AWS Region is required'; - } - if (!REGION_PATTERN.test(region)) { - return 'AWS Region should look like us-east-1'; - } - return ''; - }, [region]); - - const accessKeyErrorRef = useRef(undefined); - const secretKeyErrorRef = useRef(undefined); - const regionErrorRef = useRef(undefined); - - useEffect(() => { - syncError('accessKeyId', accessKeyError, setErrors, accessKeyErrorRef); - }, [accessKeyError, setErrors]); - - useEffect(() => { - syncError('secretAccessKey', secretKeyError, setErrors, secretKeyErrorRef); - }, [secretKeyError, setErrors]); - - useEffect(() => { - syncError('region', regionError, setErrors, regionErrorRef); - }, [regionError, setErrors]); - - const handleAccessKeyChange = (e: ChangeEvent) => { - setValues({ accessKeyId: e.target.value.trim() }); - }; - - const handleSecretKeyChange = (e: ChangeEvent) => { - setValues({ secretAccessKey: e.target.value.trim() }); - }; - - const handleRegionChange = (e: ChangeEvent) => { - setValues({ region: e.target.value.trim() }); - }; - - const handleAuthTypeChange = (e: any) => { - const newAuthType = e.target.value; - setValues({ authType: newAuthType }); - - // Clear access key fields when switching to IAM role - if (newAuthType === 'iam_role') { - setValues({ - authType: newAuthType, - accessKeyId: '', - secretAccessKey: '', - }); - } - }; - - return ( - <> - - - Access Key & Secret - IAM Role (for EC2/ECS/Lambda) - - - - {isAccessKeyAuth && ( - <> - - - {accessKeyError && ( -
{accessKeyError}
- )} - - - - - {secretKeyError && ( -
{secretKeyError}
- )} - - - )} - - {!isAccessKeyAuth && ( - -
-

- Make sure the IAM role has the necessary S3 permissions to access your bucket. No additional credentials - are required when using IAM role authentication. -

-
- - )} - - - - {regionError &&
{regionError}
} - - - ); -}; diff --git a/config-ui/src/plugins/register/q-dev/connection-fields/connection-test.tsx b/config-ui/src/plugins/register/q-dev/connection-fields/connection-test.tsx deleted file mode 100644 index 5157338fc45..00000000000 --- a/config-ui/src/plugins/register/q-dev/connection-fields/connection-test.tsx +++ /dev/null @@ -1,190 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -import { useState } from 'react'; -import { Button, Alert, Space } from 'antd'; -import { CheckCircleOutlined, ExclamationCircleOutlined, LoadingOutlined } from '@ant-design/icons'; - -import API from '@/api'; -import { operator } from '@/utils'; - -interface Props { - plugin: string; - connectionId?: ID; - values: any; - initialValues: any; - disabled?: boolean; -} - -interface TestResult { - success: boolean; - message: string; - details?: { - s3Access?: boolean; - identityCenterAccess?: boolean; - }; -} - -export const QDevConnectionTest = ({ plugin, connectionId, values, initialValues, disabled }: Props) => { - const [testing, setTesting] = useState(false); - const [testResult, setTestResult] = useState(null); - - const handleTest = async () => { - setTesting(true); - setTestResult(null); - - try { - const [success, result] = await operator( - () => { - if (connectionId) { - // Test existing connection with only changed values - return API.connection.test(plugin, connectionId, { - authType: values.authType !== initialValues.authType ? values.authType : undefined, - accessKeyId: values.accessKeyId !== initialValues.accessKeyId ? values.accessKeyId : undefined, - secretAccessKey: - values.secretAccessKey !== initialValues.secretAccessKey ? values.secretAccessKey : undefined, - region: values.region !== initialValues.region ? values.region : undefined, - bucket: values.bucket !== initialValues.bucket ? values.bucket : undefined, - identityStoreId: - values.identityStoreId !== initialValues.identityStoreId ? values.identityStoreId : undefined, - identityStoreRegion: - values.identityStoreRegion !== initialValues.identityStoreRegion - ? values.identityStoreRegion - : undefined, - rateLimitPerHour: - values.rateLimitPerHour !== initialValues.rateLimitPerHour ? values.rateLimitPerHour : undefined, - proxy: values.proxy !== initialValues.proxy ? values.proxy : undefined, - } as any); - } else { - // Test new connection with all values - return API.connection.testOld(plugin, { - authType: values.authType || 'access_key', - accessKeyId: values.accessKeyId || '', - secretAccessKey: values.secretAccessKey || '', - region: values.region || '', - bucket: values.bucket || '', - identityStoreId: values.identityStoreId || '', - identityStoreRegion: values.identityStoreRegion || '', - rateLimitPerHour: values.rateLimitPerHour || 20000, - proxy: values.proxy || '', - endpoint: '', // Not used by Q Developer - token: '', // Not used by Q Developer - } as any); - } - }, - { - setOperating: () => {}, // We handle loading state ourselves - hideToast: true, // We show our own success/error messages - }, - ); - - if (success && result) { - setTestResult({ - success: true, - message: 'Connection test successful! AWS credentials and S3 access verified.', - details: { - s3Access: true, - identityCenterAccess: values.identityStoreId ? true : undefined, - }, - }); - } else { - setTestResult({ - success: false, - message: 'Connection test failed. Please check your configuration.', - }); - } - } catch (error: any) { - let errorMessage = 'Connection test failed. Please check your configuration.'; - - if (error?.response?.data?.message) { - errorMessage = error.response.data.message; - } else if (error?.message) { - errorMessage = error.message; - } - - // Provide more specific error messages based on common issues - if (errorMessage.includes('InvalidAccessKeyId') || errorMessage.includes('SignatureDoesNotMatch')) { - errorMessage = 'Invalid AWS credentials. Please check your Access Key ID and Secret Access Key.'; - } else if (errorMessage.includes('NoSuchBucket')) { - errorMessage = 'S3 bucket not found. Please check the bucket name and region.'; - } else if (errorMessage.includes('AccessDenied')) { - errorMessage = 'Access denied. Please check your AWS permissions for S3 and IAM Identity Center.'; - } else if (errorMessage.includes('InvalidBucketName')) { - errorMessage = 'Invalid S3 bucket name. Please check the bucket name format.'; - } else if (errorMessage.includes('NoCredentialsError')) { - errorMessage = - 'AWS credentials not found. Please provide valid Access Key ID and Secret Access Key, or ensure IAM role is properly configured.'; - } - - setTestResult({ - success: false, - message: errorMessage, - }); - } finally { - setTesting(false); - } - }; - - const getAlertType = () => { - if (!testResult) return undefined; - return testResult.success ? 'success' : 'error'; - }; - - const getAlertIcon = () => { - if (testing) return ; - if (!testResult) return undefined; - return testResult.success ? : ; - }; - - return ( - - - - {(testResult || testing) && ( - -
✓ S3 Access: Verified
- {testResult.details.identityCenterAccess &&
✓ IAM Identity Center: Configured
} - {!values.identityStoreId && ( -
- ⚠️ IAM Identity Center not configured - user display names will show as user IDs -
- )} - - ) : undefined - } - showIcon - style={{ marginTop: 8 }} - /> - )} - - ); -}; diff --git a/config-ui/src/plugins/register/q-dev/connection-fields/identity-center-config.tsx b/config-ui/src/plugins/register/q-dev/connection-fields/identity-center-config.tsx deleted file mode 100644 index c43fdd3edb2..00000000000 --- a/config-ui/src/plugins/register/q-dev/connection-fields/identity-center-config.tsx +++ /dev/null @@ -1,126 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -import { ChangeEvent, useEffect, useMemo, useRef } from 'react'; -import { Input } from 'antd'; - -import { Block } from '@/components'; - -interface Props { - initialValues: any; - values: any; - setValues: (values: any) => void; - setErrors: (errors: any) => void; -} - -const STORE_ID_PATTERN = /^d-[a-z0-9]{10}$/; -const REGION_PATTERN = /^[a-z]{2}-[a-z]+-\d$/; - -export const IdentityCenterConfig = ({ initialValues, values, setValues, setErrors }: Props) => { - const identityStoreId = values.identityStoreId ?? ''; - const identityStoreRegion = values.identityStoreRegion ?? ''; - - useEffect(() => { - if (values.identityStoreId === undefined) { - setValues({ identityStoreId: initialValues.identityStoreId ?? '' }); - } - }, [initialValues.identityStoreId, values.identityStoreId, setValues]); - - useEffect(() => { - if (values.identityStoreRegion === undefined) { - setValues({ identityStoreRegion: initialValues.identityStoreRegion ?? '' }); - } - }, [initialValues.identityStoreRegion, values.identityStoreRegion, setValues]); - - const storeIdError = useMemo(() => { - if (!identityStoreId) { - return ''; - } - if (!STORE_ID_PATTERN.test(identityStoreId)) { - return 'Expected format d-xxxxxxxxxx (lowercase letters and digits).'; - } - return ''; - }, [identityStoreId]); - - const regionError = useMemo(() => { - if (!identityStoreRegion) { - return identityStoreId ? 'Identity Center region is required when providing an Identity Store ID.' : ''; - } - if (!REGION_PATTERN.test(identityStoreRegion)) { - return 'Region should look like us-east-1.'; - } - return ''; - }, [identityStoreRegion, identityStoreId]); - - const storeIdErrorRef = useRef(undefined); - const regionErrorRef = useRef(undefined); - - useEffect(() => { - if (storeIdErrorRef.current !== storeIdError) { - storeIdErrorRef.current = storeIdError; - setErrors({ identityStoreId: storeIdError }); - } - }, [storeIdError, setErrors]); - - useEffect(() => { - if (regionErrorRef.current !== regionError) { - regionErrorRef.current = regionError; - setErrors({ identityStoreRegion: regionError }); - } - }, [regionError, setErrors]); - - const handleStoreIdChange = (e: ChangeEvent) => { - setValues({ identityStoreId: e.target.value.trim() }); - }; - - const handleRegionChange = (e: ChangeEvent) => { - setValues({ identityStoreRegion: e.target.value.trim() }); - }; - - return ( - <> - - - {storeIdError &&
{storeIdError}
} - - - - - {regionError &&
{regionError}
} - - - ); -}; diff --git a/config-ui/src/plugins/register/q-dev/connection-fields/index.ts b/config-ui/src/plugins/register/q-dev/connection-fields/index.ts deleted file mode 100644 index 2ad90f77539..00000000000 --- a/config-ui/src/plugins/register/q-dev/connection-fields/index.ts +++ /dev/null @@ -1,21 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -export * from './aws-credentials'; -export * from './s3-config'; -export * from './identity-center-config'; diff --git a/config-ui/src/plugins/register/q-dev/connection-fields/s3-config.tsx b/config-ui/src/plugins/register/q-dev/connection-fields/s3-config.tsx deleted file mode 100644 index fe6f74c1c9f..00000000000 --- a/config-ui/src/plugins/register/q-dev/connection-fields/s3-config.tsx +++ /dev/null @@ -1,76 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -import { ChangeEvent, useEffect, useMemo, useRef } from 'react'; -import { Input } from 'antd'; - -import { Block } from '@/components'; - -interface Props { - initialValues: any; - values: any; - setValues: (values: any) => void; - setErrors: (errors: any) => void; -} - -const BUCKET_PATTERN = /^[a-z0-9](?:[a-z0-9.-]{1,61}[a-z0-9])?$/; - -export const S3Config = ({ initialValues, values, setValues, setErrors }: Props) => { - const bucket = values.bucket ?? ''; - - useEffect(() => { - if (values.bucket === undefined) { - setValues({ bucket: initialValues.bucket ?? '' }); - } - }, [initialValues.bucket, values.bucket, setValues]); - - const bucketError = useMemo(() => { - if (!bucket) { - return 'S3 bucket name is required.'; - } - if (!BUCKET_PATTERN.test(bucket) || bucket.length < 3 || bucket.length > 63 || bucket.includes('..')) { - return 'Bucket names must be 3-63 characters, lowercase, numbers, dots or hyphens.'; - } - return ''; - }, [bucket]); - - const bucketErrorRef = useRef(undefined); - useEffect(() => { - if (bucketErrorRef.current !== bucketError) { - bucketErrorRef.current = bucketError; - setErrors({ bucket: bucketError }); - } - }, [bucketError, setErrors]); - - const handleBucketChange = (e: ChangeEvent) => { - setValues({ bucket: e.target.value.trim() }); - }; - - return ( - - - {bucketError &&
{bucketError}
} - - ); -}; diff --git a/config-ui/src/plugins/register/q-dev/data-scope.tsx b/config-ui/src/plugins/register/q-dev/data-scope.tsx deleted file mode 100644 index 657d0bdb890..00000000000 --- a/config-ui/src/plugins/register/q-dev/data-scope.tsx +++ /dev/null @@ -1,463 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -import { useEffect, useMemo } from 'react'; -import { Button, Checkbox, Flex, Form, Input, InputNumber, Segmented, Table, Tooltip, Typography } from 'antd'; -import type { ColumnsType } from 'antd/es/table'; -import { DeleteOutlined } from '@ant-design/icons'; - -interface ScopeData { - prefix?: string; - year?: number; - month?: number | null; - basePath?: string; - accountId?: string; -} - -interface ScopeItem { - id: string; - name: string; - fullName: string; - data?: ScopeData; -} - -interface Props { - connectionId: ID; - disabledItems?: Array<{ id: ID }>; - selectedItems: ScopeItem[]; - onChangeSelectedItems: (items: ScopeItem[]) => void; -} - -const CURRENT_YEAR = new Date().getUTCFullYear(); -const MONTHS = Array.from({ length: 12 }, (_, idx) => idx + 1); -const MONTH_LABELS = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']; - -const DEFAULT_BASE_PATH = 'user-report'; - -const ensureLeadingZero = (value: number) => value.toString().padStart(2, '0'); - -const normalizeBasePath = (value: string) => value.trim().replace(/^\/+/, '').replace(/\/+$/, ''); - -const trimTrailingSlashes = (value: string) => value.replace(/\/+$/, ''); - -const extractScopeMeta = (item: ScopeItem) => { - const data = item.data ?? {}; - const rawPrefix = data.prefix ?? item.fullName ?? item.id; - const prefix = typeof rawPrefix === 'string' ? trimTrailingSlashes(rawPrefix) : ''; - const segments = prefix ? prefix.split('/').filter(Boolean) : []; - - let month = data.month ?? null; - if (month === undefined || month === null) { - const last = segments[segments.length - 1]; - if (last && /^(0[1-9]|1[0-2])$/.test(last)) { - month = Number(last); - } else { - month = null; - } - } - - let year = data.year; - if (year === undefined || year === null) { - const idx = month ? segments.length - 2 : segments.length - 1; - const candidate = idx >= 0 ? segments[idx] : undefined; - if (candidate && /^\d{4}$/.test(candidate)) { - year = Number(candidate); - } else { - year = undefined; - } - } - - let baseSegments: string[]; - if (segments.length === 0) { - baseSegments = []; - } else if (month) { - baseSegments = segments.slice(0, Math.max(segments.length - 2, 0)); - } else { - baseSegments = segments.slice(0, Math.max(segments.length - 1, 0)); - } - - const basePath = normalizeBasePath(data.basePath ?? (baseSegments.length ? baseSegments.join('/') : '')); - - const accountId = data.accountId ?? ''; - - return { - basePath, - year: typeof year === 'number' ? year : null, - month, - prefix, - accountId, - }; -}; - -const deriveBasePathFromSelection = (items: ScopeItem[]) => { - for (const item of items) { - const meta = extractScopeMeta(item); - if (meta.basePath !== undefined) { - return meta.basePath; - } - } - return undefined; -}; - -const deriveAccountIdFromSelection = (items: ScopeItem[]) => { - for (const item of items) { - const meta = extractScopeMeta(item); - if (meta.accountId) { - return meta.accountId; - } - } - return undefined; -}; - -const buildPrefix = (basePath: string, year: number, month: number | null, accountId?: string) => { - const segments = [] as string[]; - const sanitizedBase = normalizeBasePath(basePath); - if (sanitizedBase) { - segments.push(sanitizedBase); - } - if (accountId) { - segments.push(accountId); - } - segments.push(String(year)); - if (month !== null && month !== undefined) { - segments.push(ensureLeadingZero(month)); - } - return segments.join('/'); -}; - -const createScopeItem = (basePath: string, year: number, month: number | null, accountId?: string): ScopeItem => { - const sanitizedBase = normalizeBasePath(basePath); - const prefix = buildPrefix(sanitizedBase, year, month, accountId); - const isFullYear = month === null; - const timeLabel = isFullYear - ? `${year} (Full Year)` - : `${year}-${ensureLeadingZero(month as number)} (${MONTH_LABELS[(month as number) - 1]})`; - const name = accountId ? `${accountId} ${timeLabel}` : timeLabel; - - return { - id: prefix, - name, - fullName: prefix, - data: { - basePath: sanitizedBase, - accountId: accountId || undefined, - prefix, - year, - month, - }, - }; -}; - -const formatScopeLabel = (item: ScopeItem) => { - const meta = extractScopeMeta(item); - if (!meta.year) { - return item.name; - } - - if (meta.month) { - const monthLabel = MONTH_LABELS[meta.month - 1] ?? ensureLeadingZero(meta.month); - return `${meta.year}-${ensureLeadingZero(meta.month)} (${monthLabel})`; - } - - return `${meta.year} (Full Year)`; -}; - -const MONTH_OPTIONS = MONTHS.map((value) => ({ - label: `${MONTH_LABELS[value - 1]} (${ensureLeadingZero(value)})`, - value, -})); - -type FormValues = { - basePath: string; - accountId: string; - year: number; - mode: 'year' | 'months'; - months?: number[]; -}; - -export const QDevDataScope = ({ - connectionId: _connectionId, - disabledItems, - selectedItems, - onChangeSelectedItems, -}: Props) => { - const [form] = Form.useForm(); - - const disabledIds = useMemo(() => new Set(disabledItems?.map((it) => String(it.id)) ?? []), [disabledItems]); - - const derivedBasePath = useMemo( - () => deriveBasePathFromSelection(selectedItems) ?? DEFAULT_BASE_PATH, - [selectedItems], - ); - - const derivedAccountId = useMemo(() => deriveAccountIdFromSelection(selectedItems) ?? '', [selectedItems]); - - useEffect(() => { - if (!form.isFieldsTouched(['basePath'])) { - form.setFieldsValue({ basePath: derivedBasePath }); - } - }, [derivedBasePath, form]); - - useEffect(() => { - if (!form.isFieldsTouched(['accountId'])) { - form.setFieldsValue({ accountId: derivedAccountId }); - } - }, [derivedAccountId, form]); - - useEffect(() => { - form.setFieldsValue({ mode: 'year', year: form.getFieldValue('year') ?? CURRENT_YEAR }); - }, [form]); - - const handleAdd = async () => { - const { basePath, accountId, year, mode, months = [] } = await form.validateFields(); - - const normalizedBase = normalizeBasePath(basePath ?? ''); - const normalizedAccountId = (accountId ?? '').trim(); - const normalizedYear = Number(year); - if (!normalizedYear || Number.isNaN(normalizedYear)) { - return; - } - - const currentIds = new Set(selectedItems.map((item) => item.id)); - const hasFullYear = selectedItems.some((item) => { - const meta = extractScopeMeta(item); - return ( - meta.basePath === normalizedBase && - meta.accountId === normalizedAccountId && - meta.year === normalizedYear && - (meta.month === null || meta.month === undefined) - ); - }); - - const additions: ScopeItem[] = []; - - if (mode === 'year') { - if (hasFullYear) { - return; - } - - const hasMonths = selectedItems.some((item) => { - const meta = extractScopeMeta(item); - return ( - meta.basePath === normalizedBase && - meta.accountId === normalizedAccountId && - meta.year === normalizedYear && - meta.month !== null - ); - }); - - if (hasMonths) { - return; - } - - const item = createScopeItem(normalizedBase, normalizedYear, null, normalizedAccountId || undefined); - if (!currentIds.has(item.id) && !disabledIds.has(item.id)) { - additions.push(item); - } - } else { - if (hasFullYear) { - return; - } - - const uniqueMonths = Array.from(new Set(months)) - .map((m) => Number(m)) - .filter((m) => !Number.isNaN(m)); - uniqueMonths.sort((a, b) => a - b); - - uniqueMonths.forEach((month) => { - if (month < 1 || month > 12) { - return; - } - - const item = createScopeItem(normalizedBase, normalizedYear, month, normalizedAccountId || undefined); - if (currentIds.has(item.id) || disabledIds.has(item.id)) { - return; - } - additions.push(item); - }); - } - - if (!additions.length) { - return; - } - - const next = [...selectedItems, ...additions]; - next.sort((a, b) => a.id.localeCompare(b.id)); - onChangeSelectedItems(next); - - if (mode === 'months') { - form.setFieldsValue({ months: [] }); - } - }; - - const handleRemove = (id: string) => { - onChangeSelectedItems(selectedItems.filter((item) => item.id !== id)); - }; - - const columns: ColumnsType = [ - { - title: 'Time Range', - dataIndex: 'id', - key: 'name', - render: (_: unknown, item) => formatScopeLabel(item), - }, - { - title: 'Scope Path', - dataIndex: 'id', - key: 'prefix', - render: (_: unknown, item) => { - const meta = extractScopeMeta(item); - if (meta.accountId) { - const timePart = meta.month ? `${meta.year}/${ensureLeadingZero(meta.month)}` : `${meta.year}`; - return ( - - - {meta.basePath}/…/{meta.accountId}/…/{timePart} - - - ); - } - return {meta.prefix}; - }, - }, - { - title: 'Account ID', - dataIndex: 'id', - key: 'accountId', - render: (_: unknown, item) => { - const meta = extractScopeMeta(item); - return meta.accountId ? ( - {meta.accountId} - ) : ( - — - ); - }, - }, - { - title: '', - dataIndex: 'id', - key: 'action', - width: 80, - align: 'center', - render: (id: string) => ( - - - - - - - - {selectedItems.length > 0 && ( - - These selections will be stored as S3 prefixes and used during data collection. - - )} - - ); -}; diff --git a/config-ui/src/plugins/register/q-dev/index.ts b/config-ui/src/plugins/register/q-dev/index.ts deleted file mode 100644 index de415db39ab..00000000000 --- a/config-ui/src/plugins/register/q-dev/index.ts +++ /dev/null @@ -1,19 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -export * from './config'; diff --git a/e2e/qdev-full-flow.spec.ts b/e2e/qdev-full-flow.spec.ts deleted file mode 100644 index 0498df67baa..00000000000 --- a/e2e/qdev-full-flow.spec.ts +++ /dev/null @@ -1,248 +0,0 @@ -/* -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -import { test, expect, request, Page } from '@playwright/test'; -import * as path from 'path'; -import * as fs from 'fs'; - -const API = 'http://localhost:8080'; -const UI = 'http://localhost:4000'; -const GRAFANA = 'http://localhost:3002'; -const SCREENSHOT_DIR = path.join(__dirname, 'screenshots'); - -// Use existing connection with valid credentials -const EXISTING_CONNECTION_ID = 5; - -const state: { - connectionId: number; - scopeId: string; - blueprintId: number; - pipelineId: number; -} = { connectionId: EXISTING_CONNECTION_ID, scopeId: '', blueprintId: 0, pipelineId: 0 }; - -fs.mkdirSync(SCREENSHOT_DIR, { recursive: true }); - -async function grafanaLogin(page: Page) { - await page.goto(`${GRAFANA}/grafana/login`); - await page.waitForLoadState('networkidle'); - if (page.url().includes('/login')) { - await page.locator('input[name="user"]').fill('admin'); - await page.locator('input[name="password"]').fill('admin'); - await page.locator('button[type="submit"]').click(); - await page.waitForTimeout(2000); - // Handle "change password" prompt if shown - const skipBtn = page.locator('a:has-text("Skip")'); - if (await skipBtn.isVisible({ timeout: 2000 }).catch(() => false)) { - await skipBtn.click(); - } - await page.waitForTimeout(1000); - } -} - -async function openGrafanaDashboard(page: Page, uid: string, screenshotPath: string) { - await grafanaLogin(page); - await page.goto(`${GRAFANA}/grafana/d/${uid}?orgId=1&from=now-90d&to=now`); - - // Wait for first panel data to load - try { - await page.waitForResponse( - (resp) => resp.url().includes('/api/ds/query') && resp.status() === 200, - { timeout: 30000 } - ); - } catch { /* some dashboards may not fire queries immediately */ } - - // Wait for rendering to settle - await page.waitForTimeout(5000); - - // Take viewport screenshot (top section) - await page.screenshot({ path: screenshotPath.replace('.png', '-top.png') }); - - // Scroll down and take more sections - const scrollHeight = await page.evaluate(() => document.body.scrollHeight); - let section = 1; - for (let y = 900; y < scrollHeight; y += 900) { - await page.evaluate((scrollY) => window.scrollTo(0, scrollY), y); - await page.waitForTimeout(3000); - section++; - await page.screenshot({ path: screenshotPath.replace('.png', `-section${section}.png`) }); - } - - // Also take full page screenshot - await page.evaluate(() => window.scrollTo(0, 0)); - await page.waitForTimeout(2000); - await page.screenshot({ path: screenshotPath, fullPage: true }); -} - -test.describe.serial('Q-Dev Plugin Full Flow', () => { - - test('Step 1: Verify Existing Connection via API', async () => { - const api = await request.newContext({ baseURL: API }); - - const resp = await api.get(`/plugins/q_dev/connections/${state.connectionId}`); - expect(resp.ok()).toBeTruthy(); - const conn = await resp.json(); - console.log(`Using connection: id=${conn.id}, name=${conn.name}, bucket=${conn.bucket}`); - - const testResp = await api.post(`/plugins/q_dev/connections/${state.connectionId}/test`); - const testBody = await testResp.json(); - console.log('Test connection:', testBody.success ? 'OK' : testBody.message); - expect(testResp.ok()).toBeTruthy(); - }); - - test('Step 2: View Config-UI Home', async ({ page }) => { - await page.goto(UI); - await page.waitForLoadState('networkidle'); - await page.waitForTimeout(1000); - await page.screenshot({ path: path.join(SCREENSHOT_DIR, '01-config-ui-home.png'), fullPage: true }); - console.log('Screenshot: Config-UI home'); - }); - - test('Step 3: Create Scope (S3 Slice) via API', async () => { - const api = await request.newContext({ baseURL: API }); - - const resp = await api.put(`/plugins/q_dev/connections/${state.connectionId}/scopes`, { - data: { - data: [ - { - accountId: '034362076319', - basePath: '', - year: 2026, - month: 3, - }, - ], - }, - }); - - const body = await resp.json(); - console.log('Scope created:', resp.status()); - expect(resp.ok()).toBeTruthy(); - state.scopeId = body[0]?.id; - expect(state.scopeId).toBeTruthy(); - console.log(`Scope id: ${state.scopeId}`); - }); - - test('Step 4: Create Blueprint via API', async () => { - const api = await request.newContext({ baseURL: API }); - - const resp = await api.post('/blueprints', { - data: { - name: `e2e-blueprint-${Date.now()}`, - mode: 'NORMAL', - enable: true, - cronConfig: '0 0 * * *', - isManual: true, - connections: [ - { - pluginName: 'q_dev', - connectionId: state.connectionId, - scopes: [{ scopeId: state.scopeId }], - }, - ], - }, - }); - - const body = await resp.json(); - expect(resp.ok()).toBeTruthy(); - state.blueprintId = body.id; - console.log(`Blueprint created: id=${state.blueprintId}`); - }); - - test('Step 5: Trigger Pipeline via API', async () => { - const api = await request.newContext({ baseURL: API }); - - const resp = await api.post(`/blueprints/${state.blueprintId}/trigger`, { data: {} }); - const body = await resp.json(); - expect(resp.ok()).toBeTruthy(); - state.pipelineId = body.id; - console.log(`Pipeline triggered: id=${state.pipelineId}`); - }); - - test('Step 6: Wait for Pipeline to Complete', async () => { - const api = await request.newContext({ baseURL: API }); - const maxWait = 120000; - const start = Date.now(); - let status = ''; - - while (Date.now() - start < maxWait) { - const resp = await api.get(`/pipelines/${state.pipelineId}`); - const pipeline = await resp.json(); - status = pipeline.status; - console.log(`Pipeline status: ${status} (${Math.round((Date.now() - start) / 1000)}s)`); - if (['TASK_COMPLETED', 'TASK_FAILED', 'TASK_PARTIAL'].includes(status)) break; - await new Promise((r) => setTimeout(r, 3000)); - } - - // Print task details - const tasksResp = await api.get(`/pipelines/${state.pipelineId}/tasks`); - if (tasksResp.ok()) { - const { tasks } = await tasksResp.json(); - for (const t of tasks || []) { - console.log(` Task ${t.id}: ${t.status}${t.failedSubTask ? ` (failed: ${t.failedSubTask})` : ''}`); - if (t.message) console.log(` Error: ${t.message.substring(0, 300)}`); - } - } - - expect(status).toBe('TASK_COMPLETED'); - }); - - test('Step 7: Verify Data via MySQL', async () => { - const api = await request.newContext({ baseURL: API }); - - // Use pipeline tasks to confirm data was processed - const tasksResp = await api.get(`/pipelines/${state.pipelineId}/tasks`); - const { tasks } = await tasksResp.json(); - expect(tasks[0].status).toBe('TASK_COMPLETED'); - console.log(`Pipeline completed in ${tasks[0].spentSeconds}s`); - }); - - test('Step 8: Grafana - Kiro Usage Dashboard (new format)', async ({ page }) => { - await openGrafanaDashboard(page, 'qdev_user_report', path.join(SCREENSHOT_DIR, '02-dashboard-user-report.png')); - console.log('Screenshot: Kiro Usage Dashboard'); - }); - - test('Step 9: Grafana - Kiro Legacy Feature Metrics', async ({ page }) => { - await openGrafanaDashboard(page, 'qdev_feature_metrics', path.join(SCREENSHOT_DIR, '03-dashboard-feature-metrics.png')); - console.log('Screenshot: Kiro Legacy Feature Metrics'); - }); - - test('Step 10: Grafana - Kiro AI Activity Insights (logging)', async ({ page }) => { - await openGrafanaDashboard(page, 'qdev_logging', path.join(SCREENSHOT_DIR, '04-dashboard-logging.png')); - console.log('Screenshot: Kiro AI Activity Insights'); - }); - - test('Step 11: Grafana - Kiro Executive Dashboard', async ({ page }) => { - await openGrafanaDashboard(page, 'qdev_executive', path.join(SCREENSHOT_DIR, '05-dashboard-executive.png')); - console.log('Screenshot: Kiro Executive Dashboard'); - }); - - test('Step 12: View Pipeline in Config-UI', async ({ page }) => { - // Navigate to the API proxy route for pipelines - await page.goto(`${UI}/api/pipelines?pageSize=5`); - await page.waitForLoadState('networkidle'); - await page.screenshot({ path: path.join(SCREENSHOT_DIR, '06-config-ui-pipelines.png'), fullPage: true }); - console.log('Screenshot: Pipelines API response'); - }); - - test('Step 13: Cleanup', async () => { - const api = await request.newContext({ baseURL: API }); - if (state.blueprintId) { - await api.delete(`/blueprints/${state.blueprintId}`); - console.log(`Deleted blueprint ${state.blueprintId}`); - } - console.log('Cleanup complete'); - }); -});