General: Track token usage of LLM service requests
#9455
Conversation
github-actions
bot
added
tests
server
alexjoham
changed the title
alexjoham
changed the title
src/main/java/de/tum/cit/aet/artemis/core/domain/LLMTokenUsage.java
Outdated
Show resolved
Hide resolved
src/main/java/de/tum/cit/aet/artemis/core/domain/LLMTokenUsage.java
Outdated
Show resolved
Hide resolved
src/main/java/de/tum/cit/aet/artemis/core/domain/LLMServiceType.java
Outdated
Show resolved
Hide resolved
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
WalkthroughThe changes introduced in this pull request encompass the addition of new classes and modifications to existing ones within the Artemis application. Key updates include the introduction of Changes
Suggested labels
Suggested reviewers
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
There was a problem hiding this comment.
Actionable comments posted: 25
🧹 Outside diff range comments (2)
src/main/java/de/tum/cit/aet/artemis/iris/dto/IrisChatWebsocketDTO.java (1)
Line range hint
62-65: Updateequalsmethod to include the newtokensfield.The
equalsmethod has not been updated to include the newtokensfield. This could lead to inconsistent behavior when comparingIrisChatWebsocketDTOobjects.Please update the
equalsmethod to include thetokensfield in the comparison. Here's a suggested implementation:@Override public boolean equals(Object o) { if (this == o) { return true; } if (o == null || getClass() != o.getClass()) { return false; } IrisChatWebsocketDTO that = (IrisChatWebsocketDTO) o; return type == that.type && Objects.equals(message, that.message) && Objects.equals(rateLimitInfo, that.rateLimitInfo) && Objects.equals(stages, that.stages) && Objects.equals(suggestions, that.suggestions) && Objects.equals(tokens, that.tokens); }Also, consider overriding the
hashCodemethod to maintain consistency with the updatedequalsmethod.src/test/java/de/tum/cit/aet/artemis/iris/IrisCompetencyGenerationIntegrationTest.java (1)
Line range hint
37-85: Test method adheres well to coding guidelines. Consider minor enhancements.The test method
generateCompetencies_asEditor_shouldSucceedgenerally follows the provided coding guidelines:
- Descriptive naming
- Small, specific test case
- Use of fixed data
- JUnit 5 features
- assertThat for assertions
- Avoidance of direct DB access
- Use of mocking
Consider the following enhancements:
- Add explicit DB query count tracking if relevant to this test case.
- Ensure ArchUnit is used elsewhere to enforce package rules if not applicable here.
- If possible, add a comment or mechanism to address context restart minimization.
These suggestions aim to fully align the test with all aspects of the provided coding guidelines.
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: ASSERTIVE
⛔ Files ignored due to path filters (2)
src/main/resources/config/liquibase/changelog/20241012125003_changelog.xmlis excluded by!**/*.xmlsrc/main/resources/config/liquibase/master.xmlis excluded by!**/*.xml
📒 Files selected for processing (16)
- src/main/java/de/tum/cit/aet/artemis/core/domain/LLMServiceType.java (1 hunks)
- src/main/java/de/tum/cit/aet/artemis/core/domain/LLMTokenUsage.java (1 hunks)
- src/main/java/de/tum/cit/aet/artemis/core/repository/LLMTokenUsageRepository.java (1 hunks)
- src/main/java/de/tum/cit/aet/artemis/core/service/LLMTokenUsageService.java (1 hunks)
- src/main/java/de/tum/cit/aet/artemis/iris/dto/IrisChatWebsocketDTO.java (3 hunks)
- src/main/java/de/tum/cit/aet/artemis/iris/service/IrisCompetencyGenerationService.java (4 hunks)
- src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/dto/chat/PyrisChatStatusUpdateDTO.java (1 hunks)
- src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/dto/competency/PyrisCompetencyStatusUpdateDTO.java (2 hunks)
- src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/dto/data/PyrisLLMCostDTO.java (1 hunks)
- src/main/java/de/tum/cit/aet/artemis/iris/service/session/IrisCourseChatSessionService.java (4 hunks)
- src/main/java/de/tum/cit/aet/artemis/iris/service/session/IrisExerciseChatSessionService.java (4 hunks)
- src/main/java/de/tum/cit/aet/artemis/iris/service/websocket/IrisChatWebsocketService.java (4 hunks)
- src/main/webapp/app/entities/iris/iris-message.model.ts (2 hunks)
- src/test/java/de/tum/cit/aet/artemis/iris/IrisChatMessageIntegrationTest.java (1 hunks)
- src/test/java/de/tum/cit/aet/artemis/iris/IrisChatWebsocketTest.java (1 hunks)
- src/test/java/de/tum/cit/aet/artemis/iris/IrisCompetencyGenerationIntegrationTest.java (1 hunks)
🧰 Additional context used
📓 Path-based instructions (16)
src/main/java/de/tum/cit/aet/artemis/core/domain/LLMServiceType.java (1)
Pattern
src/main/java/**/*.java: naming:CamelCase; principles:{single_responsibility,small_methods,no_duplication}; db:{perf_queries,datetime_not_timestamp}; rest:{stateless,singleton,delegate_logic,http_only,minimal_dtos}; dtos:{java_records,no_entities,min_data,single_resp}; di:constructor_injection; kiss:simple_code; file_handling:os_indep_paths; practices:{least_access,avoid_transactions,code_reuse,static_member_ref,prefer_primitives}; sql:{param_annotation,uppercase,avoid_subqueries};java:avoid_star_importssrc/main/java/de/tum/cit/aet/artemis/core/domain/LLMTokenUsage.java (1)
Pattern
src/main/java/**/*.java: naming:CamelCase; principles:{single_responsibility,small_methods,no_duplication}; db:{perf_queries,datetime_not_timestamp}; rest:{stateless,singleton,delegate_logic,http_only,minimal_dtos}; dtos:{java_records,no_entities,min_data,single_resp}; di:constructor_injection; kiss:simple_code; file_handling:os_indep_paths; practices:{least_access,avoid_transactions,code_reuse,static_member_ref,prefer_primitives}; sql:{param_annotation,uppercase,avoid_subqueries};java:avoid_star_importssrc/main/java/de/tum/cit/aet/artemis/core/repository/LLMTokenUsageRepository.java (1)
Pattern
src/main/java/**/*.java: naming:CamelCase; principles:{single_responsibility,small_methods,no_duplication}; db:{perf_queries,datetime_not_timestamp}; rest:{stateless,singleton,delegate_logic,http_only,minimal_dtos}; dtos:{java_records,no_entities,min_data,single_resp}; di:constructor_injection; kiss:simple_code; file_handling:os_indep_paths; practices:{least_access,avoid_transactions,code_reuse,static_member_ref,prefer_primitives}; sql:{param_annotation,uppercase,avoid_subqueries};java:avoid_star_importssrc/main/java/de/tum/cit/aet/artemis/core/service/LLMTokenUsageService.java (1)
Pattern
src/main/java/**/*.java: naming:CamelCase; principles:{single_responsibility,small_methods,no_duplication}; db:{perf_queries,datetime_not_timestamp}; rest:{stateless,singleton,delegate_logic,http_only,minimal_dtos}; dtos:{java_records,no_entities,min_data,single_resp}; di:constructor_injection; kiss:simple_code; file_handling:os_indep_paths; practices:{least_access,avoid_transactions,code_reuse,static_member_ref,prefer_primitives}; sql:{param_annotation,uppercase,avoid_subqueries};java:avoid_star_importssrc/main/java/de/tum/cit/aet/artemis/iris/dto/IrisChatWebsocketDTO.java (1)
Pattern
src/main/java/**/*.java: naming:CamelCase; principles:{single_responsibility,small_methods,no_duplication}; db:{perf_queries,datetime_not_timestamp}; rest:{stateless,singleton,delegate_logic,http_only,minimal_dtos}; dtos:{java_records,no_entities,min_data,single_resp}; di:constructor_injection; kiss:simple_code; file_handling:os_indep_paths; practices:{least_access,avoid_transactions,code_reuse,static_member_ref,prefer_primitives}; sql:{param_annotation,uppercase,avoid_subqueries};java:avoid_star_importssrc/main/java/de/tum/cit/aet/artemis/iris/service/IrisCompetencyGenerationService.java (1)
Pattern
src/main/java/**/*.java: naming:CamelCase; principles:{single_responsibility,small_methods,no_duplication}; db:{perf_queries,datetime_not_timestamp}; rest:{stateless,singleton,delegate_logic,http_only,minimal_dtos}; dtos:{java_records,no_entities,min_data,single_resp}; di:constructor_injection; kiss:simple_code; file_handling:os_indep_paths; practices:{least_access,avoid_transactions,code_reuse,static_member_ref,prefer_primitives}; sql:{param_annotation,uppercase,avoid_subqueries};java:avoid_star_importssrc/main/java/de/tum/cit/aet/artemis/iris/service/pyris/dto/chat/PyrisChatStatusUpdateDTO.java (1)
Pattern
src/main/java/**/*.java: naming:CamelCase; principles:{single_responsibility,small_methods,no_duplication}; db:{perf_queries,datetime_not_timestamp}; rest:{stateless,singleton,delegate_logic,http_only,minimal_dtos}; dtos:{java_records,no_entities,min_data,single_resp}; di:constructor_injection; kiss:simple_code; file_handling:os_indep_paths; practices:{least_access,avoid_transactions,code_reuse,static_member_ref,prefer_primitives}; sql:{param_annotation,uppercase,avoid_subqueries};java:avoid_star_importssrc/main/java/de/tum/cit/aet/artemis/iris/service/pyris/dto/competency/PyrisCompetencyStatusUpdateDTO.java (1)
Pattern
src/main/java/**/*.java: naming:CamelCase; principles:{single_responsibility,small_methods,no_duplication}; db:{perf_queries,datetime_not_timestamp}; rest:{stateless,singleton,delegate_logic,http_only,minimal_dtos}; dtos:{java_records,no_entities,min_data,single_resp}; di:constructor_injection; kiss:simple_code; file_handling:os_indep_paths; practices:{least_access,avoid_transactions,code_reuse,static_member_ref,prefer_primitives}; sql:{param_annotation,uppercase,avoid_subqueries};java:avoid_star_importssrc/main/java/de/tum/cit/aet/artemis/iris/service/pyris/dto/data/PyrisLLMCostDTO.java (1)
Pattern
src/main/java/**/*.java: naming:CamelCase; principles:{single_responsibility,small_methods,no_duplication}; db:{perf_queries,datetime_not_timestamp}; rest:{stateless,singleton,delegate_logic,http_only,minimal_dtos}; dtos:{java_records,no_entities,min_data,single_resp}; di:constructor_injection; kiss:simple_code; file_handling:os_indep_paths; practices:{least_access,avoid_transactions,code_reuse,static_member_ref,prefer_primitives}; sql:{param_annotation,uppercase,avoid_subqueries};java:avoid_star_importssrc/main/java/de/tum/cit/aet/artemis/iris/service/session/IrisCourseChatSessionService.java (1)
Pattern
src/main/java/**/*.java: naming:CamelCase; principles:{single_responsibility,small_methods,no_duplication}; db:{perf_queries,datetime_not_timestamp}; rest:{stateless,singleton,delegate_logic,http_only,minimal_dtos}; dtos:{java_records,no_entities,min_data,single_resp}; di:constructor_injection; kiss:simple_code; file_handling:os_indep_paths; practices:{least_access,avoid_transactions,code_reuse,static_member_ref,prefer_primitives}; sql:{param_annotation,uppercase,avoid_subqueries};java:avoid_star_importssrc/main/java/de/tum/cit/aet/artemis/iris/service/session/IrisExerciseChatSessionService.java (1)
Pattern
src/main/java/**/*.java: naming:CamelCase; principles:{single_responsibility,small_methods,no_duplication}; db:{perf_queries,datetime_not_timestamp}; rest:{stateless,singleton,delegate_logic,http_only,minimal_dtos}; dtos:{java_records,no_entities,min_data,single_resp}; di:constructor_injection; kiss:simple_code; file_handling:os_indep_paths; practices:{least_access,avoid_transactions,code_reuse,static_member_ref,prefer_primitives}; sql:{param_annotation,uppercase,avoid_subqueries};java:avoid_star_importssrc/main/java/de/tum/cit/aet/artemis/iris/service/websocket/IrisChatWebsocketService.java (1)
Pattern
src/main/java/**/*.java: naming:CamelCase; principles:{single_responsibility,small_methods,no_duplication}; db:{perf_queries,datetime_not_timestamp}; rest:{stateless,singleton,delegate_logic,http_only,minimal_dtos}; dtos:{java_records,no_entities,min_data,single_resp}; di:constructor_injection; kiss:simple_code; file_handling:os_indep_paths; practices:{least_access,avoid_transactions,code_reuse,static_member_ref,prefer_primitives}; sql:{param_annotation,uppercase,avoid_subqueries};java:avoid_star_importssrc/main/webapp/app/entities/iris/iris-message.model.ts (1)
src/test/java/de/tum/cit/aet/artemis/iris/IrisChatMessageIntegrationTest.java (1)
Pattern
src/test/java/**/*.java: test_naming: descriptive; test_size: small_specific; fixed_data: true; junit5_features: true; assert_use: assertThat; assert_specificity: true; archunit_use: enforce_package_rules; db_query_count_tests: track_performance; util_service_factory_pattern: true; avoid_db_access: true; mock_strategy: static_mocks; context_restart_minimize: truesrc/test/java/de/tum/cit/aet/artemis/iris/IrisChatWebsocketTest.java (1)
Pattern
src/test/java/**/*.java: test_naming: descriptive; test_size: small_specific; fixed_data: true; junit5_features: true; assert_use: assertThat; assert_specificity: true; archunit_use: enforce_package_rules; db_query_count_tests: track_performance; util_service_factory_pattern: true; avoid_db_access: true; mock_strategy: static_mocks; context_restart_minimize: truesrc/test/java/de/tum/cit/aet/artemis/iris/IrisCompetencyGenerationIntegrationTest.java (1)
Pattern
src/test/java/**/*.java: test_naming: descriptive; test_size: small_specific; fixed_data: true; junit5_features: true; assert_use: assertThat; assert_specificity: true; archunit_use: enforce_package_rules; db_query_count_tests: track_performance; util_service_factory_pattern: true; avoid_db_access: true; mock_strategy: static_mocks; context_restart_minimize: true
📓 Learnings (2)
src/main/java/de/tum/cit/aet/artemis/iris/service/websocket/IrisChatWebsocketService.java (2)
Learnt from: Hialus PR: ls1intum/Artemis#8286 File: src/main/java/de/tum/in/www1/artemis/service/iris/websocket/IrisChatWebsocketService.java:48-0 Timestamp: 2024-10-08T15:35:42.972Z Learning: In the `IrisChatWebsocketService.java` file, the `sendMessage` method is designed to allow exceptions to propagate without specific handling, as this is an intentional part of the error handling strategy.Learnt from: Hialus PR: ls1intum/Artemis#8286 File: src/main/java/de/tum/in/www1/artemis/service/iris/websocket/IrisChatWebsocketService.java:48-0 Timestamp: 2024-10-08T15:35:52.595Z Learning: In the `IrisChatWebsocketService.java` file, the `sendMessage` method is designed to allow exceptions to propagate without specific handling, as this is an intentional part of the error handling strategy.src/test/java/de/tum/cit/aet/artemis/iris/IrisCompetencyGenerationIntegrationTest.java (2)
Learnt from: julian-christl PR: ls1intum/Artemis#8052 File: src/test/java/de/tum/in/www1/artemis/lecture/CompetencyIntegrationTest.java:310-310 Timestamp: 2024-02-23T00:03:06.365Z Learning: Modifications to parameters in `competencyProgressUtilService.createCompetencyProgress` for debugging purposes are considered irrelevant to the test outcomes but helpful for clarity during debugging.Learnt from: julian-christl PR: ls1intum/Artemis#8052 File: src/test/java/de/tum/in/www1/artemis/lecture/CompetencyIntegrationTest.java:310-310 Timestamp: 2024-10-08T15:35:52.595Z Learning: Modifications to parameters in `competencyProgressUtilService.createCompetencyProgress` for debugging purposes are considered irrelevant to the test outcomes but helpful for clarity during debugging.
🔇 Additional comments (31)
src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/dto/data/PyrisLLMCostDTO.java (3)
1-1: LGTM: Package declaration is correct.The package declaration follows the standard Java naming conventions and aligns with the project structure.
3-3: LGTM: Import statement is appropriate.The import statement is specific and imports only the necessary
LLMServiceTypeenum, which is used in the record declaration. This adheres to the guideline of avoiding star imports.
1-6: Overall, well-structured DTO with minor naming convention issue.This new
PyrisLLMCostDTOrecord is well-designed and adheres to most of our coding guidelines:
- It follows the single responsibility principle.
- Uses Java records for DTOs as recommended.
- Promotes immutability.
- Uses primitive types where appropriate.
- Captures necessary information without redundancy.
The only improvement needed is to adjust the field names to follow Java's camelCase convention. Once this is addressed, the file will fully comply with our coding standards.
src/main/java/de/tum/cit/aet/artemis/core/domain/LLMServiceType.java (2)
1-1: LGTM: Package declaration is correct.The package declaration follows the expected naming convention and structure.
4-5: Past review comment has been addressed.The distinction between
ATHENA_PRELIMINARY_FEEDBACKandATHENA_FEEDBACK_SUGGESTIONsuggested in the past review comment has been implemented in the current version of the enum.src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/dto/chat/PyrisChatStatusUpdateDTO.java (2)
7-7: LGTM: New import added correctlyThe import for
PyrisLLMCostDTOis correctly added, which is necessary for the newtokensfield in the record. This change aligns with the PR objectives of tracking token usage.
11-11: LGTM: Record updated to include token usage trackingThe
PyrisChatStatusUpdateDTOrecord has been correctly updated to include the newtokensfield of typeList<PyrisLLMCostDTO>. This change:
- Aligns with the PR objectives of tracking token usage in Iris requests.
- Uses Java records, which is in line with the coding guidelines for DTOs.
- Follows the naming convention of CamelCase.
- Adheres to the principle of minimal DTOs by only adding the necessary field for token usage tracking.
The use of a List for
tokensallows for multiple token usage entries per chat status update, which provides flexibility in tracking token usage across different stages or components of a chat interaction.src/main/java/de/tum/cit/aet/artemis/core/repository/LLMTokenUsageRepository.java (5)
1-1: LGTM: Package declaration follows conventions.The package declaration adheres to Java naming conventions and follows the project's structure.
3-9: LGTM: Imports are well-organized and follow guidelines.The import statements are properly organized, grouped logically, and avoid wildcard imports as per the coding guidelines.
11-12: LGTM: Annotations are appropriate and align with project requirements.The
@Repositoryannotation correctly identifies this as a Spring Data repository, and the@Profile(PROFILE_IRIS)annotation ensures that this repository is only active when the Iris profile is in use, which aligns with the PR objectives.
13-14: LGTM: Interface declaration follows best practices and coding guidelines.The
LLMTokenUsageRepositoryinterface:
- Follows CamelCase naming convention.
- Extends the appropriate base repository (
ArtemisJpaRepository).- Adheres to the single responsibility principle.
- Correctly uses generics to specify the entity type (
LLMTokenUsage) and ID type (Long).The empty interface is appropriate as it inherits necessary methods from the base repository.
1-14: Overall: Excellent implementation of the LLMTokenUsageRepository.This new repository interface is well-designed and aligns perfectly with the PR objectives for tracking token usage in the Iris service. It adheres to all relevant coding guidelines, including:
- Proper package structure and naming conventions.
- Correct use of imports without wildcards.
- Appropriate use of Spring annotations (@repository and @Profile).
- Following the single responsibility principle and extending the correct base repository.
The empty interface is sufficient as it inherits all necessary methods from ArtemisJpaRepository. This implementation provides a solid foundation for managing LLMTokenUsage entities within the Artemis application.
src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/dto/competency/PyrisCompetencyStatusUpdateDTO.java (2)
7-7: LGTM: New import added for PyrisLLMCostDTOThe import for
PyrisLLMCostDTOis correctly added to support the new parameter in the record.
19-19: LGTM: Record updated to include token informationThe
PyrisCompetencyStatusUpdateDTOrecord has been correctly updated to include theList<PyrisLLMCostDTO> tokensparameter. This change aligns with the PR objectives to track token usage in Iris requests.A few observations:
- The naming convention follows CamelCase as required.
- The record adheres to the single responsibility principle, focusing on competency status updates.
- The use of
Listfor all parameters suggests consistency in data structure.- The record uses Java records, which aligns with the DTO guidelines.
src/main/webapp/app/entities/iris/iris-message.model.ts (1)
Line range hint
1-31: Overall structure and changes look goodThe overall structure of the file and the changes made align well with the PR objective of tracking token usage. The file adheres to most of the coding guidelines, including:
- Using single quotes for strings
- Using PascalCase for enums and types
- Following the Angular style guide
Once the property naming is adjusted to camelCase as suggested in the previous comments, the file will fully comply with the coding guidelines.
src/main/java/de/tum/cit/aet/artemis/iris/dto/IrisChatWebsocketDTO.java (3)
12-12: LGTM: Import statement added correctly.The new import for
PyrisLLMCostDTOis necessary for the addedtokensfield and follows the Java import guidelines.
25-26: LGTM: Record declaration updated appropriately.The addition of the
List<PyrisLLMCostDTO> tokensparameter to the record declaration is consistent with the PR objectives and follows coding guidelines. It maintains the single responsibility principle for this DTO.
Line range hint
1-73: Overall, the changes effectively implement token usage tracking.The modifications to
IrisChatWebsocketDTOsuccessfully incorporate the newtokensfield for tracking LLM token usage. The changes are consistent and adhere to coding guidelines. Minor improvements have been suggested for documentation and object comparison. Once these are addressed, the implementation will be robust and aligned with the PR objectives.src/main/java/de/tum/cit/aet/artemis/iris/service/session/IrisCourseChatSessionService.java (3)
22-22: LGTM: Import statement added correctly.The import for
LLMTokenUsageServiceis properly placed and follows the coding guideline of avoiding star imports.
47-48: LGTM: Field added correctly following best practices.The
llmTokenUsageServicefield is properly declared as private and final, adhering to the principle of least access. The use of constructor injection for this dependency is in line with best practices for dependency injection.
Line range hint
1-238: Consider adding documentation and improving error handling.The changes for incorporating token usage tracking have been implemented well and align with the PR objectives. To further improve the code:
Add JavaDoc comments to the new
llmTokenUsageServicefield and the modifiedhandleStatusUpdatemethod to explain their purpose and any important details about token usage tracking.Consider adding error handling for the token usage saving operations. For example:
try { llmTokenUsageService.saveTokenUsage(...); } catch (Exception e) { // Log the error and possibly notify an admin log.error("Failed to save token usage", e); }
- Ensure that the
LLMTokenUsageServiceis properly configured and tested. You might want to add integration tests for the token usage tracking functionality.To verify the proper configuration of
LLMTokenUsageService, please run the following command:This will help ensure that the new service is properly set up in the application context.
src/main/java/de/tum/cit/aet/artemis/core/service/LLMTokenUsageService.java (2)
63-64: Ensure handling whenuseris nullThe code sets
userIdonly ifuseris not null:if (user != null) { llmTokenUsage.setUserId(user.getId()); }Is it acceptable for
userIdto be null inLLMTokenUsage? IfuserIdis a non-nullable field in the database, this may cause issues.If
userIdmust not be null, consider throwing an exception or handling this case appropriately.
66-70: Check for potential null values inPyrisLLMCostDTOpropertiesWhen accessing properties of
PyrisLLMCostDTO, such asnum_input_tokens(),cost_per_input_token(), etc., there's no check for null values. If any of these methods can return null, it could lead toNullPointerException.Confirm whether the properties of
PyrisLLMCostDTOare guaranteed to be non-null. If they can be null, add checks or default values.src/main/java/de/tum/cit/aet/artemis/iris/service/IrisCompetencyGenerationService.java (3)
11-11: ImportLLMTokenUsageServiceadded correctlyThe import statement for
LLMTokenUsageServicehas been appropriately added, ensuring the class is recognized within this file.
29-29: Declaration ofllmTokenUsageServicealigns with best practicesThe
llmTokenUsageServiceis declared as aprivate finalfield, adhering to the principles of immutability and least access. This follows the best practice for dependency injection.
35-40: Constructor updated with new dependencyThe constructor has been correctly updated to include
LLMTokenUsageService llmTokenUsageService, and the assignmentthis.llmTokenUsageService = llmTokenUsageService;has been added. This aligns with the dependency injection via constructor injection guideline.src/main/java/de/tum/cit/aet/artemis/iris/service/websocket/IrisChatWebsocketService.java (5)
14-14: Addition ofPyrisLLMCostDTOimport is appropriateThe import of
PyrisLLMCostDTOis necessary for handling token cost data in the updated methods.
45-45: EnsureIrisChatWebsocketDTOhandlesnullvalues safelyWhen constructing
IrisChatWebsocketDTOwithnullvalues forsuggestionsandtokens, verify that the class handlesnullinputs appropriately to prevent potentialNullPointerExceptionissues.
56-56: Method overloading maintains backward compatibilityThe
sendStatusUpdatemethod now delegates to the overloaded method with additional parameters. This approach maintains backward compatibility and promotes code reuse.
70-70: Handle potentialnullvalues inpayloadWhen constructing the
payloadwithnullforirisMessage, ensure that theIrisChatWebsocketDTOclass and its consumers handlenullvalues correctly to prevent unexpected behaviors.
66-66:⚠️ Potential issueChanging public method signature may introduce breaking changes
The
sendStatusUpdatemethod signature has been updated to includesuggestionsandtokens. This could potentially break existing code that calls this method with the old signature. Ensure that all external callers have been updated to match the new method signature to prevent runtime errors.Run the following script to identify any usages of
sendStatusUpdatethat may need updating:✅ Verification successful
Verification Successful: No usages of
sendStatusUpdatewith the old signature found.🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Find all calls to `sendStatusUpdate` with the old signature. # Test: Search for method calls to `sendStatusUpdate` with three arguments (excluding 'this' pointer). # Expect: No instances should be found; all calls should use the new signature with four arguments. ast-grep --lang java --pattern '$_.sendStatusUpdate($_, $_, $_)' --ignore 'IrisChatWebsocketService.java'Length of output: 326
Script:
#!/bin/bash # Description: Find all calls to `sendStatusUpdate` with the old signature, excluding IrisChatWebsocketService.java. # Using fd to list all Java files excluding IrisChatWebsocketService.java and passing them to ast-grep fd --extension java --exclude 'IrisChatWebsocketService.java' | xargs ast-grep --lang java --pattern '$_.sendStatusUpdate($_, $_, $_)'Length of output: 258
Script:
#!/bin/bash # Description: Find all calls to `sendStatusUpdate` with the old signature, excluding IrisChatWebsocketService.java and EncodingISO_8559_1.java. # Using fd to list all Java files excluding the specified problematic files and passing them to ast-grep fd --extension java --exclude 'IrisChatWebsocketService.java' --exclude 'EncodingISO_8559_1.java' | xargs ast-grep --lang java --pattern '$_.sendStatusUpdate($_, $_, $_)' 2>/dev/nullLength of output: 168
src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/dto/data/PyrisLLMCostDTO.java
Outdated
Show resolved
Hide resolved
src/main/java/de/tum/cit/aet/artemis/core/domain/LLMServiceType.java
Outdated
Show resolved
Hide resolved
...de/tum/cit/aet/artemis/iris/service/pyris/dto/competency/PyrisCompetencyStatusUpdateDTO.java
Outdated
Show resolved
Hide resolved
src/main/java/de/tum/cit/aet/artemis/core/domain/LLMTokenUsage.java
Outdated
Show resolved
Hide resolved
src/main/java/de/tum/cit/aet/artemis/core/domain/LLMTokenUsage.java
Outdated
Show resolved
Hide resolved
src/main/java/de/tum/cit/aet/artemis/core/domain/LLMTokenUsage.java
Outdated
Show resolved
Hide resolved
src/main/java/de/tum/cit/aet/artemis/iris/service/session/IrisExerciseChatSessionService.java
Outdated
Show resolved
Hide resolved
src/main/java/de/tum/cit/aet/artemis/iris/service/session/IrisExerciseChatSessionService.java
Outdated
Show resolved
Hide resolved
alexjoham
changed the title
Iris: Track token usage of iris requests
src/main/java/de/tum/cit/aet/artemis/core/domain/LLMTokenUsage.java
Outdated
Show resolved
Hide resolved
src/main/java/de/tum/cit/aet/artemis/core/domain/LLMTokenUsage.java
Outdated
Show resolved
Hide resolved
src/main/java/de/tum/cit/aet/artemis/core/service/LLMTokenUsageService.java
Outdated
Show resolved
Hide resolved
src/main/java/de/tum/cit/aet/artemis/core/service/LLMTokenUsageService.java
Outdated
Show resolved
Hide resolved
src/main/java/de/tum/cit/aet/artemis/iris/service/IrisCompetencyGenerationService.java
Outdated
Show resolved
Hide resolved
src/main/java/de/tum/cit/aet/artemis/iris/service/session/IrisExerciseChatSessionService.java
Outdated
Show resolved
Hide resolved
src/main/resources/config/liquibase/changelog/20241012125003_changelog.xml
Outdated
Show resolved
Hide resolved
src/main/java/de/tum/cit/aet/artemis/core/domain/LLMTokenUsage.java
Outdated
Show resolved
Hide resolved
src/main/java/de/tum/cit/aet/artemis/core/domain/LLMTokenUsage.java
Outdated
Show resolved
Hide resolved
src/main/java/de/tum/cit/aet/artemis/core/service/LLMTokenUsageService.java
Outdated
Show resolved
Hide resolved
bassner
dismissed stale reviews from dmytropolityka, FelixTJDietrich, krusche, and coderabbitai[bot]
via
8671e35
bassner
force-pushed
the
feature/track-usage-of-iris-requests
branch
from
8671e35
to
79b1b88
Compare
There was a problem hiding this comment.
Caution
Inline review comments failed to post. This is likely due to GitHub's limits when posting large numbers of comments.
Actionable comments posted: 8
🧹 Outside diff range and nitpick comments (12)
src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/job/TrackedSessionBasedPyrisJob.java (2)
3-6: Consider enhancing JavaDoc documentation.While the documentation is clear, consider adding:
@sincetag for version tracking- Documentation about the relationship with
PyrisJobinterface/** * A Pyris job that has a session id and stored its own LLM usage tracing ID. * This is used for chat jobs where we need to reference the trace ID later after chat suggestions have been generated. + * + * @since 1.0.0 + * @see PyrisJob The parent interface that this extends */
7-14: Consider addressing type consistency and documentation.
Type consistency:
sessionId()returns primitivelongwhiletraceId()returns wrapperLong. Consider using consistent types unless there's a specific reason for nullable trace IDs.Missing method documentation: Add JavaDoc for each method explaining:
- Parameters
- Return values
- Nullable conditions
- Any validation requirements
public interface TrackedSessionBasedPyrisJob extends PyrisJob { + /** + * Gets the session ID associated with this job. + * + * @return the session ID + */ long sessionId(); + /** + * Gets the trace ID used for LLM usage tracking. + * + * @return the trace ID, may be null if not yet assigned + */ Long traceId(); + /** + * Creates a new instance of this job with the specified trace ID. + * + * @param traceId the trace ID to assign + * @return a new instance with the specified trace ID + */ TrackedSessionBasedPyrisJob withTraceId(long traceId); }src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/job/ExerciseChatJob.java (1)
25-28: Consider adding traceId validation.The implementation correctly follows immutable pattern and KISS principle. However, consider validating the traceId parameter to prevent potential issues with negative values.
@Override public TrackedSessionBasedPyrisJob withTraceId(long traceId) { + if (traceId <= 0) { + throw new IllegalArgumentException("Trace ID must be positive"); + } return new ExerciseChatJob(jobId, courseId, exerciseId, sessionId, traceId); }src/main/java/de/tum/cit/aet/artemis/iris/service/IrisCompetencyGenerationService.java (2)
32-41: Consider grouping related fields togetherFor better code organization and readability, consider grouping related fields together. For example, group repositories together and services together.
Apply this diff to improve field organization:
private final PyrisPipelineService pyrisPipelineService; private final LLMTokenUsageService llmTokenUsageService; - private final CourseRepository courseRepository; private final IrisWebsocketService websocketService; private final PyrisJobService pyrisJobService; + private final CourseRepository courseRepository; private final UserRepository userRepository;
78-87: Consider removing unnecessary return valueThe method returns the unmodified job object, which doesn't provide any additional value to the caller.
Consider changing the method signature to void since the job object isn't modified:
- public CompetencyExtractionJob handleStatusUpdate(CompetencyExtractionJob job, PyrisCompetencyStatusUpdateDTO statusUpdate) { + public void handleStatusUpdate(CompetencyExtractionJob job, PyrisCompetencyStatusUpdateDTO statusUpdate) { // ... existing code ... - return job; }src/main/java/de/tum/cit/aet/artemis/iris/service/session/AbstractIrisChatSessionService.java (1)
117-117: Add JavaDoc for the abstract method.The abstract method lacks documentation explaining its purpose and implementation requirements for subclasses.
Add JavaDoc:
/** * Sets specific parameters for LLM token usage tracking based on the session type. * Subclasses must implement this method to provide session-specific token usage parameters. * * @param builder The builder to configure with additional parameters * @param session The chat session containing context for the parameters * @return The configured builder */ protected abstract void setLLMTokenUsageParameters(LLMTokenUsageService.LLMTokenUsageBuilder builder, S session);src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/PyrisJobService.java (1)
110-122: Add parameter validation and improve documentation.While the methods are concise and follow the single responsibility principle, they lack parameter validation and could benefit from more detailed documentation.
Consider these improvements:
/** * Remove a job from the job map. * * @param job the job to remove + * @throws IllegalArgumentException if the job is null */ public void removeJob(PyrisJob job) { + if (job == null) { + throw new IllegalArgumentException("Job cannot be null"); + } jobMap.remove(job.jobId()); } /** * Store a job in the job map. * * @param job the job to store + * @throws IllegalArgumentException if the job is null */ public void updateJob(PyrisJob job) { + if (job == null) { + throw new IllegalArgumentException("Job cannot be null"); + } jobMap.put(job.jobId(), job); }src/main/java/de/tum/cit/aet/artemis/iris/service/session/IrisTextExerciseChatSessionService.java (3)
119-119: Update JavaDoc to reflect the return value.The method signature change to return
TextExerciseChatJoblooks good, but the JavaDoc should be updated to document the return value.Add the
@returntag to the JavaDoc:/** * Handles the status update of a text exercise chat job. * * @param job The job that is updated * @param statusUpdate The status update + * @return The updated job */Also applies to: 132-132
Line range hint
120-131: Add error handling for external service calls.The implementation looks good, but it's missing error handling for the external service calls (
irisMessageService.saveMessageandirisChatWebsocketService.sendMessage). These operations could fail and throw exceptions.Consider wrapping the external calls in try-catch blocks:
if (statusUpdate.result() != null) { var message = session.newMessage(); message.addContent(new IrisTextMessageContent(statusUpdate.result())); - IrisMessage savedMessage = irisMessageService.saveMessage(message, session, IrisMessageSender.LLM); - irisChatWebsocketService.sendMessage(session, savedMessage, statusUpdate.stages()); + try { + IrisMessage savedMessage = irisMessageService.saveMessage(message, session, IrisMessageSender.LLM); + irisChatWebsocketService.sendMessage(session, savedMessage, statusUpdate.stages()); + } catch (Exception e) { + // Log the error and potentially throw a custom exception + log.error("Failed to save or send message: {}", e.getMessage(), e); + throw new IrisMessageHandlingException("Failed to process status update", e); + } } else { - irisChatWebsocketService.sendMessage(session, null, statusUpdate.stages()); + try { + irisChatWebsocketService.sendMessage(session, null, statusUpdate.stages()); + } catch (Exception e) { + log.error("Failed to send message: {}", e.getMessage(), e); + throw new IrisMessageHandlingException("Failed to send status update", e); + } }🧰 Tools
🪛 ast-grep
[warning] 123-123: Detected a cookie where the
HttpOnlyflag is either missing or disabled. TheHttpOnlycookie flag instructs the browser to forbid client-side JavaScript to read the cookie. If JavaScript interaction is required, you can ignore this finding. However, set theHttpOnlyflag to true` in all other cases.
Context: (new IrisTextMessageContent(statusUpdate.result()))
Note: [CWE-1004]: Sensitive Cookie Without 'HttpOnly' Flag [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
[warning] 123-123: Detected a cookie where the
Secureflag is either missing or disabled. TheSecurecookie flag instructs the browser to forbid sending the cookie over an insecure HTTP request. Set theSecureflag totrueso the cookie will only be sent over HTTPS.
Context: (new IrisTextMessageContent(statusUpdate.result()))
Note: [CWE-614]: Sensitive Cookie in HTTPS Session Without 'Secure' Attribute [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
120-120: Consider implementing the suggested inheritance refactoring.The TODO comment suggests making this class a subclass of
AbstractIrisChatSessionService. This refactoring could improve code reuse and standardize token tracking across different chat services.Would you like me to help create a GitHub issue to track this refactoring task? I can provide a detailed plan for:
- Creating the abstract base class
- Moving common functionality
- Implementing token tracking consistently
src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/PyrisStatusUpdateService.java (2)
57-58: Correct the grammatical error in the method documentationThe documentation should use "an exercise chat job" instead of "a exercise chat job".
Apply this diff to fix the typo:
/** - * Handles the status update of a exercise chat job and forwards it to + * Handles the status update of an exercise chat job and forwards it to * {@link IrisExerciseChatSessionService#handleStatusUpdate(TrackedSessionBasedPyrisJob, PyrisChatStatusUpdateDTO)}
135-137: Address the TODO comment inhandleStatusUpdateforIngestionWebhookJobThere's a TODO comment indicating that this method should be updated to handle changes beyond logging. Consider implementing the required functionality.
Would you like assistance in generating the updated implementation or opening a GitHub issue to track this task?
🛑 Comments failed to post (8)
src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/job/CourseChatJob.java (1)
12-12: 🛠️ Refactor suggestion
Consider using primitive
longinstead ofLongfor traceId.Since tracing is a core feature for token usage tracking, consider using the primitive
longtype instead ofLongto:
- Prevent potential null pointer exceptions
- Reduce memory overhead
- Enforce mandatory tracing
-public record CourseChatJob(String jobId, long courseId, long sessionId, Long traceId) implements TrackedSessionBasedPyrisJob { +public record CourseChatJob(String jobId, long courseId, long sessionId, long traceId) implements TrackedSessionBasedPyrisJob {📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.public record CourseChatJob(String jobId, long courseId, long sessionId, long traceId) implements TrackedSessionBasedPyrisJob {src/main/java/de/tum/cit/aet/artemis/iris/service/IrisCompetencyGenerationService.java (2)
79-82: 🛠️ Refactor suggestion
Optimize database query and simplify token check
The course is retrieved even when tokens might be null or empty. Also, the token check can be simplified.
Apply this diff to optimize the code:
- Course course = courseRepository.findByIdForUpdateElseThrow(job.courseId()); - if (statusUpdate.tokens() != null && !statusUpdate.tokens().isEmpty()) { + if (statusUpdate.tokens() != null && !statusUpdate.tokens().isEmpty()) { + Course course = courseRepository.findByIdForUpdateElseThrow(job.courseId()); llmTokenUsageService.saveLLMTokenUsage(statusUpdate.tokens(), LLMServiceType.IRIS, builder -> builder.withCourse(course.getId()).withUser(job.userId())); }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.if (statusUpdate.tokens() != null && !statusUpdate.tokens().isEmpty()) { Course course = courseRepository.findByIdForUpdateElseThrow(job.courseId()); llmTokenUsageService.saveLLMTokenUsage(statusUpdate.tokens(), LLMServiceType.IRIS, builder -> builder.withCourse(course.getId()).withUser(job.userId())); }
84-84:
⚠️ Potential issueImprove error handling for user retrieval
The current implementation might throw a generic
NoSuchElementException. Consider providing a more specific error message.Apply this diff:
- var user = userRepository.findById(job.userId()).orElseThrow(); + var user = userRepository.findById(job.userId()) + .orElseThrow(() -> new IllegalStateException("User not found for job: " + job.userId()));📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.var user = userRepository.findById(job.userId()) .orElseThrow(() -> new IllegalStateException("User not found for job: " + job.userId()));src/main/java/de/tum/cit/aet/artemis/iris/service/session/AbstractIrisChatSessionService.java (2)
71-115: 🛠️ Refactor suggestion
Consider breaking down the handleStatusUpdate method.
The method is handling multiple responsibilities which makes it harder to maintain and test. Consider extracting the following into separate methods:
- Message handling logic
- Token usage tracking logic
- Trace management logic
Here's a suggested refactoring approach:
public TrackedSessionBasedPyrisJob handleStatusUpdate(TrackedSessionBasedPyrisJob job, PyrisChatStatusUpdateDTO statusUpdate) { var session = getAndValidateSession(job.sessionId()); var savedMessage = handleMessageUpdate(session, statusUpdate); var updatedJob = handleTokenUsage(job, statusUpdate, session, savedMessage); updateLatestSuggestions(session, statusUpdate.suggestions()); return updatedJob; } private S getAndValidateSession(String sessionId) { var session = (S) irisSessionRepository.findByIdWithMessagesAndContents(sessionId); if (session == null) { throw new IllegalStateException("Session not found with id: " + sessionId); } return session; } private IrisMessage handleMessageUpdate(S session, PyrisChatStatusUpdateDTO statusUpdate) { if (statusUpdate.result() != null) { var message = new IrisMessage(); message.addContent(new IrisTextMessageContent(statusUpdate.result())); var savedMessage = irisMessageService.saveMessage(message, session, IrisMessageSender.LLM); irisChatWebsocketService.sendMessage(session, savedMessage, statusUpdate.stages()); return savedMessage; } irisChatWebsocketService.sendStatusUpdate(session, statusUpdate.stages(), statusUpdate.suggestions(), statusUpdate.tokens()); return null; } private TrackedSessionBasedPyrisJob handleTokenUsage(TrackedSessionBasedPyrisJob job, PyrisChatStatusUpdateDTO statusUpdate, S session, IrisMessage savedMessage) { if (statusUpdate.tokens() == null || statusUpdate.tokens().isEmpty()) { return job; } return savedMessage != null ? handleNewTrace(job, statusUpdate, session, savedMessage) : handleExistingTrace(job, statusUpdate, session); }🧰 Tools
🪛 ast-grep
[warning] 75-75: Detected a cookie where the
HttpOnlyflag is either missing or disabled. TheHttpOnlycookie flag instructs the browser to forbid client-side JavaScript to read the cookie. If JavaScript interaction is required, you can ignore this finding. However, set theHttpOnlyflag to true` in all other cases.
Context: (new IrisTextMessageContent(statusUpdate.result()))
Note: [CWE-1004]: Sensitive Cookie Without 'HttpOnly' Flag [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
[warning] 75-75: Detected a cookie where the
Secureflag is either missing or disabled. TheSecurecookie flag instructs the browser to forbid sending the cookie over an insecure HTTP request. Set theSecureflag totrueso the cookie will only be sent over HTTPS.
Context: (new IrisTextMessageContent(statusUpdate.result()))
Note: [CWE-614]: Sensitive Cookie in HTTPS Session Without 'Secure' Attribute [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
71-73:
⚠️ Potential issueAdd null check for session retrieval.
The session retrieval could return null, but there's no null check before using it.
Add a null check:
var session = (S) irisSessionRepository.findByIdWithMessagesAndContents(job.sessionId()); +if (session == null) { + throw new IllegalStateException("Session not found with id: " + job.sessionId()); +}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.public TrackedSessionBasedPyrisJob handleStatusUpdate(TrackedSessionBasedPyrisJob job, PyrisChatStatusUpdateDTO statusUpdate) { var session = (S) irisSessionRepository.findByIdWithMessagesAndContents(job.sessionId()); if (session == null) { throw new IllegalStateException("Session not found with id: " + job.sessionId()); } IrisMessage savedMessage;src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/PyrisJobService.java (2)
81-81:
⚠️ Potential issueUpdate method signature and avoid passing null directly.
The code passes
nulldirectly to theExerciseChatJobconstructor, which is a code smell and could lead to NullPointerException. Additionally, the method signature hasn't been updated to match the constructor change.Consider these improvements:
- Update the method signature to include the new parameter
- Add parameter validation
- Update JavaDoc
- public String addExerciseChatJob(Long courseId, Long exerciseId, Long sessionId) { + /** + * Creates a new exercise chat job. + * + * @param courseId the ID of the course + * @param exerciseId the ID of the exercise + * @param sessionId the ID of the session + * @param tokenUsageData the token usage tracking data (can be null) + * @return the generated job token + */ + public String addExerciseChatJob(Long courseId, Long exerciseId, Long sessionId, PyrisLLMCostDTO tokenUsageData) { var token = generateJobIdToken(); - var job = new ExerciseChatJob(token, courseId, exerciseId, sessionId, null); + var job = new ExerciseChatJob(token, courseId, exerciseId, sessionId, tokenUsageData); jobMap.put(token, job); return token; }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements./** * Creates a new exercise chat job. * * @param courseId the ID of the course * @param exerciseId the ID of the exercise * @param sessionId the ID of the session * @param tokenUsageData the token usage tracking data (can be null) * @return the generated job token */ public String addExerciseChatJob(Long courseId, Long exerciseId, Long sessionId, PyrisLLMCostDTO tokenUsageData) { var token = generateJobIdToken(); var job = new ExerciseChatJob(token, courseId, exerciseId, sessionId, tokenUsageData); jobMap.put(token, job); return token; }
88-88:
⚠️ Potential issueUpdate method signature and avoid passing null directly.
Similar to
addExerciseChatJob, this method passesnulldirectly and needs signature updates.Apply similar improvements:
- public String addCourseChatJob(Long courseId, Long sessionId) { + /** + * Creates a new course chat job. + * + * @param courseId the ID of the course + * @param sessionId the ID of the session + * @param tokenUsageData the token usage tracking data (can be null) + * @return the generated job token + */ + public String addCourseChatJob(Long courseId, Long sessionId, PyrisLLMCostDTO tokenUsageData) { var token = generateJobIdToken(); - var job = new CourseChatJob(token, courseId, sessionId, null); + var job = new CourseChatJob(token, courseId, sessionId, tokenUsageData); jobMap.put(token, job); return token; }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements./** * Creates a new course chat job. * * @param courseId the ID of the course * @param sessionId the ID of the session * @param tokenUsageData the token usage tracking data (can be null) * @return the generated job token */ public String addCourseChatJob(Long courseId, Long sessionId, PyrisLLMCostDTO tokenUsageData) { var token = generateJobIdToken(); var job = new CourseChatJob(token, courseId, sessionId, tokenUsageData); jobMap.put(token, job); return token; }src/main/java/de/tum/cit/aet/artemis/core/service/LLMTokenUsageService.java (1)
99-141: 🛠️ Refactor suggestion
Standardize the Use of 'Id' in Variable and Method Names for Consistency
The code uses both
IDandIdin variable and method names within theLLMTokenUsageBuilderclass (e.g.,courseIDvs.courseId). According to Java naming conventions,Idis preferred overID. For consistency and readability, consider standardizing all variable and method names to useId.Apply this diff to update the naming convention:
- private Optional<Long> courseID = Optional.empty(); + private Optional<Long> courseId = Optional.empty(); - private Optional<Long> irisMessageID = Optional.empty(); + private Optional<Long> irisMessageId = Optional.empty(); - private Optional<Long> exerciseID = Optional.empty(); + private Optional<Long> exerciseId = Optional.empty(); - private Optional<Long> userID = Optional.empty(); + private Optional<Long> userId = Optional.empty(); - public LLMTokenUsageBuilder withCourse(Long courseID) { + public LLMTokenUsageBuilder withCourse(Long courseId) { - public LLMTokenUsageBuilder withIrisMessageID(Long irisMessageID) { + public LLMTokenUsageBuilder withIrisMessageId(Long irisMessageId) { - public LLMTokenUsageBuilder withExercise(Long exerciseID) { + public LLMTokenUsageBuilder withExercise(Long exerciseId) { - public LLMTokenUsageBuilder withUser(Long userID) { + public LLMTokenUsageBuilder withUser(Long userId) { - public Optional<Long> getCourseID() { + public Optional<Long> getCourseId() { - public Optional<Long> getIrisMessageID() { + public Optional<Long> getIrisMessageId() { - public Optional<Long> getExerciseID() { + public Optional<Long> getExerciseId() { - public Optional<Long> getUserID() { + public Optional<Long> getUserId() {Also, ensure that all references to these variables and methods are updated accordingly throughout the codebase.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.private Optional<Long> courseId = Optional.empty(); private Optional<Long> irisMessageId = Optional.empty(); private Optional<Long> exerciseId = Optional.empty(); private Optional<Long> userId = Optional.empty(); public LLMTokenUsageBuilder withCourse(Long courseId) { this.courseId = Optional.ofNullable(courseId); return this; } public LLMTokenUsageBuilder withIrisMessageId(Long irisMessageId) { this.irisMessageId = Optional.ofNullable(irisMessageId); return this; } public LLMTokenUsageBuilder withExercise(Long exerciseId) { this.exerciseId = Optional.ofNullable(exerciseId); return this; } public LLMTokenUsageBuilder withUser(Long userId) { this.userId = Optional.ofNullable(userId); return this; } public Optional<Long> getCourseId() { return courseId; } public Optional<Long> getIrisMessageId() { return irisMessageId; } public Optional<Long> getExerciseId() { return exerciseId; } public Optional<Long> getUserId() { return userId; }
There was a problem hiding this comment.
Actionable comments posted: 7
🧹 Outside diff range and nitpick comments (7)
src/main/java/de/tum/cit/aet/artemis/iris/service/IrisCompetencyGenerationService.java (2)
42-50: Reorder field assignments to match constructor parameter order.The field assignments in the constructor should follow the same order as the parameters for better readability and maintainability.
Apply this diff:
public IrisCompetencyGenerationService(PyrisPipelineService pyrisPipelineService, LLMTokenUsageService llmTokenUsageService, CourseRepository courseRepository, IrisWebsocketService websocketService, PyrisJobService pyrisJobService, UserRepository userRepository) { this.pyrisPipelineService = pyrisPipelineService; - this.llmTokenUsageService = llmTokenUsageService; - this.courseRepository = courseRepository; - this.websocketService = websocketService; - this.pyrisJobService = pyrisJobService; - this.userRepository = userRepository; + this.llmTokenUsageService = llmTokenUsageService; + this.courseRepository = courseRepository; + this.websocketService = websocketService; + this.pyrisJobService = pyrisJobService; + this.userRepository = userRepository; }
84-84: Improve error handling for user lookup.The
orElseThrow()without a custom message makes debugging harder. Consider adding a descriptive error message.Apply this diff:
-var user = userRepository.findById(job.userId()).orElseThrow(); +var user = userRepository.findById(job.userId()) + .orElseThrow(() -> new IllegalStateException("User not found for ID: " + job.userId()));src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/PyrisStatusUpdateService.java (4)
64-64: Consider adding final modifier to local variables.The job handling logic is well-structured and consistent across all handlers. Consider making the local variables final for better immutability:
-var updatedJob = irisExerciseChatSessionService.handleStatusUpdate(job, statusUpdate); +final var updatedJob = irisExerciseChatSessionService.handleStatusUpdate(job, statusUpdate);Also applies to: 66-66, 77-77, 79-79, 90-90, 92-92, 103-103, 105-105
109-125: Consider extracting the terminal state check to a private method.The method is well-documented and follows the single responsibility principle. To improve readability further, consider extracting the terminal state check:
private void removeJobIfTerminatedElseUpdate(List<PyrisStageDTO> stages, PyrisJob job) { - var isDone = stages.stream().map(PyrisStageDTO::state).allMatch(PyrisStageState::isTerminal); + var isDone = isJobTerminated(stages); if (isDone) { pyrisJobService.removeJob(job); } else { pyrisJobService.updateJob(job); } } +private boolean isJobTerminated(List<PyrisStageDTO> stages) { + return stages.stream() + .map(PyrisStageDTO::state) + .allMatch(PyrisStageState::isTerminal); +}
137-137: Track the TODO comment for future implementation.The method correctly uses the new job handling logic, but there's a TODO comment indicating that this method needs to be updated beyond logging.
Would you like me to create a GitHub issue to track the TODO for implementing the complete lecture ingestion handling?
Based on the verification results, I can now provide a final response:
TextExerciseChatJob needs to implement TrackedSessionBasedPyrisJob interface
The verification reveals inconsistencies in the implementation:
- ExerciseChatJob and CourseChatJob correctly implement TrackedSessionBasedPyrisJob
- TextExerciseChatJob only implements PyrisJob, despite being used in similar session-based contexts
- CompetencyExtractionJob and IngestionWebhookJob correctly implement PyrisJob as they are not session-based
Required changes:
- TextExerciseChatJob should implement TrackedSessionBasedPyrisJob interface
- Add sessionId (already present) and traceId fields
- Implement withTraceId method as required by the interface
🔗 Analysis chain
Line range hint
1-138: Verify consistent implementation of TrackedSessionBasedPyrisJob.Let's verify that all relevant job types properly implement the new interface and maintain consistent patterns.
Let me generate new shell scripts with corrected syntax to verify the implementation.
Let me generate one final verification script to check the remaining job implementations.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify consistent implementation of TrackedSessionBasedPyrisJob interface # Test 1: Check job implementations echo "Checking job implementations..." ast-grep --pattern 'class $_ implements TrackedSessionBasedPyrisJob { $$$ }' # Test 2: Check for any remaining direct PyrisJob usage that should be TrackedSessionBasedPyrisJob echo "Checking for potential missed conversions..." rg "PyrisJob(?!.*TrackedSessionBasedPyrisJob)" -g "*.java"Length of output: 690
Script:
#!/bin/bash # Let's break this down into multiple checks # 1. Find all files that mention TrackedSessionBasedPyrisJob echo "=== Files referencing TrackedSessionBasedPyrisJob ===" rg "TrackedSessionBasedPyrisJob" --type java # 2. Find all job class definitions echo -e "\n=== Job class definitions ===" rg "class \w+(?:Chat|Extraction|Webhook)Job" --type java # 3. Check PyrisJob interface/class definition echo -e "\n=== PyrisJob definition ===" rg "^(interface|class) PyrisJob" --type java -A 5 # 4. Check service implementations handling these jobs echo -e "\n=== Service methods handling jobs ===" rg "handleStatusUpdate.*Job" --type java -A 2 # 5. Check for job type usage in PyrisStatusUpdateService echo -e "\n=== Job type usage in status service ===" rg "(?:Chat|Extraction|Webhook)Job" src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/PyrisStatusUpdateService.javaLength of output: 10513
Script:
#!/bin/bash # Check implementation details of remaining job types # 1. Check TextExerciseChatJob implementation echo "=== TextExerciseChatJob Implementation ===" rg "record TextExerciseChatJob" --type java -A 5 # 2. Check CompetencyExtractionJob implementation echo -e "\n=== CompetencyExtractionJob Implementation ===" rg "record CompetencyExtractionJob" --type java -A 5 # 3. Check IngestionWebhookJob implementation echo -e "\n=== IngestionWebhookJob Implementation ===" rg "record IngestionWebhookJob" --type java -A 5 # 4. Check TrackedSessionBasedPyrisJob interface definition echo -e "\n=== TrackedSessionBasedPyrisJob Interface Definition ===" rg "interface TrackedSessionBasedPyrisJob" --type java -A 10Length of output: 3730
src/main/java/de/tum/cit/aet/artemis/core/service/LLMTokenUsageService.java (1)
36-48: Enhance JavaDoc documentationThe JavaDoc could be improved for better clarity and completeness.
Add more detailed documentation:
/** * Saves the token usage to the database. * This method records the usage of tokens by various LLM services in the system. + * The method executes asynchronously to prevent blocking the main operation flow. * * @param llmRequests List of LLM requests containing details about the token usage. * @param serviceType Type of the LLM service (e.g., IRIS, GPT-3). * @param builderFunction A function that takes an LLMTokenUsageBuilder and returns a modified LLMTokenUsageBuilder. * This function is used to set additional properties on the LLMTokenUsageTrace object, such as * the course ID, user ID, exercise ID, and Iris message ID. * Example usage: - * builder -> builder.withCourse(courseId).withUser(userId) + * builder -> builder.withCourse(courseId) + * .withUser(userId) + * .withMessageId(messageId) + * .withExercise(exerciseId) * @return The saved LLMTokenUsageTrace object, which includes the details of the token usage. + * @throws IllegalArgumentException if llmRequests is null or empty */
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
📒 Files selected for processing (9)
- src/main/java/de/tum/cit/aet/artemis/core/service/LLMTokenUsageService.java (1 hunks)
- src/main/java/de/tum/cit/aet/artemis/iris/service/IrisCompetencyGenerationService.java (3 hunks)
- src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/PyrisJobService.java (2 hunks)
- src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/PyrisStatusUpdateService.java (4 hunks)
- src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/job/CourseChatJob.java (1 hunks)
- src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/job/ExerciseChatJob.java (2 hunks)
- src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/job/TrackedSessionBasedPyrisJob.java (1 hunks)
- src/main/java/de/tum/cit/aet/artemis/iris/service/session/AbstractIrisChatSessionService.java (2 hunks)
- src/main/java/de/tum/cit/aet/artemis/iris/service/session/IrisTextExerciseChatSessionService.java (2 hunks)
🚧 Files skipped from review as they are similar to previous changes (5)
- src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/PyrisJobService.java
- src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/job/CourseChatJob.java
- src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/job/ExerciseChatJob.java
- src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/job/TrackedSessionBasedPyrisJob.java
- src/main/java/de/tum/cit/aet/artemis/iris/service/session/IrisTextExerciseChatSessionService.java
🧰 Additional context used
📓 Path-based instructions (4)
src/main/java/de/tum/cit/aet/artemis/core/service/LLMTokenUsageService.java (1)
Pattern
src/main/java/**/*.java: naming:CamelCase; principles:{single_responsibility,small_methods,no_duplication}; db:{perf_queries,datetime_not_timestamp}; rest:{stateless,singleton,delegate_logic,http_only,minimal_dtos}; dtos:{java_records,no_entities,min_data,single_resp}; di:constructor_injection; kiss:simple_code; file_handling:os_indep_paths; practices:{least_access,avoid_transactions,code_reuse,static_member_ref,prefer_primitives}; sql:{param_annotation,uppercase,avoid_subqueries};java:avoid_star_importssrc/main/java/de/tum/cit/aet/artemis/iris/service/IrisCompetencyGenerationService.java (1)
Pattern
src/main/java/**/*.java: naming:CamelCase; principles:{single_responsibility,small_methods,no_duplication}; db:{perf_queries,datetime_not_timestamp}; rest:{stateless,singleton,delegate_logic,http_only,minimal_dtos}; dtos:{java_records,no_entities,min_data,single_resp}; di:constructor_injection; kiss:simple_code; file_handling:os_indep_paths; practices:{least_access,avoid_transactions,code_reuse,static_member_ref,prefer_primitives}; sql:{param_annotation,uppercase,avoid_subqueries};java:avoid_star_importssrc/main/java/de/tum/cit/aet/artemis/iris/service/pyris/PyrisStatusUpdateService.java (1)
Pattern
src/main/java/**/*.java: naming:CamelCase; principles:{single_responsibility,small_methods,no_duplication}; db:{perf_queries,datetime_not_timestamp}; rest:{stateless,singleton,delegate_logic,http_only,minimal_dtos}; dtos:{java_records,no_entities,min_data,single_resp}; di:constructor_injection; kiss:simple_code; file_handling:os_indep_paths; practices:{least_access,avoid_transactions,code_reuse,static_member_ref,prefer_primitives}; sql:{param_annotation,uppercase,avoid_subqueries};java:avoid_star_importssrc/main/java/de/tum/cit/aet/artemis/iris/service/session/AbstractIrisChatSessionService.java (1)
Pattern
src/main/java/**/*.java: naming:CamelCase; principles:{single_responsibility,small_methods,no_duplication}; db:{perf_queries,datetime_not_timestamp}; rest:{stateless,singleton,delegate_logic,http_only,minimal_dtos}; dtos:{java_records,no_entities,min_data,single_resp}; di:constructor_injection; kiss:simple_code; file_handling:os_indep_paths; practices:{least_access,avoid_transactions,code_reuse,static_member_ref,prefer_primitives}; sql:{param_annotation,uppercase,avoid_subqueries};java:avoid_star_imports
🪛 ast-grep
src/main/java/de/tum/cit/aet/artemis/core/service/LLMTokenUsageService.java
[warning] 53-53: Detected a cookie where the
HttpOnlyflag is either missing or disabled. TheHttpOnlycookie flag instructs the browser to forbid client-side JavaScript to read the cookie. If JavaScript interaction is required, you can ignore this finding. However, set theHttpOnlyflag to true` in all other cases.
Context: (new LLMTokenUsageBuilder())
Note: [CWE-1004]: Sensitive Cookie Without 'HttpOnly' Flag [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
[warning] 53-53: Detected a cookie where the
Secureflag is either missing or disabled. TheSecurecookie flag instructs the browser to forbid sending the cookie over an insecure HTTP request. Set theSecureflag totrueso the cookie will only be sent over HTTPS.
Context: (new LLMTokenUsageBuilder())
Note: [CWE-614]: Sensitive Cookie in HTTPS Session Without 'Secure' Attribute [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfigurationsrc/main/java/de/tum/cit/aet/artemis/iris/service/IrisCompetencyGenerationService.java
[warning] 66-66: Detected a cookie where the
HttpOnlyflag is either missing or disabled. TheHttpOnlycookie flag instructs the browser to forbid client-side JavaScript to read the cookie. If JavaScript interaction is required, you can ignore this finding. However, set theHttpOnlyflag to true` in all other cases.
Context: (user.getLogin(), websocketTopic(course.getId()), new PyrisCompetencyStatusUpdateDTO(stages, null, null))
Note: [CWE-1004]: Sensitive Cookie Without 'HttpOnly' Flag [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
[warning] 66-66: Detected a cookie where the
Secureflag is either missing or disabled. TheSecurecookie flag instructs the browser to forbid sending the cookie over an insecure HTTP request. Set theSecureflag totrueso the cookie will only be sent over HTTPS.
Context: (user.getLogin(), websocketTopic(course.getId()), new PyrisCompetencyStatusUpdateDTO(stages, null, null))
Note: [CWE-614]: Sensitive Cookie in HTTPS Session Without 'Secure' Attribute [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfigurationsrc/main/java/de/tum/cit/aet/artemis/iris/service/session/AbstractIrisChatSessionService.java
[warning] 76-76: Detected a cookie where the
HttpOnlyflag is either missing or disabled. TheHttpOnlycookie flag instructs the browser to forbid client-side JavaScript to read the cookie. If JavaScript interaction is required, you can ignore this finding. However, set theHttpOnlyflag to true` in all other cases.
Context: (new IrisTextMessageContent(statusUpdate.result()))
Note: [CWE-1004]: Sensitive Cookie Without 'HttpOnly' Flag [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
[warning] 76-76: Detected a cookie where the
Secureflag is either missing or disabled. TheSecurecookie flag instructs the browser to forbid sending the cookie over an insecure HTTP request. Set theSecureflag totrueso the cookie will only be sent over HTTPS.
Context: (new IrisTextMessageContent(statusUpdate.result()))
Note: [CWE-614]: Sensitive Cookie in HTTPS Session Without 'Secure' Attribute [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
🔇 Additional comments (6)
src/main/java/de/tum/cit/aet/artemis/iris/service/session/AbstractIrisChatSessionService.java (3)
4-5: LGTM: Well-structured field declarations and imports.The new fields follow proper encapsulation, immutability, and dependency injection principles. The imports are organized logically and avoid star imports as per coding guidelines.
Also applies to: 10-14, 17-20, 26-31
34-40: LGTM: Constructor properly implements dependency injection.The constructor follows best practices by using constructor injection and properly initializing all fields.
118-118: LGTM: Well-designed abstract method.The abstract method follows the template method pattern and has appropriate access level. The method name and parameters clearly convey its purpose.
src/main/java/de/tum/cit/aet/artemis/iris/service/pyris/PyrisStatusUpdateService.java (2)
23-23: LGTM: Import statements follow conventions.The new imports are properly organized and follow the coding guidelines.
Also applies to: 25-25
57-58: LGTM: JavaDoc updates are accurate and well-maintained.The documentation properly reflects the parameter type changes and maintains correct method references.
Also applies to: 84-84
src/main/java/de/tum/cit/aet/artemis/core/service/LLMTokenUsageService.java (1)
1-34: LGTM! Class structure follows Spring best practices.The service is properly configured with constructor injection and appropriate annotations.
src/main/java/de/tum/cit/aet/artemis/iris/service/IrisCompetencyGenerationService.java
Show resolved
Hide resolved
src/main/java/de/tum/cit/aet/artemis/iris/service/IrisCompetencyGenerationService.java
Show resolved
Hide resolved
src/main/java/de/tum/cit/aet/artemis/iris/service/session/AbstractIrisChatSessionService.java
Show resolved
Hide resolved
src/main/java/de/tum/cit/aet/artemis/iris/service/session/AbstractIrisChatSessionService.java
Show resolved
Hide resolved
src/main/java/de/tum/cit/aet/artemis/core/service/LLMTokenUsageService.java
Show resolved
Hide resolved
src/main/java/de/tum/cit/aet/artemis/core/service/LLMTokenUsageService.java
Show resolved
Hide resolved
src/main/java/de/tum/cit/aet/artemis/core/service/LLMTokenUsageService.java
Show resolved
Hide resolved
bassner
temporarily deployed
to
artemis-test3.artemis.cit.tum.de
GitHub Actions
Inactive
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Outside diff range and nitpick comments (2)
src/main/java/de/tum/cit/aet/artemis/iris/service/IrisCompetencyGenerationService.java (2)
32-40: Consider reordering fields for better readabilityThe field declarations could be grouped more logically. Consider keeping related fields together:
- Pipeline-related: pyrisPipelineService
- Repository-related: courseRepository, userRepository
- Service-related: llmTokenUsageService, websocketService, pyrisJobService
79-83: Consider adding transaction managementThe method performs multiple database operations (course and token usage). Consider adding transaction management to ensure data consistency:
+ @Transactional public CompetencyExtractionJob handleStatusUpdate(CompetencyExtractionJob job, PyrisCompetencyStatusUpdateDTO statusUpdate) {This ensures that both the course retrieval and token usage saving are executed within the same transaction.
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
📒 Files selected for processing (2)
- src/main/java/de/tum/cit/aet/artemis/iris/service/IrisCompetencyGenerationService.java (3 hunks)
- src/main/java/de/tum/cit/aet/artemis/iris/service/session/IrisTextExerciseChatSessionService.java (2 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- src/main/java/de/tum/cit/aet/artemis/iris/service/session/IrisTextExerciseChatSessionService.java
🧰 Additional context used
📓 Path-based instructions (1)
src/main/java/de/tum/cit/aet/artemis/iris/service/IrisCompetencyGenerationService.java (1)
Pattern
src/main/java/**/*.java: naming:CamelCase; principles:{single_responsibility,small_methods,no_duplication}; db:{perf_queries,datetime_not_timestamp}; rest:{stateless,singleton,delegate_logic,http_only,minimal_dtos}; dtos:{java_records,no_entities,min_data,single_resp}; di:constructor_injection; kiss:simple_code; file_handling:os_indep_paths; practices:{least_access,avoid_transactions,code_reuse,static_member_ref,prefer_primitives}; sql:{param_annotation,uppercase,avoid_subqueries};java:avoid_star_imports
🪛 ast-grep
src/main/java/de/tum/cit/aet/artemis/iris/service/IrisCompetencyGenerationService.java
[warning] 66-66: Detected a cookie where the
Secureflag is either missing or disabled. TheSecurecookie flag instructs the browser to forbid sending the cookie over an insecure HTTP request. Set theSecureflag totrueso the cookie will only be sent over HTTPS.
Context: (user.getLogin(), websocketTopic(course.getId()), new PyrisCompetencyStatusUpdateDTO(stages, null, null))
Note: [CWE-614]: Sensitive Cookie in HTTPS Session Without 'Secure' Attribute [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
[warning] 66-66: Detected a cookie where the
HttpOnlyflag is either missing or disabled. TheHttpOnlycookie flag instructs the browser to forbid client-side JavaScript to read the cookie. If JavaScript interaction is required, you can ignore this finding. However, set theHttpOnlyflag to true` in all other cases.
Context: (user.getLogin(), websocketTopic(course.getId()), new PyrisCompetencyStatusUpdateDTO(stages, null, null))
Note: [CWE-1004]: Sensitive Cookie Without 'HttpOnly' Flag [OWASP A05:2021]: Security Misconfiguration [REFERENCES]
- https://owasp.org/Top10/A05_2021-Security_Misconfiguration
🔇 Additional comments (1)
src/main/java/de/tum/cit/aet/artemis/iris/service/IrisCompetencyGenerationService.java (1)
42-49: LGTM! Constructor follows best practicesThe constructor properly uses dependency injection and correctly initializes all fields.
src/main/java/de/tum/cit/aet/artemis/iris/service/IrisCompetencyGenerationService.java
Show resolved
Hide resolved
Checklist
General
Server
Motivation and Context
This PR fixes parts of an open Feature request: Issue
The feature request also includes a graphical interface, but this will be implemented in a follow-up.
Description
I worked with Pyris and Athena to implement the requested feature for both LLM services. Pyris now sends tokens with a status message if tokens were used. Those tokens are of a new Type called
PyrisLLMCostDTOthat saves input and output tokens, the used LLM and also the service that used those tokens:If Artemis receives those tokens it saves them using the
LLMTokenUsageServicein two new table calledllm_token_usage_requestgrouped byllm_token_usage_trace. Those tables contains all the information related to the LLM call: model, number and cost of input and output tokens, the course, exercise, user and IrisMessage Id and a trace. The trace is used to group calls that were made for one message.Steps for Testing
Prerequisites:
Prerequisite:
Testing with Iris
feature/track-usage-of-iris-requests(related PR).llm_token_usage_requestandllm_token_usage_tracetables exist.Testing with Athena
developof Athena)llm_token_usage_requestandllm_token_usage_tracetables exist.Testserver States
Note
These badges show the state of the test servers.
Green = Currently available, Red = Currently locked
Click on the badges to get to the test servers.
Review Progress
Performance Review
Code Review
Manual Tests
Performance Tests
Test Coverage
Server
Summary by CodeRabbit
New Features
IRIS_LECTURE_RETRIEVAL_PIPELINEandIRIS_LECTURE_INGESTION.ResponseMetaDTOfor encapsulating response metadata from the Athena service.TrackedSessionBasedPyrisJobfor managing jobs requiring session tracking and LLM usage tracing.Bug Fixes
Tests