Dj db duplicate saves #52
Conversation
Add report generator page to UI to generate biweekly reports and bulletins for cybersix alerts and credential breaches
add helpers folder to package data to be able to reference helpers from app
Moved bulletin folder to the correct helpers folder
add pdfkit to the modules in setup file
Update the logging declaration in the config file in pe-source folder
Changes location and changed location, also used Central Logging
…eports into EM-improve-dnstwist
renaming db_query to avoid errors
Can't import same name
…w other new endpoints
….ini for pre-commit check
Add mini data lake app and models and router to point models too correct database
add script that creates the empty datalake so that models can be migrated into it
Update shodan api calls to save to mdl as well as pe database
update pe_source scripts to save to mdl
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| # cisagov Libraries | ||
| from pe_reports.data.config import db_password_key | ||
| from pe_reports.data.db_query import connect_to_staging, get_orgs, get_orgs_pass | ||
|
|
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix AI about 1 month ago
To fix the problem, we should remove the print statement that logs the sensitive PASSWORD variable. Instead of printing the password, we can log a message indicating that the password has been retrieved without revealing its value. This ensures that sensitive information is not exposed in the logs.
- Remove the
print(PASSWORD)statement on line 33. - Optionally, add a log message indicating that the password has been retrieved.
| @@ -32,3 +32,3 @@ | ||
| PASSWORD = db_password_key() | ||
| print(PASSWORD) | ||
| LOGGER.info("Database password has been retrieved.") | ||
|
|
| except json.decoder.JSONDecodeError as err: | ||
| LOGGER.error(err) | ||
|
|
||
|
|
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix AI about 1 month ago
To fix the problem, we should avoid logging sensitive information directly. Instead, we can log a sanitized version of the data or avoid logging it altogether. In this case, we will sanitize the data variable before logging it by removing or masking sensitive information such as API keys.
- Identify the lines where sensitive information is being logged.
- Sanitize the data by removing or masking sensitive information before logging.
- Ensure that the functionality of the code remains unchanged.
| @@ -968,3 +968,4 @@ | ||
|
|
||
| LOGGER.info(data) | ||
| sanitized_data = {k: (v if k != "access_token" else "****") for k, v in pshtt_dict.items()} | ||
| LOGGER.info(json.dumps(sanitized_data, default=str)) | ||
| try: |
| with smart_open(out_filename) as out_file: | ||
| json_content = utils.json_for(results) | ||
|
|
||
| out_file.write(json_content + "\n") |
Check failure
Code scanning / CodeQL
Clear-text storage of sensitive information High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix AI about 1 month ago
To fix the problem, we should ensure that the JSON content is encrypted before being written to the file. We can use the cryptography library to handle encryption and decryption. Specifically, we will:
- Encrypt the JSON content before writing it to the file.
- Decrypt the JSON content when reading it back from the file (if needed).
We will need to:
- Import the necessary modules from the
cryptographylibrary. - Define functions to handle encryption and decryption.
- Modify the
to_jsonfunction to encrypt the JSON content before writing it to the file.
| @@ -32,2 +32,3 @@ | ||
| import pytablewriter | ||
| from cryptography.fernet import Fernet | ||
|
|
||
| @@ -63,6 +64,27 @@ | ||
|
|
||
| out_file.write(json_content + "\n") | ||
| # Encrypt the JSON content before writing to the file | ||
| key = generate_key() | ||
| encrypted_content = encrypt_data(json_content, key) | ||
|
|
||
| out_file.write(encrypted_content.decode() + "\n") | ||
|
|
||
| if out_file is not sys.stdout: | ||
| logging.warning("Wrote results to %s.", out_filename) | ||
| logging.warning("Wrote encrypted results to %s.", out_filename) | ||
|
|
||
|
|
||
| def generate_key(): | ||
| """Generate a key for encryption.""" | ||
| return Fernet.generate_key() | ||
|
|
||
|
|
||
| def encrypt_data(data, key): | ||
| """Encrypt the provided data using the provided key.""" | ||
| fernet = Fernet(key) | ||
| return fernet.encrypt(data.encode()) | ||
|
|
||
|
|
||
| def decrypt_data(data, key): | ||
| """Decrypt the provided data using the provided key.""" | ||
| fernet = Fernet(key) | ||
| return fernet.decrypt(data).decode() | ||
|
|
| @@ -3 +3,3 @@ | ||
| wheel | ||
|
|
||
| cryptography==44.0.0 |
| Package | Version | Security advisories |
| cryptography (pypi) | 44.0.0 | None |
| with smart_open(out_filename) as out_file: | ||
| json_content = utils.json_for(results) | ||
|
|
||
| out_file.write(json_content + "\n") |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Copilot Autofix AI about 1 month ago
Copilot could not generate an autofix suggestion
Copilot could not generate an autofix suggestion for this alert. Try pushing a new commit or if the problem persists contact support.
| logging.warning( | ||
| "%s: Not publicly trusted - not trusted by %s.", | ||
| endpoint.url, | ||
| ", ".join(public_not_trusted_names), |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix AI about 1 month ago
To fix the problem, we should avoid logging the names of the trust stores directly. Instead, we can log a generic message indicating that the certificate is not publicly trusted without revealing specific details. This approach maintains the functionality of informing about the trust status while protecting potentially sensitive information.
- Replace the logging statement on line 889 to avoid logging the specific names of the trust stores.
- Ensure that the new logging message still conveys the necessary information without exposing sensitive details.
| @@ -886,5 +886,4 @@ | ||
| logging.warning( | ||
| "%s: Not publicly trusted - not trusted by %s.", | ||
| endpoint.url, | ||
| ", ".join(public_not_trusted_names), | ||
| "%s: Not publicly trusted by common trust stores.", | ||
| endpoint.url | ||
| ) |
|
|
||
| i += 1 | ||
| LOGGER.info("%s: Completed running PSHTT", thread) | ||
|
|
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix AI about 1 month ago
To fix the problem, we should avoid logging the entire results object directly. Instead, we can log a sanitized version of the results object that excludes any sensitive information. This can be achieved by creating a function that filters out sensitive fields from the results object before logging it.
- Create a function to sanitize the
resultsobject by removing or masking sensitive fields. - Replace the direct logging of
resultswith the sanitized version. - Ensure that the changes are made in the
run_pshttfunction where the logging occurs.
| @@ -93,2 +93,10 @@ | ||
|
|
||
| def sanitize_results(results): | ||
| """Sanitize the results object by removing or masking sensitive fields.""" | ||
| sanitized_results = [] | ||
| for result in results: | ||
| sanitized_result = {key: value for key, value in result.items() if key not in ["sensitive_field1", "sensitive_field2"]} | ||
| sanitized_results.append(sanitized_result) | ||
| return sanitized_results | ||
|
|
||
| def run_pshtt(domains, thread): | ||
| @@ -123,3 +131,4 @@ | ||
| LOGGER.error("%s: %s", thread, e) | ||
| LOGGER.error("%s: failed result %s", thread, results) | ||
| sanitized_results = sanitize_results(results) | ||
| LOGGER.error("%s: failed result %s", thread, sanitized_results) | ||
|
|
🗣 Description
💭 Motivation and context
🧪 Testing
✅ Pre-approval checklist
in code comments.
to reflect the changes in this PR.
✅ Pre-merge checklist
✅ Post-merge checklist