-
-
Notifications
You must be signed in to change notification settings - Fork 883
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SSO Support #4881
SSO Support #4881
Conversation
Wonderful! We’re also building an OIDC provider for indies, so this is a very welcome development 😊 I wasn’t able to find it in your PR; are you implementing PKCE grant flow? It’s more secure, and some identity providers (like ours) rely on it to function. Prior art in fediverse: |
@erlend-sh This PR doesn't support PKCE grant flow yet but, once it's merged, adding PKCE support should be a small change. We'll try to make some time to add it in a separate PR. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The migrations need to be tested manually, because CI currently only checks one migration, and multiple migrations were added.
@@ -0,0 +1,3 @@ | |||
ALTER TABLE local_user | |||
ALTER COLUMN password_encrypted SET NOT NULL; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This can fail if oauth accounts were already created. Not sure what to do about that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Correct. This is similar to the already integrated migration called "2024-04-15-105932_community_followers_url_optional".
16fbbec
to
45c0a00
Compare
Could you give some instructions how we can test this functionality? Maybe by writing some documentation (that will be necessary anyway). |
@Nutomic we'll write some docs for users but given that local testing is slightly more complicated, here are the steps to test it: 1- You will need the changes from these 3 PRs locally: 2- The lemmy-ui changes require the latest version of lemmy-js-client (the PR does not include this version yet since it requires publishing to npm. You will need to add it manually). cd $LEMMY_JS_CLIENT_DIR && pnpm run prepare
cd $LEMMY_UI_DIR pnpm add $LEMMY_JS_CLIENT_DIR 3- Run the DB migrations
8- The remaining fields "oauth_client_id" and "oauth_client_secret" are instance specific and require you to create an account with your preferred OIDC provider. With Privacy Portal, you can create a free account and test this setup like the following:
9- Now you should have all the fields filled in your configuration, click "save" and sign out from the admin account. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is great! We should def try to get it into 0.20.0
.
I'll mark it as draft, bc we're doing that for all the breaking changes PRs atm.
client_secret text NOT NULL, | ||
scopes text NOT NULL, | ||
auto_verify_email boolean DEFAULT TRUE NOT NULL, | ||
auto_approve_application boolean DEFAULT TRUE NOT NULL, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's already local_site.registration_mode
. I'm not sure how registration applications would work with this, but I don't think they should be bypassed if its RequireApplication
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was a bit weird to implement because when someone sign up with SSO, there's no room in the sign up process to submit a manual application so we're auto-filling applications with the text "Signed up with X OIDC provider".
We can certainly remove this toggle if you think it's best.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I want registration applications to work for SSO
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree, I think this column should be removed, because registration applications should still work with SSO.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just to make sure we're on the same page, the way this is currently implemented, the admin can select whether or not they want to auto_approve applications (when applications are required). If this is set to TRUE, signups using SSO bypass the application approval (per provider). If this is set to FALSE, sign ups using SSO fill the application field with the text "Signed up with X OIDC provider" and the application would require approval for the user to be able to sign in with SSO.
The changes that can be made here are:
1- Completely removing the "auto_approve_application" option.
2- Somewhere during SSO sign up, adding an input field for users to fill a custom application text. We're looking into this to see at which stage this could be implemented while staying compliant with OAUTH.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Both those sound correct, but I don't know how possible it is. Seems like it should be a pretty common use-case tho.
If its absolutely not possible, then I spose we have no choice but to ignore the registration_mode
when using SSO signups, although that scares me a little. Nothing has stopped spam attacks better than registration applications.
To clarify tho, yes remove the auto_approve_application
column, as that overlaps with the local_site.registration_mode
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Removed the "auto_approve_application" field.
Looking into the custom application field part...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added support for registration applications with custom user text.
crates/api_crud/src/site/read.rs
Outdated
oauth_provider.updated = None; | ||
true | ||
} else { | ||
false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Something tells me there's a better way to organize this. Like maybe it should be added as an impl function on the OAuthProvider
struct.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you mean the whole filter function or are you talking about the highlighted part?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The whole filter function, but I wouldn't worry about it for now, I can always fix later.
let conn = &mut get_conn(pool).await?; | ||
let oauth_providers = oauth_provider::table | ||
.order(oauth_provider::id) | ||
.select(oauth_provider::all_columns) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isn't the only difference that client_secret
is included in the unsafe?
In that case you might not need any of the extra types and functions below, since you already have serde_skip
on that field.
Check out the site
table, which just has pub private_key: Option<SensitiveString>,
, and doesn't need a separate SiteSafe
/ SiteUnsafe
version.
@Nutomic might be able to give better guidance than I can on this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Besides the client_secret, OAuthProvider is a version of UnsafeOAuthProvider that is decoupled from the DB schema and contains multiple Optional fields that are filtered out depending on whether the user is an admin or not.
The following fields are getting filtered out for normal users: ["issuer", "token_endpoint", "userinfo_endpoint", "id_claim", "name_claim", "auto_verify_email", "auto_approve_application", "account_linking_enabled", "enabled", "published", "updated"]
.
There's no security risk in leaking this data to the UI. It's just data that is not needed in the UI and that will be loaded on every login page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could also just add serde skip to all of those also.
I'm good with either way I spose, but I'd like to hear @Nutomic 's thoughts on this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The reason for the complexity here is that we're using the "Site" data both for the login page and for the admin oauth settings. This is something that existed in the original PR that we didn't change.
We have 3 options here:
1- If the current implementation is acceptable we keep it.
2- We simply allow these fields to be sent to the login page (We could still filter disabled OAUTH providers).
3- We could use serde skip for all these fields for the "Site" data and we create separate API call for the admin settings containing the rest of the data. We would need an additional struct for this solution too though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would definitely like to get rid of these duplicate struct and implementations because it makes maintenance more difficult. Though I agree that we shouldnt include unnecessary data in GetSiteResponse
to avoid bloated data. To keep it simple I would consider always sending these fields for admin accounts, and skipping them for normal users. I would also move the fields into a separate struct (and table?) so this can be done more easily.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using #[diesel(embed)]
created a lot of issues because it's not compatible with Identifiable
and a couple other things which created problems with Crud
.
We had to go a bit deeper into rust and dealt with it with custom serialization using a single Struct and a wrapper Struct. Any concerns about this approach?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't fully understand why that's necessary, you could just:
- Eliminate the duplicate struct, and use serde_skip for all the necessary fields.
- Make oauth_providers in
GetSiteResponse
optional, and only include it for admins.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
a separation of structs imo does kind of make sense because these optionals are always dangerous since setting/leaving them out are a contract based on purely comments and easy to misuse later.
i don't have a strong opinion though (except i think it's better to avoid too much bike shedding and only block merged for more important issues)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@dessalines oauth_providers in GetSiteResponse is always needed. It's needed (with limited fields) for the login page to display the SSO buttons. It's also needed with all fields (except client_secret) for admin pages.
In the latest changes we:
- Use serde_skip to skip fields that should never be sent to the UI such as client_secret.
- Removed the duplication in structs. We use a single struct containing all the fields needed for admin data.
- Wrap the struct with a transparent struct to use custom serialization that skips more fields and only keeps public data.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sweet
6ba2bdb
to
af6bb9d
Compare
@dessalines @dullbananas we've addressed most of the issues raised. Is there anything we're missing? Thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mainly just need @Nutomic to look at this also, about the Safe/Unsafe
variants.
Also test this to make sure that it works correctly with registration applications, that is a must.
I don't see anything else besides those two things.
Also it might be a while till this gets merged into main since we're not merging breaking changes at the moment, but that's on us to do once we get everything major resolved.
crates/api_crud/src/site/read.rs
Outdated
oauth_provider.updated = None; | ||
true | ||
} else { | ||
false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The whole filter function, but I wouldn't worry about it for now, I can always fix later.
@@ -312,6 +323,19 @@ pub fn diesel_url_update(opt: Option<&str>) -> LemmyResult<Option<Option<DbUrl>> | |||
} | |||
} | |||
|
|||
/// Takes an optional API URL-type input, and converts it to an optional diesel DB update (for non | |||
/// nullable properties). Also cleans the url params. | |||
pub fn diesel_required_url_update(opt: Option<&str>) -> LemmyResult<Option<DbUrl>> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These look good, thx.
client_secret text NOT NULL, | ||
scopes text NOT NULL, | ||
auto_verify_email boolean DEFAULT TRUE NOT NULL, | ||
auto_approve_application boolean DEFAULT TRUE NOT NULL, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree, I think this column should be removed, because registration applications should still work with SSO.
let conn = &mut get_conn(pool).await?; | ||
let oauth_providers = oauth_provider::table | ||
.order(oauth_provider::id) | ||
.select(oauth_provider::all_columns) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could also just add serde skip to all of those also.
I'm good with either way I spose, but I'd like to hear @Nutomic 's thoughts on this.
Also there's this clippy error: https://woodpecker.join-lemmy.org/repos/129/pipeline/8173/17 |
1c40b02
to
f8afeca
Compare
f8afeca
to
cfa0174
Compare
@Nutomic @dessalines Yesterday we rebased this branch, resolved all conflicts, and addressed the remaining issues raised. Please let us know if there's anything else remaining or whether this can be merged. Thanks! |
Good job, thank you! |
* Added OAUTH2 OIDC support * Fixes and improvements based on review feedback * use derive_new::new instead of TypedBuilder * merge migrations into a single file * fixes based on review feedback * remove unnecessary hostname_ui config * improvement based on review feedback * improvements based on review feedback * delete user oauth accounts at account deletion * fixes and improvements based on review feedback * removed auto_approve_application * support registration application with sso * improvements based on review feedback * making the TokenResponse an internal struct as it should be * remove duplicate struct * prevent oauth linking to unverified accounts * switched to manually entered username and removed the oauth name claim * fix cargo fmt * fix compile error * improvements based on review feedback * fixes and improvements based on review feedback --------- Co-authored-by: privacyguard <[email protected]>
@privacyguard any chance of adding PKCE support? |
* Add option to search exclusively by post title (#5015) * Add option to search exclusively by post title * Address format issues * Remove duplicated 'removed' filter * Replace url_search with search_term * Build generic PostQuery before search match * Create default queries. Move title_only to Search struct. Rename Url to PostURL * Revert PostUrl to Url * Upgrading webmention to 0.6.0, removes native-tls (#4976) * Removing embedded pict-rs. (#5023) Some reasons for removing this: - Even as an optional dependency, it locks us to many specific versions of rust deps. - Pict-rs is a large app that can and should be run in on its own. - Violates the philosophy of separation of concerns. * Adding clearurls crate to clean tracking params from links and markdown. (#5018) * Adding clearurls crate to clean tracking params from links and markdown. - Thanks to @jenrdikw for creating this - Fixes #4905 * Upgrading to new version of clearurls * Fix clippy * Remove enable nsfw (#5017) * Remove `local_site.enable_nsfw` in favor of `site.content_warning` (fixes #4627) * cleanup usage of SiteView::read_local * test * uppercase * SSO Support (#4881) * Added OAUTH2 OIDC support * Fixes and improvements based on review feedback * use derive_new::new instead of TypedBuilder * merge migrations into a single file * fixes based on review feedback * remove unnecessary hostname_ui config * improvement based on review feedback * improvements based on review feedback * delete user oauth accounts at account deletion * fixes and improvements based on review feedback * removed auto_approve_application * support registration application with sso * improvements based on review feedback * making the TokenResponse an internal struct as it should be * remove duplicate struct * prevent oauth linking to unverified accounts * switched to manually entered username and removed the oauth name claim * fix cargo fmt * fix compile error * improvements based on review feedback * fixes and improvements based on review feedback --------- Co-authored-by: privacyguard <[email protected]> * Adding ability to restore content on user unban. (#4845) * Adding ability to restore content on user unban. - Fixes #4721 * Fixing api tests. * Fix package.json * Fixing lemmy-js-client dep. * Adding API test for restoring content. * Adding a default_comment_sort_type column for local_site and local_user. (#4469) * Adding a default_comment_sort_type column for local_site and local_user. - Renamed SortType to PostSortType in the DB and code. - Renamed references to default_sort_type to default_post_sort_type. - Fixes #4128 * Renaming migration to current date. * Simplifying PostSortType. * Simplify tests using default (#5026) * Feature/custom emoji and tagline views (#4580) * Add custom_emoji list route * Add tagline list route * Apply linting * Remove unecessary TaglineView * Add category filter for custom emoji * Add create tagline endpoint * Add update tagline endpoint * Add delete tagline endpoint * Format through lint.sh * Remove custom_emojis and taglines from site resource * Get random tagline on site requets * Impl Crud for Tagline Remove superfluous properties * Move tagline endpoints under /admin * Impl Crud for CustomEmoji * Remove delete from tagline and custom emoji impls * Check markdown for tagline * Validate markdown on tagline * Make content fields non optional Add error types for tagline validation * Use process_markdown instead of process_markdown_opt * Consolidate Tagline error types * Remove unecessary clone * Updat misleading comments * Remove local_site_id from tagline and custom_emoji * Update TaglineInserForm and TaglineUpdateForm * Add ignore_page_limits for custom emojis EmojiPicker needs to be able to retrieve all emojis in 1 call * Update custom_emoji_view Only keep get_all als helper function calling list with paging ignored Only order on category when filtering on category * Removing pointless get_all fn. * remove tagline length checks * make fields of TaglineInsertForm and TaglineUpdateForm mandatory * move emoji order statement * add comment for GetSiteResponse.tagline --------- Co-authored-by: Freakazoid182 <> Co-authored-by: SleeplessOne1917 <[email protected]> Co-authored-by: Dessalines <[email protected]> Co-authored-by: Felix Ableitner <[email protected]> * Add category to RSS feeds (fixes #3446) (#5030) * Unittest for Search by title only (#5033) * added test for search by title only * formatted rust files * Upgrading to rust 1.81 (#5032) * Remove TypedBuilder in favor of derive_new (fixes #4863) (#5020) * Remove TypedBuilder in favor of derive_new (fixes #4863) * fix * fix * Ignore zero values when setting rate limits (fixes #4280) (#5029) * Ignore zero values when setting rate limits (fixes #4280) Havent bothered to add an error message for such an uncommon case. * fmt * reorder, add test * Always save remote image data (#4875) * Always save remote image data * cleanup --------- Co-authored-by: Felix Ableitner <[email protected]> Co-authored-by: Dessalines <[email protected]> * Get rid of a lot of pointless mut form initializations. (#5037) * Get rid of a lot of pointless mut form initializations. - Fixes #5036 * Fix clippy. * Simplify handling of NotFound SQL errors (fixes #4633) (#5031) * Simplify handling of NotFound SQL errors (fixes #4633) * fmt * wip * compiling * clippy * api tests * fix * Adding saved_only, liked_only, and disliked_only filters to search. (#5034) * Adding saved_only, liked_only, and disliked_only filters to search. - Fixes #4547 * Removing duplicate Url return type for search (was actually post). - This now works like the post_title_only filter. * Address PR comments. * Add saved_only post_view test. * Removing a few more Result<bool> . (#4977) * Removing a few more Result<bool> . * Running taplo fmt. * Running fmt. * Adding email taken test. * Fixing tests. * Adding back in missing admin check. * Rename check_has_local_followers function. * Conditionally hide comments on nsfw posts (fixes #4237) (#5028) * Conditionally hide comments on nsfw posts (fixes #4237) * fix test * Post scheduling (fixes #234) (#5025) * Post scheduling (fixes #234) * clippy * replace map_err with inspect_err * ignore unpublished posts in read queries * add api test * fmt * add some checks * address some review comments * allow updating schedule time * rewrite scheduled task * fmt * machete * compare date in sql, more filters * check for community ban in sql * remove api test (scheduled task only runs every 10 mins) * remove mut * add index * remove Post::read impl * fmt * fix * correctly handle changes to schedule time * normal users can only schedule up to 10 posts * Remove redundant local_user.auto_expand setting. (#5041) - Fixes #4643 Co-authored-by: SleeplessOne1917 <[email protected]> * Add ability to search for Community by its description (or title only). (#5044) - This changes the post_title_only for Search to title_only, since its also used in the community query now. - Fixes #4785 * Cleanup remaining use of Result<bool, Error> (fixes #4862) (#5047) * Replace clippy allow annotation with expect (fixes #5012) (#5048) * Add skip_serialize_none to OAuth structs with option fields (#5046) * Add skip_serialize_none to OAuth structs with option fields * PR feedback * Remove serde and ts export from SSO db-only structs * Apply scheduled post limit to future posts instead of past posts, and verify this in test (#5054) * test scheduled_post_count * fix syntax error * fix formatting * fix argument order * fix user_scheduled_post_count function * Avoid breaking changes, keep response fields as deprecated (#5058) * Adding skip_serializing_none to another OAuth API request. (#5060) * Handle partial settings backup (fixes #4307) (#5063) * Handle partial settings backup (fixes #4307) * clippy * Avoid stack overflow when fetching nested comments, reduce max comment depth to 50 (#5009) * Avoid stack overflow when fetching deeply nested comments * add test case * reduce comment depth, add docs * decrease * reduce max comment depth to 50 * fmt * clippy * cleanup * Update Rust crate clap to v4.5.18 (#5066) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate reqwest to v0.12.8 (#5068) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate sitemap-rs to v0.2.2 (#5069) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate tracing-actix-web to v0.7.13 (#5070) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Changing renovate to run on the weekends. (#5067) - Lemmy builds take a long time, and the current schedule only runs for about 4 hours once a month, and renovate also rate-limits itself to ~ 2 per hour. This gives it a large enough window, without clogging up my runner machines. * Update Rust crate async-trait to v0.1.83 (#5065) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency @types/jest to v29.5.13 (#5071) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Attempt to fix test for fetching deeply nested comment (#5072) Co-authored-by: Dessalines <[email protected]> * Add modlog entries for bulk removals. (#5050) * Add modlog entries for bulk removals. - Added unit tests for removal / restore to api_common/utils. - Fixes #4699 * Address PR comments. * Combining remove and restore functions. * Trigger build. * Trigger build 2. * Changing allow to expect. * Adding local site settings to reject federated upvotes or downvotes. (#5038) * Adding local site settings to reject federated upvotes or downvotes. - Should help defend against downvote spamming instances. - Fixes #4086 * Adding new vote mode types. * Simpler activitypub vote check. * Adding undo vote for failed vote mode check. * Update crates/api_common/src/utils.rs --------- Co-authored-by: Nutomic <[email protected]> * Replace `clippy::unwrap_used` in tests (#5064) * Add LemmyResult to session_middleware tests * Add LemmyResult to inboxes tests * Add LemmyResult to slurs tests * Add LemmyResult to markdown tests * Add LemmyResult to rate_limiter tests * Add LemmyResult to error tests * Add LemmyResult to api_common utils tests * Add LemmyResult to request tests * Add LemmyResult to claims tests * Propagate registration_applications errors * Remove clippy::unwrap_used from community tests * Add LemmyResult to community_view tests * Add LemmyResult to db_schema post tests * Add LemmyResult to site_aggregates tests * Add LemmyResult to private_message tests * Add LemmyResult to activity tests * Add LemmyResult to federation_allowlist tests * Add LemmyResult to comment_aggregates tests * Add LemmyResult to post_report tests * Add LemmyResult to moderator tests * Add LemmyResult to community_aggregates tests * Add LemmyResult to person_aggregates tests * Add LemmyResult to language tests * Add LemmyResult to post_aggregates tests * Add LemmyResult to db_schema comment tests * Add LemmyResult to actor_language tests * Add LemmyResult to vote_view tests * Add LemmyResult to registration_application_view tests * Add LemmyResult to private_message_view tests * Add LemmyResult to private_message_report_view tests * Add LemmyResult to post_report_view tests * Add LemmyResult to comment_report_view tests * Add LemmyResult to sitemap tests * Replace .expect() with .unwrap() * Format code * Remove clippy::unwrap_used from activity tests * Add diesel result in db_schema tests * Format code * Map to_bytes() error to LemmyErrorType * Remove clippy::unwrap_used from error tests * Removing a few more unwraps, and cleaning up language code. * Replace map_err with unwrap_or_default * Replace ok_or with and_then --------- Co-authored-by: Dessalines <[email protected]> * Adding a get_random_community endpoint. (#5042) * Adding a get_random_community endpoint. - Fixes #4698 * Fixing issue from main. * Adding ListingType to the query. * More concise query filter. * Resolve links to remote posts into local URL (#5057) * move code to new file * rewrite markdown links (fixes #2987) * add missing file * add helper fn * also convert post.url * simplify search.rs * clippy * also rewrite user/community links in markdown * Call from apub handlers, cleanup * no network requests in test * clippy * fix tests * serial * test * no mut * add api test * fix api test * Update Rust crate clap to v4.5.19 (#5080) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate html2text to v0.12.6 (#5081) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate rustls to v0.23.14 (#5083) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update rust-futures monorepo to v0.3.31 (#5090) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate actix-web-prom to 0.9.0 (#5091) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate regex to v1.11.0 (#5093) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate serde_with to v3.10.0 (#5094) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate serde_with to v3.11.0 (#5095) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency ts-jest to v29.2.5 (#5089) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency typescript to v5.6.2 (#5098) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency eslint to v9.12.0 (#5097) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update docker/dockerfile Docker tag to v1.10 (#5099) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update pnpm to v9.12.0 (#5100) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate reqwest to v0.12.8 (#5082) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency @types/node to v22.7.4 (#5096) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update tamasfe/taplo Docker tag to v0.9.3 (#5101) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update tmknom/prettier Docker tag to v3.2.5 (#5102) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Add test case for community.hidden in post_view (ref #5074) (#5106) * Resize post thumbnails (fixes #4053) (#5107) * Resize post thumbnails (fixes #4053) * 256px * Update typescript-eslint monorepo to v8.8.1 (#5103) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update renovate config (#5109) * Update renovate config * ignroe rust updates, run on first day of month * fix * Add community alphabetic sorting (#5056) * Started * Finished? Need to write tests * Formatting * Formatting * Formatting * Write tests * Formatting * Formatting * Formatting * Unnecessary lifetime * Safety * Unwrap * Formatting * Formatting * Fix local_only test * Formatting * Name consistency * Adding lower to community name sort. --------- Co-authored-by: Dessalines <[email protected]> Co-authored-by: Dessalines <[email protected]> * Support markdown sub/superscript, use external crate for spoilers (#5135) * Use external crate for spoiler tags * Also add other plugins * fix test * Remove comment_like.post_id column which is unnecessary (ref #5122) (#5134) * Allow admins to resolve removed or deleted objects via API (#5061) * Allow admins to resolve removed or deleted objects via API * Removing pointless TestUser. --------- Co-authored-by: Dessalines <[email protected]> * Remove individual user/community inboxes (#5124) * Remove endpoints for individual community/user inboxes fixes #4147 fixes #3928 * Remove shared_inbox_url columns * fmt * Fixing errors. --------- Co-authored-by: Carlos Cabello <[email protected]> Co-authored-by: Nutomic <[email protected]> Co-authored-by: privacyguard <[email protected]> Co-authored-by: privacyguard <[email protected]> Co-authored-by: Freakazoid182 <[email protected]> Co-authored-by: SleeplessOne1917 <[email protected]> Co-authored-by: leoseg <[email protected]> Co-authored-by: Sander Saarend <[email protected]> Co-authored-by: Joseph Silva <[email protected]> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: netbrum <[email protected]> Co-authored-by: Steven Vergenz <[email protected]> Co-authored-by: Richard Schwab <[email protected]>
…ct (#5024) * Add test case to ensure all errors are in use * Add test case to find errors that are only used for federation * Move federation errors into separate enum (fixes #4926) * clippy * clippy * remove serde tag * Reduce errors dess (#5138) * Add option to search exclusively by post title (#5015) * Add option to search exclusively by post title * Address format issues * Remove duplicated 'removed' filter * Replace url_search with search_term * Build generic PostQuery before search match * Create default queries. Move title_only to Search struct. Rename Url to PostURL * Revert PostUrl to Url * Upgrading webmention to 0.6.0, removes native-tls (#4976) * Removing embedded pict-rs. (#5023) Some reasons for removing this: - Even as an optional dependency, it locks us to many specific versions of rust deps. - Pict-rs is a large app that can and should be run in on its own. - Violates the philosophy of separation of concerns. * Adding clearurls crate to clean tracking params from links and markdown. (#5018) * Adding clearurls crate to clean tracking params from links and markdown. - Thanks to @jenrdikw for creating this - Fixes #4905 * Upgrading to new version of clearurls * Fix clippy * Remove enable nsfw (#5017) * Remove `local_site.enable_nsfw` in favor of `site.content_warning` (fixes #4627) * cleanup usage of SiteView::read_local * test * uppercase * SSO Support (#4881) * Added OAUTH2 OIDC support * Fixes and improvements based on review feedback * use derive_new::new instead of TypedBuilder * merge migrations into a single file * fixes based on review feedback * remove unnecessary hostname_ui config * improvement based on review feedback * improvements based on review feedback * delete user oauth accounts at account deletion * fixes and improvements based on review feedback * removed auto_approve_application * support registration application with sso * improvements based on review feedback * making the TokenResponse an internal struct as it should be * remove duplicate struct * prevent oauth linking to unverified accounts * switched to manually entered username and removed the oauth name claim * fix cargo fmt * fix compile error * improvements based on review feedback * fixes and improvements based on review feedback --------- Co-authored-by: privacyguard <[email protected]> * Adding ability to restore content on user unban. (#4845) * Adding ability to restore content on user unban. - Fixes #4721 * Fixing api tests. * Fix package.json * Fixing lemmy-js-client dep. * Adding API test for restoring content. * Adding a default_comment_sort_type column for local_site and local_user. (#4469) * Adding a default_comment_sort_type column for local_site and local_user. - Renamed SortType to PostSortType in the DB and code. - Renamed references to default_sort_type to default_post_sort_type. - Fixes #4128 * Renaming migration to current date. * Simplifying PostSortType. * Simplify tests using default (#5026) * Feature/custom emoji and tagline views (#4580) * Add custom_emoji list route * Add tagline list route * Apply linting * Remove unecessary TaglineView * Add category filter for custom emoji * Add create tagline endpoint * Add update tagline endpoint * Add delete tagline endpoint * Format through lint.sh * Remove custom_emojis and taglines from site resource * Get random tagline on site requets * Impl Crud for Tagline Remove superfluous properties * Move tagline endpoints under /admin * Impl Crud for CustomEmoji * Remove delete from tagline and custom emoji impls * Check markdown for tagline * Validate markdown on tagline * Make content fields non optional Add error types for tagline validation * Use process_markdown instead of process_markdown_opt * Consolidate Tagline error types * Remove unecessary clone * Updat misleading comments * Remove local_site_id from tagline and custom_emoji * Update TaglineInserForm and TaglineUpdateForm * Add ignore_page_limits for custom emojis EmojiPicker needs to be able to retrieve all emojis in 1 call * Update custom_emoji_view Only keep get_all als helper function calling list with paging ignored Only order on category when filtering on category * Removing pointless get_all fn. * remove tagline length checks * make fields of TaglineInsertForm and TaglineUpdateForm mandatory * move emoji order statement * add comment for GetSiteResponse.tagline --------- Co-authored-by: Freakazoid182 <> Co-authored-by: SleeplessOne1917 <[email protected]> Co-authored-by: Dessalines <[email protected]> Co-authored-by: Felix Ableitner <[email protected]> * Add category to RSS feeds (fixes #3446) (#5030) * Unittest for Search by title only (#5033) * added test for search by title only * formatted rust files * Upgrading to rust 1.81 (#5032) * Remove TypedBuilder in favor of derive_new (fixes #4863) (#5020) * Remove TypedBuilder in favor of derive_new (fixes #4863) * fix * fix * Ignore zero values when setting rate limits (fixes #4280) (#5029) * Ignore zero values when setting rate limits (fixes #4280) Havent bothered to add an error message for such an uncommon case. * fmt * reorder, add test * Always save remote image data (#4875) * Always save remote image data * cleanup --------- Co-authored-by: Felix Ableitner <[email protected]> Co-authored-by: Dessalines <[email protected]> * Get rid of a lot of pointless mut form initializations. (#5037) * Get rid of a lot of pointless mut form initializations. - Fixes #5036 * Fix clippy. * Simplify handling of NotFound SQL errors (fixes #4633) (#5031) * Simplify handling of NotFound SQL errors (fixes #4633) * fmt * wip * compiling * clippy * api tests * fix * Adding saved_only, liked_only, and disliked_only filters to search. (#5034) * Adding saved_only, liked_only, and disliked_only filters to search. - Fixes #4547 * Removing duplicate Url return type for search (was actually post). - This now works like the post_title_only filter. * Address PR comments. * Add saved_only post_view test. * Removing a few more Result<bool> . (#4977) * Removing a few more Result<bool> . * Running taplo fmt. * Running fmt. * Adding email taken test. * Fixing tests. * Adding back in missing admin check. * Rename check_has_local_followers function. * Conditionally hide comments on nsfw posts (fixes #4237) (#5028) * Conditionally hide comments on nsfw posts (fixes #4237) * fix test * Post scheduling (fixes #234) (#5025) * Post scheduling (fixes #234) * clippy * replace map_err with inspect_err * ignore unpublished posts in read queries * add api test * fmt * add some checks * address some review comments * allow updating schedule time * rewrite scheduled task * fmt * machete * compare date in sql, more filters * check for community ban in sql * remove api test (scheduled task only runs every 10 mins) * remove mut * add index * remove Post::read impl * fmt * fix * correctly handle changes to schedule time * normal users can only schedule up to 10 posts * Remove redundant local_user.auto_expand setting. (#5041) - Fixes #4643 Co-authored-by: SleeplessOne1917 <[email protected]> * Add ability to search for Community by its description (or title only). (#5044) - This changes the post_title_only for Search to title_only, since its also used in the community query now. - Fixes #4785 * Cleanup remaining use of Result<bool, Error> (fixes #4862) (#5047) * Replace clippy allow annotation with expect (fixes #5012) (#5048) * Add skip_serialize_none to OAuth structs with option fields (#5046) * Add skip_serialize_none to OAuth structs with option fields * PR feedback * Remove serde and ts export from SSO db-only structs * Apply scheduled post limit to future posts instead of past posts, and verify this in test (#5054) * test scheduled_post_count * fix syntax error * fix formatting * fix argument order * fix user_scheduled_post_count function * Avoid breaking changes, keep response fields as deprecated (#5058) * Adding skip_serializing_none to another OAuth API request. (#5060) * Handle partial settings backup (fixes #4307) (#5063) * Handle partial settings backup (fixes #4307) * clippy * Avoid stack overflow when fetching nested comments, reduce max comment depth to 50 (#5009) * Avoid stack overflow when fetching deeply nested comments * add test case * reduce comment depth, add docs * decrease * reduce max comment depth to 50 * fmt * clippy * cleanup * Update Rust crate clap to v4.5.18 (#5066) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate reqwest to v0.12.8 (#5068) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate sitemap-rs to v0.2.2 (#5069) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate tracing-actix-web to v0.7.13 (#5070) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Changing renovate to run on the weekends. (#5067) - Lemmy builds take a long time, and the current schedule only runs for about 4 hours once a month, and renovate also rate-limits itself to ~ 2 per hour. This gives it a large enough window, without clogging up my runner machines. * Update Rust crate async-trait to v0.1.83 (#5065) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency @types/jest to v29.5.13 (#5071) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Attempt to fix test for fetching deeply nested comment (#5072) Co-authored-by: Dessalines <[email protected]> * Add modlog entries for bulk removals. (#5050) * Add modlog entries for bulk removals. - Added unit tests for removal / restore to api_common/utils. - Fixes #4699 * Address PR comments. * Combining remove and restore functions. * Trigger build. * Trigger build 2. * Changing allow to expect. * Adding local site settings to reject federated upvotes or downvotes. (#5038) * Adding local site settings to reject federated upvotes or downvotes. - Should help defend against downvote spamming instances. - Fixes #4086 * Adding new vote mode types. * Simpler activitypub vote check. * Adding undo vote for failed vote mode check. * Update crates/api_common/src/utils.rs --------- Co-authored-by: Nutomic <[email protected]> * Replace `clippy::unwrap_used` in tests (#5064) * Add LemmyResult to session_middleware tests * Add LemmyResult to inboxes tests * Add LemmyResult to slurs tests * Add LemmyResult to markdown tests * Add LemmyResult to rate_limiter tests * Add LemmyResult to error tests * Add LemmyResult to api_common utils tests * Add LemmyResult to request tests * Add LemmyResult to claims tests * Propagate registration_applications errors * Remove clippy::unwrap_used from community tests * Add LemmyResult to community_view tests * Add LemmyResult to db_schema post tests * Add LemmyResult to site_aggregates tests * Add LemmyResult to private_message tests * Add LemmyResult to activity tests * Add LemmyResult to federation_allowlist tests * Add LemmyResult to comment_aggregates tests * Add LemmyResult to post_report tests * Add LemmyResult to moderator tests * Add LemmyResult to community_aggregates tests * Add LemmyResult to person_aggregates tests * Add LemmyResult to language tests * Add LemmyResult to post_aggregates tests * Add LemmyResult to db_schema comment tests * Add LemmyResult to actor_language tests * Add LemmyResult to vote_view tests * Add LemmyResult to registration_application_view tests * Add LemmyResult to private_message_view tests * Add LemmyResult to private_message_report_view tests * Add LemmyResult to post_report_view tests * Add LemmyResult to comment_report_view tests * Add LemmyResult to sitemap tests * Replace .expect() with .unwrap() * Format code * Remove clippy::unwrap_used from activity tests * Add diesel result in db_schema tests * Format code * Map to_bytes() error to LemmyErrorType * Remove clippy::unwrap_used from error tests * Removing a few more unwraps, and cleaning up language code. * Replace map_err with unwrap_or_default * Replace ok_or with and_then --------- Co-authored-by: Dessalines <[email protected]> * Adding a get_random_community endpoint. (#5042) * Adding a get_random_community endpoint. - Fixes #4698 * Fixing issue from main. * Adding ListingType to the query. * More concise query filter. * Resolve links to remote posts into local URL (#5057) * move code to new file * rewrite markdown links (fixes #2987) * add missing file * add helper fn * also convert post.url * simplify search.rs * clippy * also rewrite user/community links in markdown * Call from apub handlers, cleanup * no network requests in test * clippy * fix tests * serial * test * no mut * add api test * fix api test * Update Rust crate clap to v4.5.19 (#5080) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate html2text to v0.12.6 (#5081) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate rustls to v0.23.14 (#5083) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update rust-futures monorepo to v0.3.31 (#5090) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate actix-web-prom to 0.9.0 (#5091) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate regex to v1.11.0 (#5093) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate serde_with to v3.10.0 (#5094) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate serde_with to v3.11.0 (#5095) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency ts-jest to v29.2.5 (#5089) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency typescript to v5.6.2 (#5098) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency eslint to v9.12.0 (#5097) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update docker/dockerfile Docker tag to v1.10 (#5099) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update pnpm to v9.12.0 (#5100) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate reqwest to v0.12.8 (#5082) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency @types/node to v22.7.4 (#5096) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update tamasfe/taplo Docker tag to v0.9.3 (#5101) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update tmknom/prettier Docker tag to v3.2.5 (#5102) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Add test case for community.hidden in post_view (ref #5074) (#5106) * Resize post thumbnails (fixes #4053) (#5107) * Resize post thumbnails (fixes #4053) * 256px * Update typescript-eslint monorepo to v8.8.1 (#5103) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update renovate config (#5109) * Update renovate config * ignroe rust updates, run on first day of month * fix * Add community alphabetic sorting (#5056) * Started * Finished? Need to write tests * Formatting * Formatting * Formatting * Write tests * Formatting * Formatting * Formatting * Unnecessary lifetime * Safety * Unwrap * Formatting * Formatting * Fix local_only test * Formatting * Name consistency * Adding lower to community name sort. --------- Co-authored-by: Dessalines <[email protected]> Co-authored-by: Dessalines <[email protected]> * Support markdown sub/superscript, use external crate for spoilers (#5135) * Use external crate for spoiler tags * Also add other plugins * fix test * Remove comment_like.post_id column which is unnecessary (ref #5122) (#5134) * Allow admins to resolve removed or deleted objects via API (#5061) * Allow admins to resolve removed or deleted objects via API * Removing pointless TestUser. --------- Co-authored-by: Dessalines <[email protected]> * Remove individual user/community inboxes (#5124) * Remove endpoints for individual community/user inboxes fixes #4147 fixes #3928 * Remove shared_inbox_url columns * fmt * Fixing errors. --------- Co-authored-by: Carlos Cabello <[email protected]> Co-authored-by: Nutomic <[email protected]> Co-authored-by: privacyguard <[email protected]> Co-authored-by: privacyguard <[email protected]> Co-authored-by: Freakazoid182 <[email protected]> Co-authored-by: SleeplessOne1917 <[email protected]> Co-authored-by: leoseg <[email protected]> Co-authored-by: Sander Saarend <[email protected]> Co-authored-by: Joseph Silva <[email protected]> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: netbrum <[email protected]> Co-authored-by: Steven Vergenz <[email protected]> Co-authored-by: Richard Schwab <[email protected]> * Search with LemmyErrorType prefix --------- Co-authored-by: Dessalines <[email protected]> Co-authored-by: Carlos Cabello <[email protected]> Co-authored-by: privacyguard <[email protected]> Co-authored-by: privacyguard <[email protected]> Co-authored-by: Freakazoid182 <[email protected]> Co-authored-by: SleeplessOne1917 <[email protected]> Co-authored-by: leoseg <[email protected]> Co-authored-by: Sander Saarend <[email protected]> Co-authored-by: Joseph Silva <[email protected]> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: netbrum <[email protected]> Co-authored-by: Steven Vergenz <[email protected]> Co-authored-by: Richard Schwab <[email protected]> Co-authored-by: Dessalines <[email protected]>
Implements #2930.
This PR is based on this previous PR by @thepaperpilot.
Related PRs:
We noticed that the original PR is outdated and has a lot of conflicts with the recent changes. We tried to keep the previous commits whenever possible (in lemmy-js-client and lemmy-ui).
How it works?
Available Configuration
auto_approve_application: When enabled, users signing up using OIDC won't need manual approval even if applications are required.Disclaimer
This is our first ever rust contribution.
Who we are? Why are we contributing to Lemmy?
Privacy Portal is an OIDC provider and an email aliasing service focused on privacy. We have decided to contribute to select open source projects that empower Free Speech online.
Our OIDC provider services are currently offered free of charge. In the future, we will have a generous free plan that will cover most deployments.
Using Privacy Portal as your OIDC provider offers your users great privacy benefits. User emails will automatically get replaced by single-purpose Privacy Aliases during sign up. Users will be able to enter any name (to be used as username). Users can benefit from email encryption and much more.