Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add defresolver async? options #90

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
8 changes: 8 additions & 0 deletions src/gosura/helpers/error.clj
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,11 @@
(error nil resolver-errors))
([resolved-value resolver-errors]
(resolve-as resolved-value resolver-errors)))

(defn error-response
[throwable]
{:message (ex-message throwable)
:info (str throwable)
:type (.getName (class throwable))
:stacktrace (->> (.getStackTrace throwable)
(map str))})
37 changes: 26 additions & 11 deletions src/gosura/helpers/resolver2.clj
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
(ns gosura.helpers.resolver2
"gosura.helpers.resolver의 v2입니다."
(:require [com.walmartlabs.lacinia.resolve :refer [resolve-as]]
(:require [com.walmartlabs.lacinia.resolve :refer [resolve-as] :as resolve]
[failjure.core :as f]
[gosura.auth :as auth]
[gosura.helpers.error :as error]
Expand All @@ -20,16 +20,29 @@
[catch-exceptions? body]
(if catch-exceptions?
`(try
~@body
~body
(catch Exception e#
(log/error e#)
(resolve-as
nil
{:message (ex-message e#)
:info (str e#)
:type (.getName (class e#))
:stacktrace (->> (.getStackTrace e#)
(map str))})))
(error/error-response e#))))
body))

(defn transform-body
[body return-camel-case?]
(cond-> body
return-camel-case? (update-resolver-result transform-keys->camelCaseKeyword)))

(defmacro wrap-async-body
[async? return-camel-case? body]
(if async?
`(let [result# (resolve/resolve-promise)]
(.start (Thread.
Copy link

@devgrapher devgrapher Jan 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

혹시 이게 모든 resolve마다 새로운 스레드를 만드는거라면 스레드가 너무 많아지는건 아닐까 싶은데..
future는 어떨까요? future는 cachedThreadPool을 쓰는것 같네요.
https://stackoverflow.com/questions/35641757/does-future-always-create-a-new-thread

혹은 커스텀하게 풀을 만들어서 사용해도 좋을거같아요!

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

궁극적으로는 core.async를 써야할거같지만 문서에도 나온것처럼 이건 테스트가 좀 필요한듯하여..

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

제가 궁금했던 부분도 이거였어요. 스레드를 비동기 리졸버에서 매번 새로 생성하면 문제가 될 것 같아서요.
저는 core.async 보다 프로미스가 좋을 것 같습니다.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

간단히 테스트해보니.. future나 thread나 비슷한 결과를 보여주네요.
게시글 하나당 여러번의 필드리졸버가 호출되면서 많은 수의 스레드가 만들어지는군요.
thread를 쓴다하더라도 어차피 한번 사용하고 버려질꺼고..
future를 쓴다고 해도 동시에 여러 필드리졸버가 호출되면 이것도 어쩔수없이 스레드가 많이 생성되네요. (스레드가 재활용이된건지모르겠네요)
물론 시간이 지나면 스레드 개수는 다시 내려왔습니다.

아래 커맨드로 병렬로 10개의 요청을 보내는 작업얼 여러번 진행했습니다.
seq 1 10 | xargs -n1 -P10 curl 'http://localhost:8080/graphql' ...

스크린샷 2023-01-11 오후 2 11 54

스크린샷 2023-01-11 오후 2 21 16

스크린샷 2023-01-11 오후 1 45 20

Copy link

@devgrapher devgrapher Jan 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

그냥 superlifter가 편할까.. 싶은생각도.. 이경우 중복된 id에 대한 요청은 알아서 한번만 보내주는 효과도 있으니..
아 이건 urania로도 커버가 되긴하네요

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

일단은 superlifter나 promesa를 탐구해봐야겠습니다

Copy link
Contributor

@ruseel ruseel Jan 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

우리 경우엔 스레드로 보내는 대부분의 작업들이 i/o대기용인데.. 이 대기를 위해서 개별 스레드를 점유하고 있는거니까요.
그래서 뭔가 이벤트기반의 솔루션을 탐색하게 되네요..

이 댓글이 참 맞는 말이다 싶어서
검색을 한 참 하다

여기로 갔는데요.
https://martintrojer.github.io/clojure/2013/07/07/coreasync-and-blocking-io

결국 i/o 대기를 위해서 개별 쓰레드를 점유하지 않으려면

aws/invoke 를 aws/invoke-async 로 바꿔야 할 것 같아요.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

오 맞아요 요것도 있었죠!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

저는 invoke-async를 함 츄라이 해볼래요

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@devgrapher promesa 코드보니 vthread도 있네요. 코드는 보다가 시간이 없어서 여기까지만..

#(try
(resolve/deliver! result# (transform-body ~body ~return-camel-case?))
(catch Throwable t#
(resolve/deliver! result# nil (error/error-response t#))))))
result#)
body))

(defmacro wrap-resolver-body
Expand All @@ -47,10 +60,11 @@
"
[{:keys [this ctx arg parent]} option args body]
(let [{:keys [auth kebab-case? return-camel-case? required-keys-in-parent
filters decode-ids-by-keys catch-exceptions?]
filters decode-ids-by-keys catch-exceptions? async?]
:or {kebab-case? true
return-camel-case? true
catch-exceptions? true
async? false
required-keys-in-parent []}} option
result (gensym 'result_)
auth-filter-opts `(auth/->auth-result ~auth ~ctx)
Expand All @@ -67,9 +81,10 @@
(if (or (nil? ~auth-filter-opts)
(and ~auth-filter-opts
(not (f/failed? ~auth-filter-opts))))
(let [~result (do (let ~let-mapping (wrap-catch-body ~catch-exceptions? ~body)))]
(cond-> ~result
~return-camel-case? (update-resolver-result transform-keys->camelCaseKeyword)))
(let [~result (do (let ~let-mapping (->> ~@body
(wrap-async-body ~async? ~return-camel-case?)
(wrap-catch-body ~catch-exceptions?))))]
(transform-body ~result ~return-camel-case?))
(resolve-as nil {:message "Unauthorized"})))))


Expand Down
81 changes: 58 additions & 23 deletions test/gosura/helpers/resolver_test.clj
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
(ns gosura.helpers.resolver-test
(:require [clojure.test :refer [are deftest is run-tests testing]]
[gosura.helpers.resolver :as gosura-resolver]
[com.walmartlabs.lacinia.resolve :as resolve]
[gosura.helpers.resolver2 :as gosura-resolver2])
(:import [clojure.lang ExceptionInfo]))

Expand Down Expand Up @@ -79,12 +80,10 @@
(get-in ctx [:identity :cc]))

(def auth-column-name :userId)
(def arg-in-resolver (atom {}))

(gosura-resolver2/defresolver test-resolver
{:auth [user-auth auth-column-name]}
[ctx arg parent]
(reset! arg-in-resolver arg)
{:ctx ctx
:arg arg
:parent parent})
Expand All @@ -100,11 +99,19 @@
:arg
:userId) "1"))))
(testing "arg/parent가 default True로 kebab-case 설정이 잘 동작한다"
(let [ctx {:identity {:id "1"}}
arg {:intArg 1
:strArg "str"}
parent {}
_ (test-resolver ctx arg parent)]
(let [arg-in-resolver (atom {})
_ (gosura-resolver2/defresolver test-resolver-1
{:auth [user-auth auth-column-name]}
[ctx arg parent]
(reset! arg-in-resolver arg)
{:ctx ctx
:arg arg
:parent parent})
ctx {:identity {:id "1"}}
arg {:intArg 1
:strArg "str"}
parent {}
_ (test-resolver-1 ctx arg parent)]
(is (= @arg-in-resolver {:int-arg 1
:str-arg "str"
:user-id "1"}))))
Expand Down Expand Up @@ -148,7 +155,6 @@
{:auth [user-auth auth-column-name]
:filters {:country-code get-country-code}}
[ctx arg parent]
(reset! arg-in-resolver arg)
{:ctx ctx
:arg arg
:parent parent})
Expand Down Expand Up @@ -206,31 +212,60 @@
(is (= (-> result
:arg) {:testCol "1"}))))
(testing "에러가 던져졌을 때 GraphQL errors를 반환한다"
(let [_ (gosura-resolver2/defresolver test-resolver-7
[_ctx _arg _parent]
(throw (ex-info "something wrong!" {})))
ctx {}
arg {}
parent {}
resolved (test-resolver-7 ctx arg parent)
message (get-in resolved [:resolved-value :data :message])
info (get-in resolved [:resolved-value :data :info])
type' (get-in resolved [:resolved-value :data :type])
(let [_ (gosura-resolver2/defresolver test-resolver-7
[_ctx _arg _parent]
(throw (ex-info "something wrong!" {})))
ctx {}
arg {}
parent {}
resolved (test-resolver-7 ctx arg parent)
message (get-in resolved [:resolved-value :data :message])
info (get-in resolved [:resolved-value :data :info])
type' (get-in resolved [:resolved-value :data :type])
stacktrace (get-in resolved [:resolved-value :data :stacktrace])]
(are [expected result] (= expected result)
"something wrong!" message
"clojure.lang.ExceptionInfo: something wrong! {}" info
(some? type') true
(some? stacktrace) true)))
(testing "catch-exceptions? 설정이 false일 때 에러가 던져지면 그대로 throw한다"
(let [_ (gosura-resolver2/defresolver test-resolver-8
{:catch-exceptions? false}
[_ctx _arg _parent]
(throw (ex-info "something wrong!" {})))
(let [_ (gosura-resolver2/defresolver test-resolver-8
{:catch-exceptions? false}
[_ctx _arg _parent]
(throw (ex-info "something wrong!" {})))
ctx {}
arg {}
parent {}]
(is (thrown? ExceptionInfo (test-resolver-8 ctx arg parent))))))
(is (thrown? ExceptionInfo (test-resolver-8 ctx arg parent)))))
(testing "async? 설정이 false 일 때, 원래 동작을 잘 보장한다"
(let [_ (gosura-resolver2/defresolver test-resolver-9
{:async? false}
[ctx arg parent]
{:ctx ctx
:arg arg
:parent parent})
ctx {:identity {:id "1"}}
arg {:intArg 1
:strArg "str"}
parent {}
result (test-resolver-9 ctx arg parent)]
(is (= result {:ctx {:identity {:id "1"}}
:arg {:intArg 1
:strArg "str"}
:parent {}}))))
(testing "async? 설정이 true 일 때, Promise를 잘 반환한다"
(let [_ (gosura-resolver2/defresolver test-resolver-10
{:async? true}
[ctx arg parent]
{:ctx ctx
:arg arg
:parent parent})
ctx {:identity {:id "1"}}
arg {:intArg 1
:strArg "str"}
parent {}
resolved (test-resolver-10 ctx arg parent)]
(is (satisfies? resolve/ResolverResultPromise resolved)))))

(comment
(run-tests))