Invalid model error when calling encoding_for_model on Heroku

[developius]

Hi, thanks for publishing the gem! I'm using it on Heroku with stack heroku-22 but getting an error when calling encoding_for_model. Locally it works just fine.

Ruby: 3.2.2
OS: Ubuntu 22.04

Here's a Dockerfile that exhibits the issue (tested on play-with-docker):

FROM heroku/heroku:22
RUN gpg --keyserver hkp://keyserver.ubuntu.com --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
RUN curl -sSL https://get.rvm.io | bash -s stable
RUN /bin/bash -l -c ". /etc/profile.d/rvm.sh && rvm install 3.2.2 && rvm use 3.2.2"
RUN /bin/bash -l -c ". /etc/profile.d/rvm.sh && ruby -v"
RUN /bin/bash -l -c ". /etc/profile.d/rvm.sh && gem install tiktoken"
RUN /bin/bash -l -c ". /etc/profile.d/rvm.sh && ruby -e 'require \"tiktoken\"; Tiktoken.encoding_for_model(\"text-embedding-ada-002\")'"

And the error:

/usr/local/rvm/gems/ruby-3.2.2/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken.rb:16:in `_encoding_for_model': Invalid model:  (ArgumentError)

    Tiktoken._encoding_for_model(model, extra_special_tokens)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
        from /usr/local/rvm/gems/ruby-3.2.2/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken.rb:16:in `encoding_for_model'
        from -e:1:in `<main>'

On my Mac (M1), the Tiktoken.encoding_for_model("text-embedding-ada-002") works just fine.

Any ideas what might be causing this? Perhaps an architecture problem?

[developius]

I got it to segfault on Heroku too, when calling encoding_for_model:

Segfault

irb(main):001:0> Tiktoken.encoding_for_model("gpt-3.5-turbo")
/app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken.rb:16:in `_encoding_for_model': Invalid model:  (ArgumentError)

    Tiktoken._encoding_for_model(model, extra_special_tokens)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
irb(main):002:0> Tiktoken.get_encoding("cl100k_base")
/app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken.rb:12: [BUG] Segmentation fault at 0x0000000000000010
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0033 p:---- s:0208 e:000207 CFUNC  :_get_encoding
c:0032 p:0012 s:0202 e:000201 METHOD /app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken.rb:12
c:0031 p:0006 s:0196 e:000195 EVAL   (irb):2 [FINISH]
c:0030 p:---- s:0193 e:000192 CFUNC  :eval
c:0029 p:0020 s:0185 e:000184 METHOD /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/workspace.rb:119
c:0028 p:0153 s:0177 e:000175 METHOD /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/context.rb:502
c:0027 p:0119 s:0164 e:000163 BLOCK  /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:588
c:0026 p:0024 s:0158 e:000157 METHOD /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:777
c:0025 p:0007 s:0152 e:000151 BLOCK  /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:567
c:0024 p:0126 s:0147 e:000146 BLOCK  /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/ruby-lex.rb:267 [FINISH]
c:0023 p:---- s:0143 e:000142 CFUNC  :loop
c:0022 p:0005 s:0139 e:000138 BLOCK  /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/ruby-lex.rb:249 [FINISH]
c:0021 p:---- s:0136 e:000135 CFUNC  :catch
c:0020 p:0010 s:0131 e:000130 METHOD /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/ruby-lex.rb:248
c:0019 p:0050 s:0126 E:000bf8 METHOD /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:566
c:0018 p:0003 s:0121 e:000120 BLOCK  /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:500 [FINISH]
c:0017 p:---- s:0118 e:000117 CFUNC  :catch
c:0016 p:0050 s:0113 E:000ba0 METHOD /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:499
c:0015 p:0069 s:0107 e:000106 METHOD /app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:421
c:0014 p:0114 s:0101 e:000100 METHOD /app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/commands/console/console_command.rb:70
c:0013 p:0009 s:0097 e:000096 METHOD /app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/commands/console/console_command.rb:19
c:0012 p:0033 s:0092 e:000091 METHOD /app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/commands/console/console_command.rb:102
c:0011 p:0054 s:0088 e:000087 METHOD /app/vendor/bundle/ruby/3.2.0/gems/thor-1.2.1/lib/thor/command.rb:27
c:0010 p:0040 s:0080 e:000079 METHOD /app/vendor/bundle/ruby/3.2.0/gems/thor-1.2.1/lib/thor/invocation.rb:127
c:0009 p:0213 s:0073 e:000072 METHOD /app/vendor/bundle/ruby/3.2.0/gems/thor-1.2.1/lib/thor.rb:392
c:0008 p:0030 s:0060 e:000059 METHOD /app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/command/base.rb:87
c:0007 p:0147 s:0053 e:000052 METHOD /app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/command.rb:48
c:0006 p:0038 s:0041 e:000040 TOP    /app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/commands.rb:18 [FINISH]
c:0005 p:---- s:0036 e:000035 CFUNC  :require
c:0004 p:0011 s:0031 e:000030 METHOD <internal:/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37
c:0003 p:0158 s:0016 e:000015 METHOD /app/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32
c:0002 p:0024 s:0006 e:000005 EVAL   /app/bin/rails:4 [FINISH]
c:0001 p:0000 s:0003 E:001c00 DUMMY  [FINISH]

-- Ruby level backtrace information ----------------------------------------
/app/bin/rails:4:in `<main>'
/app/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
<internal:/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
<internal:/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
/app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/commands.rb:18:in `<main>'
/app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/command.rb:48:in `invoke'
/app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/command/base.rb:87:in `perform'
/app/vendor/bundle/ruby/3.2.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
/app/vendor/bundle/ruby/3.2.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
/app/vendor/bundle/ruby/3.2.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
/app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/commands/console/console_command.rb:102:in `perform'
/app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/commands/console/console_command.rb:19:in `start'
/app/vendor/bundle/ruby/3.2.0/gems/railties-7.0.4.3/lib/rails/commands/console/console_command.rb:70:in `start'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:421:in `start'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:499:in `run'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:499:in `catch'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:500:in `block in run'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:566:in `eval_input'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/ruby-lex.rb:248:in `each_top_level_statement'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/ruby-lex.rb:248:in `catch'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/ruby-lex.rb:249:in `block in each_top_level_statement'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/ruby-lex.rb:249:in `loop'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/ruby-lex.rb:267:in `block (2 levels) in each_top_level_statement'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:567:in `block in eval_input'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:777:in `signal_status'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb.rb:588:in `block (2 levels) in eval_input'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/context.rb:502:in `evaluate'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/workspace.rb:119:in `evaluate'
/app/vendor/ruby-3.2.2/lib/ruby/3.2.0/irb/workspace.rb:119:in `eval'
(irb):2:in `<main>'
/app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken.rb:12:in `get_encoding'
/app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken.rb:12:in `_get_encoding'

-- Machine register context ------------------------------------------------
 RIP: 0x00007f68d79d3610 RBP: 0x00007fff769f7558 RSP: 0x00007fff769f7028
 RAX: 0x00007f68d7e17bc0 RBX: 0x0000000000000008 RCX: 0x00007f68d73c0b80
 RDX: 0x00007f68d73c0000 RDI: 0x0000000000000008 RSI: 0x00007f68d3c5e930
  R8: 0x0000000000000000  R9: 0x0000000000000000 R10: 0x0000000000000007
 R11: 0x0000000000000007 R12: 0x00007fff769f7280 R13: 0x00007fff769f76f0
 R14: 0x00007fff769f70a8 R15: 0x0000000000000004 EFL: 0x0000000000010202

-- C level backtrace information -------------------------------------------
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_print_backtrace+0xd) [0x7f68d7b2bc9f] /ruby-3.2.2/vm_dump.c:785
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_vm_bugreport) /ruby-3.2.2/vm_dump.c:1080
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_bug_for_fatal_signal+0xf4) [0x7f68d7922154] /ruby-3.2.2/error.c:813
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(sigsegv+0x4d) [0x7f68d7a7b23d] /ruby-3.2.2/signal.c:964
/lib/x86_64-linux-gnu/libc.so.6(0x7f68d7424520) [0x7f68d7424520]
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(RBASIC_CLASS+0x0) [0x7f68d79d3610] /ruby-3.2.2/object.c:885
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_class_of) ./include/ruby/internal/globals.h:175
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_obj_is_kind_of) /ruby-3.2.2/object.c:848
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_set_errinfo+0x3f) [0x7f68d792dacf] /ruby-3.2.2/eval.c:1878
/app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken/tiktoken.so(0x7f68d02296db) [0x7f68d02296db]
/app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken/tiktoken.so(0x7f68d022aae5) [0x7f68d022aae5]
/app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken/tiktoken.so(0x7f68d0225df6) [0x7f68d0225df6]
/app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken/tiktoken.so(0x7f68d0223a3d) [0x7f68d0223a3d]
/app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken/tiktoken.so(0x7f68d02243e4) [0x7f68d02243e4]
/app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken/tiktoken.so(0x7f68d022145d) [0x7f68d022145d]
/app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken/tiktoken.so(0x7f68d021ce3b) [0x7f68d021ce3b]
/app/vendor/bundle/ruby/3.2.0/gems/tiktoken-0.1.0-x86_64-linux/lib/tiktoken/tiktoken.so(0x7f68d0222312) [0x7f68d0222312]
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_call_cfunc_with_frame+0x127) [0x7f68d7b003b7] /ruby-3.2.2/vm_insnhelper.c:3268
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_sendish+0x97) [0x7f68d7b108e4] /ruby-3.2.2/vm_insnhelper.c:5080
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_exec_core) /ruby-3.2.2/insns.def:820
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_vm_exec+0xd3) [0x7f68d7b16213] /ruby-3.2.2/vm.c:2374
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_f_eval+0x160) [0x7f68d7b170e0] /ruby-3.2.2/vm_eval.c:1791
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_call_cfunc_with_frame+0x127) [0x7f68d7b003b7] /ruby-3.2.2/vm_insnhelper.c:3268
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_sendish+0x97) [0x7f68d7b108e4] /ruby-3.2.2/vm_insnhelper.c:5080
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_exec_core) /ruby-3.2.2/insns.def:820
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_vm_exec+0x5b2) [0x7f68d7b166f2] /ruby-3.2.2/vm.c:2383
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(invoke_block_from_c_bh+0x212) [0x7f68d7b176cf] /ruby-3.2.2/vm.c:1398
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_yield_with_cref) /ruby-3.2.2/vm.c:1509
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_yield) /ruby-3.2.2/vm.c:1517
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_yield_0) /ruby-3.2.2/vm_eval.c:1348
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(loop_i) /ruby-3.2.2/vm_eval.c:1447
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_vrescue2+0x11e) [0x7f68d792b5be] /ruby-3.2.2/eval.c:917
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_rescue2+0x8e) [0x7f68d792b84e] /ruby-3.2.2/eval.c:898
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_call_cfunc_with_frame+0x127) [0x7f68d7b003b7] /ruby-3.2.2/vm_insnhelper.c:3268
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_sendish+0xd2) [0x7f68d7b060f2] /ruby-3.2.2/vm_insnhelper.c:5080
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_exec_core+0x1e2) [0x7f68d7b109c2] /ruby-3.2.2/insns.def:801
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_vm_exec+0xd3) [0x7f68d7b16213] /ruby-3.2.2/vm.c:2374
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(catch_i+0x2d8) [0x7f68d7b189e8] /ruby-3.2.2/vm.c:1398
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_catch_protect+0xee) [0x7f68d7b0907e] /ruby-3.2.2/vm_eval.c:2393
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_catch_obj+0x51) [0x7f68d7b09221] /ruby-3.2.2/vm_eval.c:2419
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_call_cfunc_with_frame+0x127) [0x7f68d7b003b7] /ruby-3.2.2/vm_insnhelper.c:3268
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_sendish+0xd2) [0x7f68d7b060f2] /ruby-3.2.2/vm_insnhelper.c:5080
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_exec_core+0x1e2) [0x7f68d7b109c2] /ruby-3.2.2/insns.def:801
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_vm_exec+0xd3) [0x7f68d7b16213] /ruby-3.2.2/vm.c:2374
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(catch_i+0x2d8) [0x7f68d7b189e8] /ruby-3.2.2/vm.c:1398
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_catch_protect+0xee) [0x7f68d7b0907e] /ruby-3.2.2/vm_eval.c:2393
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_catch_obj+0x51) [0x7f68d7b09221] /ruby-3.2.2/vm_eval.c:2419
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_call_cfunc_with_frame+0x127) [0x7f68d7b003b7] /ruby-3.2.2/vm_insnhelper.c:3268
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_sendish+0xd2) [0x7f68d7b060f2] /ruby-3.2.2/vm_insnhelper.c:5080
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_exec_core+0x1e2) [0x7f68d7b109c2] /ruby-3.2.2/insns.def:801
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_vm_exec+0x5b2) [0x7f68d7b166f2] /ruby-3.2.2/vm.c:2383
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(load_iseq_eval+0x3c) [0x7f68d7994401] /ruby-3.2.2/load.c:697
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(require_internal) /ruby-3.2.2/load.c:1203
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_require_string+0x42) [0x7f68d7994e62] /ruby-3.2.2/load.c:1294
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_call_cfunc_with_frame+0x127) [0x7f68d7b003b7] /ruby-3.2.2/vm_insnhelper.c:3268
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_call_alias+0x7d) [0x7f68d7b25fbd] /ruby-3.2.2/vm_insnhelper.c:3447
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_sendish+0x97) [0x7f68d7b108e4] /ruby-3.2.2/vm_insnhelper.c:5080
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(vm_exec_core) /ruby-3.2.2/insns.def:820
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_vm_exec+0x5b2) [0x7f68d7b166f2] /ruby-3.2.2/vm.c:2383
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(rb_ec_exec_node+0xb1) [0x7f68d7927441] /ruby-3.2.2/eval.c:289
/app/vendor/ruby-3.2.2/bin/../lib/libruby.so.3.2(ruby_run_node+0x93) [0x7f68d792d903] /ruby-3.2.2/eval.c:330
/app/vendor/ruby-3.2.2/bin/ruby(rb_main+0x21) [0x559404d55187] ./main.c:38
/app/vendor/ruby-3.2.2/bin/ruby(main) ./main.c:57
/lib/x86_64-linux-gnu/libc.so.6(0x7f68d740bd90) [0x7f68d740bd90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f68d740be40]
[0x559404d551d5]

[lstone]

Ever solve this issue?