Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Markdown Support & Translator Model #9

Merged
merged 8 commits into from
Dec 25, 2024
Merged

Conversation

fuma-nama
Copy link
Contributor

@fuma-nama fuma-nama commented Dec 24, 2024

Goal

Support Markdown, this includes support for handling diffs.

Changes

  • Redesign the current translate command
    • Add a layer of Translator
    • Two types of operation: update and new. I found the differences between the two operations could be large and no longer can be grouped by a single force: true.
    • Re-implemented the current translators (e.g. json, js/ts)
  • Added unit tests with Vitest
    • ensure the previous translators still work
    • do not require a token, contributor-friendly

Context

Usually we want to maintain the context of docs, it would be better to provide the full content.

This PR will provide the full-context, which needs more input tokens. Maybe we think of a better algorithm(?)

(I ran bun format, some previous files are somehow updated too.)

Diff

the current diff command cannot detect the real updates, translator handled the diff step. We can probably make it work with the new translator model by another PR

Copy link

vercel bot commented Dec 24, 2024

@fuma-nama is attempting to deploy a commit to the Pontus Abrahamsson's projects Team on Vercel.

A member of the Team first needs to authorize it.

@fuma-nama
Copy link
Contributor Author

fuma-nama commented Dec 25, 2024

I changed the current implementation:

  • JSON translator will compare the two objects instead of using git diff, it should be more accurate.
  • JS/TS translator directly replace strings, like:
const v = "this will be translated"

We assume the order of strings will not be changed during translation, so it can be cached by reusing previous translations

I have added some details via code comments to the new algorithm.

@fuma-nama fuma-nama marked this pull request as ready for review December 25, 2024 08:10
@fuma-nama fuma-nama changed the title Add Markdown Support Add Markdown Support & Translator Model Dec 25, 2024
@fuma-nama
Copy link
Contributor Author

Tested only with o4 on unit tests, might need your help to test it with other models and on CLI

@pontusab pontusab merged commit b06b375 into midday-ai:main Dec 25, 2024
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants