Skip to content

Commit

Permalink
Merge pull request #815 from NGWi/patch-1
Browse files Browse the repository at this point in the history
Update README.mkd
  • Loading branch information
nsantacruz authored Apr 26, 2022
2 parents d8c67a8 + fa1cab8 commit 6b4b9bc
Showing 1 changed file with 45 additions and 40 deletions.
85 changes: 45 additions & 40 deletions README.mkd
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
## New Interfaces for Jewish Texts

[Sefaria](https://www.sefaria.org) is creating interfaces, apps (like a source sheet builder) and infrastructure (like an API and a structured dataset) for Jewish texts and textual learning.
[Sefaria](https://www.sefaria.org) is creating interfaces, apps (like a source sheet builder), and infrastructure (like an API and a structured dataset) for Jewish texts and textual learning.

You can find outputs of our entire database in [Sefaria-Export](https://github.com/Sefaria/Sefaria-Export).

Interested developers should join the [sefaria-dev mailing list](https://groups.google.com/forum/#!forum/sefaria-dev).

For general discussion questions about the project, please email [[email protected]](mailto:[email protected]).

If you find textual problems in our library please email [[email protected]](mailto:[email protected]).
If you find textual problems in our library, please email [[email protected]](mailto:[email protected]).

You can post bugs or request/discuss features on [GitHub Issues](https://github.com/sefaria/Sefaria-Project/issues?direction=desc&page=1&sort=created&state=open). Tackling an issue marked as a "Starter Project" is a good way to sink your teeth into Sefaria.

Expand All @@ -19,7 +19,7 @@ If you're interested in working on a project you see listed here, please email t
### Getting Started
First clone the Sefaria-Project repository to a directory on your computer, then follow the instructions:

*Note: if you are a developer that might want to contribute code to Sefaria, we suggest first making a fork of this respository by clicking the "Fork" button above when logged in to GitHub.*
*Note: if you are a developer that might want to contribute code to Sefaria, we suggest first making a fork of this repository by clicking the "Fork" button above when logged in to GitHub.*

*Note for macOS users - Install `Homebrew`:*

Expand All @@ -28,23 +28,23 @@ First clone the Sefaria-Project repository to a directory on your computer, then
#### 1) Install Python 3

###### Linux and macOS
Most UNIX systems come with a python interpreter pre-installed. However, this is generally still Python 2. The recommended way to get Python 3, and not mess up any software the OS is dependant on, is by using Pyenv. You can use the instructions [here](https://github.com/pyenv/pyenv#installation) and also [here](https://opensource.com/article/19/5/python-3-default-mac#what-we-should-do).
Most UNIX systems come with a python interpreter pre-installed. However, this is generally still Python 2. The recommended way to get Python 3, and not mess up any software the OS is dependent on, is by using Pyenv. You can use the instructions [here](https://github.com/pyenv/pyenv#installation) and also [here](https://opensource.com/article/19/5/python-3-default-mac#what-we-should-do).

In a later step we can configure virtual environments to work with Pyenv so you can completely isolate your Sefaria stack.
In a later step, we can configure virtual environments to work with Pyenv so you can completely isolate your Sefaria stack.

###### Windows:
The Pyenv repository above also has recommendations for Windows.

In order to just plain install Python:
- Read the [Official Python documentation on Windows](https://docs.python.org/3.7/using/windows.html)
- Go to the [Python Download Page](https://www.python.org/downloads/release/python-375/) and download and install python.
- Go to the [Python Download Page](https://www.python.org/downloads/release/python-375/) and download and install Python.
- Add the python directory to your OS' PATH variable if the installer has not done so.

#### 2) Install virtualenv (Recommended, but optional):
If you work on many Python projects, you may want to keep Sefaria's python installation separate using Virtualenv. If you're happy having Sefaria requirements in your main Python installation, skip this step.

##### Use With Pyenv (Recommended)
You can use virtualenv functionality while also using Pyenv, this allows you to further isolate you code and requirements from other projects.
You can use virtualenv functionality while also using Pyenv; this allows you to further isolate your code and requirements from other projects.

###### Unix & Windows
Instructions [here](https://github.com/pyenv/pyenv-virtualenv#installing-as-a-pyenv-plugin).
Expand All @@ -55,34 +55,34 @@ Create [a pyenv virtualenv](https://github.com/pyenv/pyenv-virtualenv#using-pyen

In your Sefaria directory, run `pyenv local [venv-name]`. This will create a `.python-version` and write the version name provided to the file (e.g. `3.7.5/envs/sefaria-venv`). This should serve to activate the virtualenv whenever you are in the Sefaria directory.

*Note: If after following the installation and configuration instructions, running `python -V` still displays the system version, you may have to manually add the shims directory to your path.* If that does not correct the issue, you should check your bash init file (.zshrc, .bashrc or the like) for the line `eval "$(pyenv init -)"` (not inside an if statement) and change it to `eval "$(pyenv init --path)"` and restar the shell.
*Note: If, after following the installation and configuration instructions, running `python -V` still displays the system version, you may have to manually add the shims directory to your path.* If that does not correct the issue, you should check your bash init file (.zshrc, .bashrc, or the like) for the line `eval "$(pyenv init -)"` (not inside an if statement) and change it to `eval "$(pyenv init --path)"` and restart the shell.

*Note: If you are using an IDE like PyCharm, you can (and should) configure the interpreter options on your Sefaria-Project to point to the python executable of this virtualenv (e.g. `~/.pyenv/versions/3.7.5/envs/sefaria-venv/bin/python3.7`)*

##### Classic virtualenv
Install [virtualenv](http://pypi.python.org/pypi/virtualenv) then enter these commands:
Install [virtualenv](http://pypi.python.org/pypi/virtualenv), then enter these commands:

*Note: You may need to install Pip (see below) first in order to install virtualenv*

*Note: You can perform this step from anywhere in your command line, but it might be easier and tidier to run this step from the root of your project directory that you just cloned. e.g `~/web-projecs/Sefaria-Project $`*
*Note: You can perform this step from anywhere in your command line, but it might be easier and tidier to run this step from the root of your project directory that you just cloned. e.g `~/web-projects/Sefaria-Project $`*

virtualenv venv --distribute
source venv/bin/activate

Now you should see `(venv)` in front of your command prompt. The second command sets your shell to use the Python virtual environment that you've just created. This is something that you have to run everytime you open a new shell and want to run the Sefaria demo. You can always tell if you're in the virtual environment by checking if `(venv)` is at the beginning of your command prompt.
Now you should see `(venv)` in front of your command prompt. The second command sets your shell to use the Python virtual environment that you've just created. This is something that you have to run every time you open a new shell and want to run the Sefaria demo. You can always tell if you're in the virtual environment by checking if `(venv)` is at the beginning of your command prompt.

#### 3) Pip:
If you used pyenv, Pip should be available via the pyenv version of python.
If you used pyenv, Pip should be available via the pyenv version of Python.
###### Unix
If you don't already have it in your Python installation, install [pip](https://pip.pypa.io/en/stable/installing/). Then use it to install the required Python packages.
If you don't already have it in your Python installation, install [Pip](https://pip.pypa.io/en/stable/installing/). Then use it to install the required Python packages.

###### Windows
Use instructions [here](http://www.tylerbutler.com/2012/05/how-to-install-python-pip-and-virtualenv-on-windows-with-powershell/) and then make sure that the scripts sub folder of the python installation directory is also in PATH.
Use instructions [here](http://www.tylerbutler.com/2012/05/how-to-install-python-pip-and-virtualenv-on-windows-with-powershell/) and then make sure that the scripts subfolder of the python installation directory is also in PATH.


*Note: this step (and **most** of the following command line instructions) must be run from the Sefaria-Project root directory*

Please *temporarily* comment out the line `psycopg2==2.8.6` in `requirements.txt.` and then run
Please *temporarily* comment out the line `psycopg2==2.8.6` in `requirements.txt.` and then run:

pip install -r requirements.txt

Expand All @@ -99,7 +99,7 @@ If you are *not* using virtualenv, you may have to run it with sudo: `sudo pip i
sudo dnf install python2-devel libpq-devel


*Note: If you see an error that `pg_config executable not found`, you need to install PostgreSQL. If on MacOS you see an error while `Building wheel for psycopg2` about `linker command failed with exit code 1`, you may need to add the path to OpenSSL*
*Note: If you see an error that `pg_config executable not found`, you need to install PostgreSQL. If on macOS you see an error while `Building wheel for psycopg2` about `linker command failed with exit code 1`, you may need to add the path to OpenSSL*

```
export LDFLAGS="-L/usr/local/opt/openssl/lib"
Expand Down Expand Up @@ -148,42 +148,47 @@ Make sure that the server user has write access to it by using a command such as

If you don't already have it, [install MongoDB](https://www.mongodb.com/docs/manual/administration/install-community/). Our current data dump requires MongoDB version 4.4 or later. After installing Mongo according to the instructions, you should have a mongo service automatically running in the background.

Otherwise run the mongo daemon with:
Otherwise, run the mongo daemon with:

mongod
(only use sudo if necessary, it may result in a locked socket file being created that may prevent mongo from running later on)

or user your os service manager to have it start on startup.
(Only use sudo if necessary, as it may result in a locked socket file being created that may prevent mongo from running later on).

Or use your os service manager to have it start on startup.

Mongo usually sets all the correct paths fr itself to run properly nowadays, but see [here](https://www.mongodb.com/docs/manual/tutorial/manage-mongodb-processes/#start-mongod-processes) for details on setting a specific path for it to store data in,
Mongo usually sets all the correct paths for itself to run properly nowadays, but see [here](https://www.mongodb.com/docs/manual/tutorial/manage-mongodb-processes/#start-mongod-processes) for details on setting a specific path for it to store data in.


#### 8) Put some texts in your database:

MongoDB dumps of our database are available to download.

The recommended dump (which is of a manageable size) is available [here](https://storage.googleapis.com/sefaria-mongo-backup/dump_small.tar.gz).
The recommended dump (which is a more manageable size) is available [here](https://storage.googleapis.com/sefaria-mongo-backup/dump_small.tar.gz).

A complete dump is also available [here](https://storage.googleapis.com/sefaria-mongo-backup/dump.tar.gz). The complete dump includes the `history` collections, which contains a complete revision history of every text in our library. For many applications, this data is not relevant. We recommend using the smaller dump unless you're specifically interested in texts revision history within Sefaria.

Unzip the file and extract the `dump` folder. If you don't have an app for unzipping, this can be done from Command Prompt by navigating to the folder containing the download and using `tar -x dump.tar.gz` or `tar -x dump_small.tar.gz` (depending on which dump you downloaded).

A complete dump is also available [here](https://storage.googleapis.com/sefaria-mongo-backup/dump.tar.gz). The complete dump includes the `history` collections which contains a complete revision history of every text our library. For many applications this data is not relevant. We recommend using the smaller dump unless you're specifically interested in texts revision history within Sefaria.
This `dump` must be restored as a MongoDB database.

From MongoDB version 4.4, mongorestore comes separately from the MongoDB server, you'll also need to download the [Database Tools](https://www.mongodb.com/try/download/database-tools?tck=docs_databasetools)
From MongoDB version 4.4, mongorestore comes separately from the MongoDB server, so if you don't already have it, you'll also need to download and unzip the [Database Tools](https://www.mongodb.com/try/download/database-tools?tck=docs_databasetools). You may then need to add its \bin\ directory to your PATH environment variables.

Once you've download and unzipped this content, from the parent directory which contains `dump` run:
Once you have the unzipped `dump`; from the folder which contains `dump`, run:

mongorestore --drop

This will create (or overwrite) a mongo database called `sefaria`.

If you have used `dump_small.tar.gz`, use the mongo client shell, or a GUI you have installed to create an empty collection inside the `sefaria` database called `history`.
If you used the recommended dump, `dump_small.tar.gz`, create an empty collection inside the `sefaria` database called `history`. This can be done using the mongo client shell or a GUI you have installed, such as MongoDB Compass.

#### 9) Set up Django's local server

Sefaria is using Google's reCAPTCHA to verify the user is not a bot. For a deployment you should register and use your own reCAPTCHA keys (https://pypi.org/project/django-recaptcha/#installation).
For local development the default test keys would suffice. The warning can be suprassed by uncommenting the following in the local_settings.py file:
Sefaria is using Google's reCAPTCHA to verify the user is not a bot. For a deployment, you should register and use your own reCAPTCHA keys (https://pypi.org/project/django-recaptcha/#installation).
For local development, the default test keys would suffice. The warning can be suppressed by uncommenting the following in the local_settings.py file:

SILENCED_SYSTEM_CHECKS = ['captcha.recaptcha_test_key_error']

`manage.py` is used to run and to manage the local server. It is located in the root directory of the `Sefaria-Project` code base.
`manage.py` is used to run and manage the local server. It is located in the root directory of the `Sefaria-Project` codebase.

Django auth features run on a separate database. To init this database and set up Django's auth system, switch to the root directory of the `Sefaria-Project` code base, and run (from the project root):

Expand All @@ -196,9 +201,9 @@ Django auth features run on a separate database. To init this database and set u

Node is now required to run the site. Even if you choose to have js run only on the client, we are also using [Webpack](https://webpack.js.org/) to bundle our javascript.

Instaling Node and npm from the main installers on Node's homepage may cause permission issues. For that reason it is recommended to use one of the alternative methods (with a preference to a version manager like nvm) listed [here](https://nodejs.org/en/download/package-manager/):
Instaling Node and npm from the main installers on Node's homepage may cause permission issues. For that reason, it is recommended to use one of the alternative methods (with a preference for a version manager like nvm) listed [here](https://nodejs.org/en/download/package-manager/).

###### Debian, Ubuntu and Linux Mint
###### Debian, Ubuntu, and Linux Mint
You are better off using `apt-get nodejs` or following the instructions [here](https://github.com/nodesource/distributions/blob/master/README.md). They will install both Node and npm.

###### macOS
Expand All @@ -210,7 +215,7 @@ Now download the required Javascript libraries and install some global tools for
npm install
npm run setup

If the second command fails, you may have to install using `sudo npm run setup`
If the second command fails, you may have to install using `sudo npm run setup`.


#### Run Webpack
Expand All @@ -226,29 +231,29 @@ to bundle once. To watch the javascript for changes and automatically rebuild, r

#### Server-side rendering with Node:

Sefaria uses React.js. To render HTML server-side, we use a Node.js server. For development, the site is fully functional without server-side rendering. For deploying in a production environment however server side HTML is very important for bots and SEO.
Sefaria uses React.js. To render HTML server-side, we use a Node.js server. For development, the site is fully functional without server-side rendering. For deploying in a production environment, however, server-side HTML is very important for bots and SEO.

##### Install Redis

To use server side rendering, you must also install Redis Cache. `brew install redis` or `sudo apt-get install redis`.
To use server-side rendering, you must also install Redis Cache. `brew install redis` or `sudo apt-get install redis`.

To run redis: `redis-server`. On MacOS: `brew services start redis`
To run redis: `redis-server`. On macOS: `brew services start redis`


##### Configure Django and Node to use Redis as a shared data store
##### Configure Django and Node to use Redis as a shared datastore

**Django**

Update your `local_settings.py` file and replace the `CACHES` variable with the `CACHES` variable meant for server side rendering, commented out in `local_settings_example.py`.
Update your `local_settings.py` file and replace the `CACHES` variable with the `CACHES` variable meant for server-side rendering, commented out in `local_settings_example.py`.

Also make sure the following are set:
Also, make sure the following are set:
```
SESSION_CACHE_ALIAS = "default" #declares where Django's session engine will store data
USER_AGENTS_CACHE = 'default' #declares where the Django user agent middleware will store data (cannot be JSON encoded!)
SHARED_DATA_CACHE_ALIAS = 'shared' #Tells the application where to store data placed in cache by `sefaria.system.cache` shared-cache functions
USE_NODE = True
NODE_HOST = "http://localhost:3000" #Or whatever host and port you set node up to listen on
NODE_HOST = "http://localhost:3000" #Or whatever host and port you set Node up to listen on
```

Expand All @@ -265,7 +270,7 @@ The following environment variables, defined in `./node/local_settings.js`, can
| `REDIS_PORT` | `6379` | The Default port Redis listens on |


These variables can be set via command line explicitly, or set up to be defined when your machine's shell runs or set up in your IDE settings for running the node server.
These variables can be set via command line explicitly or set up to be defined when your machine's shell runs or set up in your IDE settings for running the node server.

For development, you can run the Node server using nodemon with:

Expand Down Expand Up @@ -299,7 +304,7 @@ The shell script `cli` will invoke a python interpreter with the core models loa

## Thanks

We're grateful to the following orgazations for providing us with donated services:
We're grateful to the following organizations for providing us with donated services:

- [Google Apps for Work](https://apps.google.com/)
- [PyCharm](https://www.jetbrains.com/pycharm/)
Expand Down

0 comments on commit 6b4b9bc

Please sign in to comment.