Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recherche via Elasticsearch #4096

Merged
merged 110 commits into from
Feb 5, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
110 commits
Select commit Hold shift + click to select a range
afbd1c0
add elastisearch modules and define base models
pierre-24 Dec 23, 2016
221b5ff
basic indexing with and without flags
pierre-24 Dec 23, 2016
cb01421
add migration for post
pierre-24 Dec 24, 2016
e61ffc1
check if pk and id match
pierre-24 Dec 24, 2016
280a4fe
Include forum into search
pierre-24 Dec 24, 2016
682010f
Merry christmas, @vhf and @artragis
pierre-24 Dec 25, 2016
c68752b
Attempt to index published contents and chapters (with some tricks)
pierre-24 Dec 27, 2016
f72b30f
use a slightly different trick for chapters
pierre-24 Dec 27, 2016
eb8aaed
Index text directly with PublishedContent if article or mini-tuto
pierre-24 Dec 27, 2016
fcc1cfa
add custom analyzer
pierre-24 Dec 27, 2016
a6f341c
Improve analyzer for programming language
pierre-24 Dec 28, 2016
987a7cf
argparse is magic, thx @artragis !
pierre-24 Dec 28, 2016
f7439ae
asciifolding is a wrong choice after all
pierre-24 Dec 28, 2016
d7addd3
implementation of basic search (no boosting, no type selection, no hi…
pierre-24 Dec 29, 2016
ddd2e72
elif instead of if
pierre-24 Dec 29, 2016
0e41258
weighting the different models
pierre-24 Dec 29, 2016
2a8b603
improve index creation (set mapping at the same time)
pierre-24 Dec 29, 2016
40bee12
Improve stuffs:
pierre-24 Dec 30, 2016
a278ef5
smaller is better (thx again, @artragis)
pierre-24 Dec 30, 2016
1c73993
to be consistant, use text_html for posts
pierre-24 Dec 30, 2016
553fee8
fix templatetags to prevent errors comming from fixtures
pierre-24 Dec 30, 2016
282ff34
attempt highlighting with ES
pierre-24 Dec 30, 2016
1f32be3
dont use order=score if continuous text
pierre-24 Dec 30, 2016
c665752
boost is now also performed based on the other criterions
pierre-24 Dec 30, 2016
437f913
Implement correct deletion (with weirds relationships for published c…
pierre-24 Dec 30, 2016
8a13142
invisible posts must still be indexed but wiped out from the query
pierre-24 Dec 30, 2016
dc96397
form is not invalid
pierre-24 Dec 30, 2016
0240817
correct way to connect, and prevent work when there is no connection
pierre-24 Dec 30, 2016
f166f51
shut up, Travis
pierre-24 Dec 30, 2016
bd062bf
prevent research page from failing while ES is not there
pierre-24 Dec 31, 2016
2b99fdc
more control over index creation through settings
pierre-24 Jan 2, 2017
1d7c0a6
use a custom tokeinzer to get rid of all the special characters inclu…
pierre-24 Jan 2, 2017
2b92ec7
Supprime le CSS de la recherche de l'ancienne page d'accueil (pre-ZEP 4)
Situphen Dec 29, 2016
4a25510
Utilise la barre de recherche de la page d'accueil pour la page de re…
Situphen Dec 29, 2016
0b3f61f
Improve form:
pierre-24 Jan 2, 2017
437c961
protect LaTeX from stemming as well
pierre-24 Jan 2, 2017
669954c
Logging instead of printing
pierre-24 Jan 2, 2017
32ba8af
Remove haystack and use the brand new search
pierre-24 Jan 2, 2017
d6f4cbe
yet another term to protect
pierre-24 Jan 2, 2017
6104d9f
add documentation
pierre-24 Jan 3, 2017
83d7224
update to version 5 of elasticsearch-py and use true delete_by_query()
pierre-24 Jan 3, 2017
83e662c
only instance is needed for delete_document_in_elasticsearch()
pierre-24 Jan 3, 2017
14b20a0
es_manager will fail if no connection to ES
pierre-24 Jan 3, 2017
5a28679
add a test for the command es_manager
pierre-24 Jan 3, 2017
48af348
try me, travis
pierre-24 Jan 4, 2017
3544f3e
good. Now, try arder, travis
pierre-24 Jan 4, 2017
22a9b11
refresh index, otherwise it is not available for research!
pierre-24 Jan 4, 2017
4142a93
use custom logger
pierre-24 Jan 4, 2017
28215cb
correct spelling mistakes, thx @vhf!
pierre-24 Jan 4, 2017
2c22184
add some tests for ESIndexManager
pierre-24 Jan 4, 2017
13ef3a1
add test for view (more will come, but dodo before\!)
pierre-24 Jan 4, 2017
2c14aac
New tests for view
pierre-24 Jan 5, 2017
f0ce199
Publishable is not published
pierre-24 Jan 6, 2017
bf36ebb
online editor is against pep 8
pierre-24 Jan 6, 2017
869df81
Let's be coherent with tutorialv2
pierre-24 Jan 6, 2017
944d5a6
old search is past history!
pierre-24 Jan 6, 2017
c70d210
upgrade doc as well
pierre-24 Jan 6, 2017
98f786a
Use SearchForm on home page
pierre-24 Jan 8, 2017
d662c7a
Merge remote-tracking branch 'upstream/dev' into add_elasticsearch
pierre-24 Jan 8, 2017
28c3677
Fix migration conflict
pierre-24 Jan 8, 2017
79cc588
where does this error come from?
pierre-24 Jan 8, 2017
7feac1b
Merge branch 'dev' of https://github.com/zestedesavoir/zds-site into …
pierre-24 Jan 8, 2017
c1f4745
Implement modifications of @vhf
pierre-24 Jan 9, 2017
7adb8fb
add information about memory usage
pierre-24 Jan 9, 2017
422dbc3
pip freeze stuffs
pierre-24 Jan 9, 2017
964acf9
update.md
pierre-24 Jan 9, 2017
adebf61
improve .travis.yml, thx to @firm1
pierre-24 Jan 9, 2017
ba6867f
Merge branch 'dev' of https://github.com/zestedesavoir/zds-site into …
pierre-24 Jan 10, 2017
429c8da
Merge branch 'dev' of https://github.com/zestedesavoir/zds-site into …
pierre-24 Jan 11, 2017
92f38ab
use single quote when possible (to keep the work of @vhf intact)
pierre-24 Jan 11, 2017
f7cab5b
correction for @situphen
pierre-24 Jan 15, 2017
d04b82c
Munin plugin
pierre-24 Jan 15, 2017
eb0dd5a
now, its elasticsearch 5.1.2
pierre-24 Jan 15, 2017
aa05f19
snakecase for subcommands
pierre-24 Jan 15, 2017
bda0ea0
mark_keywords is now in settings.py
pierre-24 Jan 15, 2017
4ff0a26
of course, I need to change the tests as well --"
pierre-24 Jan 15, 2017
da373f6
Post is directly hidden in ES
pierre-24 Jan 15, 2017
ea150e8
add note to contributors
pierre-24 Jan 15, 2017
8f6d2c0
change shards and replicas
pierre-24 Jan 15, 2017
1eacfcd
use ES_Q() instead of Q()
pierre-24 Jan 18, 2017
38ebf2d
add thumbnail and tags
pierre-24 Jan 18, 2017
9671892
search filters below the bar and not in sidebar
pierre-24 Jan 18, 2017
76e8748
mark corresponding posts updatable if topic is moved
pierre-24 Jan 20, 2017
3f3f28a
Delete previous chapters before reindexation
pierre-24 Jan 21, 2017
9f6192d
Improve update.md
pierre-24 Jan 21, 2017
01c5890
Merge branch 'dev' of https://github.com/zestedesavoir/zds-site into …
pierre-24 Jan 21, 2017
a5b632e
fix test and quotes
pierre-24 Jan 21, 2017
ae7dec1
batching the indexing
pierre-24 Jan 21, 2017
e4eeb0e
correct some mistakes
pierre-24 Jan 21, 2017
62e2f9c
update.md
pierre-24 Jan 21, 2017
e3754a4
correct batching !
pierre-24 Jan 21, 2017
0b969e1
Correct the error of @DevHugo
pierre-24 Jan 21, 2017
1e3f160
yield at the end
pierre-24 Jan 21, 2017
38a69d9
there is no need for with there
pierre-24 Jan 21, 2017
fa9f102
print and small optimisation
pierre-24 Jan 21, 2017
55c66d7
skip chapters correctly
pierre-24 Jan 22, 2017
7db240f
slight improvement of the documentation
pierre-24 Jan 22, 2017
de57f24
Batch par modèle
vhf Jan 22, 2017
3add954
fix tests
vhf Jan 22, 2017
4b4c6fb
Enlève les dépendences à solr/haystack
vhf Jan 23, 2017
8f02288
Optimisation de l'indexation initiale
vhf Jan 23, 2017
df7cd38
tests
vhf Jan 28, 2017
451280c
Groupe les contenus de recherche
vhf Jan 28, 2017
e9fcf7d
Améliore le design de la page de recherche
Situphen Jan 21, 2017
1b69595
add commands in Makefile
pierre-24 Jan 29, 2017
e804a2f
some code reviews
pierre-24 Jan 30, 2017
e415c99
done and done
pierre-24 Jan 31, 2017
f4b6221
Merge branch 'dev' of https://github.com/zestedesavoir/zds-site into …
pierre-24 Feb 4, 2017
0fda5a8
fix migrations (again :p)
pierre-24 Feb 4, 2017
c03dca7
Upgrade to lastest version
pierre-24 Feb 4, 2017
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ addons:
- mysql-client-5.6
- language-pack-fr
- unzip
- oracle-java8-set-default

git:
depth: 1
Expand All @@ -21,7 +22,7 @@ env:
global:
- secure: "azmDZZQZzf88zpbkYpLpxI66vpEVyv+kniW0QdWAt4qlys8P5OcO3VJBR5YT85vlvnjN9b6raWQAL1ymee0WmVdTmzXed8XjZv7t9QXVw7pfezxMKlEftVp/4Cu4wtvbew0ViZXNWV2hNXHzEqlhgnoIOq94i0UzZ7grMrI0xm0="
matrix:
- TEST_APP="-e back_mysql -- zds.member zds.mp zds.utils zds.forum zds.gallery zds.pages zds.search zds.featured zds.notification"
- TEST_APP="-e back_mysql -- zds.member zds.mp zds.utils zds.forum zds.gallery zds.pages zds.featured zds.notification zds.searchv2"
- TEST_APP="-e back_mysql -- zds.tutorialv2"
- TEST_APP="-e front"

Expand All @@ -45,6 +46,10 @@ cache:
- $HOME/.texlive
- $HOME/bin

before_install:
# see https://docs.travis-ci.com/user/database-setup/#Installing-specific-versions-of-ElasticSearch
- curl -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.2.0.deb && sudo dpkg -i --force-confnew elasticsearch-5.2.0.deb && sudo service elasticsearch start

before_script:
# MySQL config
- sudo sed -i'' 's/\[client\]/\[client\]\ndefault-character-set=utf8mb4/' /etc/mysql/my.cnf
Expand Down
1 change: 0 additions & 1 deletion Gulpfile.js
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,6 @@ gulp.task('js', () =>
'assets/js/dropdown-menu.js',
'assets/js/editor.js',
'assets/js/featured-resource-preview.js',
'assets/js/find-solved-topics.js',
'assets/js/form-email-username.js',
'assets/js/gallery.js',
'assets/js/jquery-tabbable.js',
Expand Down
9 changes: 9 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,13 @@ reset:
shell:
python manage.py shell

index-all:
python manage.py es_manager index_all

index-flagged:
python manage.py es_manager index_flagged


## back-utils
clean-back:
find . -name '*.pyc' -exec rm {} \;
Expand Down Expand Up @@ -89,6 +96,8 @@ help:
@echo " doc to generate the html documentation"
@echo " fixtures to load every fixtures"
@echo " generate-pdf to regenerate all PDFs"
@echo " index-all to setup and (re)index all things for search"
@echo " index-flagged to index flagged things for search"
@echo " help to get this help"
@echo " install-back to install backend dependencies"
@echo " install-front to install frontend dependencies"
Expand Down
15 changes: 0 additions & 15 deletions assets/js/find-solved-topics.js

This file was deleted.

76 changes: 0 additions & 76 deletions assets/scss/components/_home-search-box.scss

This file was deleted.

147 changes: 147 additions & 0 deletions assets/scss/components/_search-box.scss
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
.search-box {
background: white;
position: relative;
margin: 30px auto 0;
max-width: 820px;

form {
display: flex;
}

label, input {
box-sizing: border-box;
display: inline-block;
line-height: 50px;
height: 50px;
}

label {
text-align: right;
padding: 0 5px;
font-size: 2rem;
font-weight: 300;
margin-left: 50px;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C'est ce truc qui fait le décalage à cause de Clem sur la home. Il faudrait donc mettre cette margin que sur la home. (la home a une classe .home sur le body)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pataper, c'est @Situphen qui a chipoté à ça. En français, il faut faire quoi pour le corriger ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mmmh, je viens de tester, et ça empêche pas la clem' de nowel ou d'halloween de fonctionner :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah effectivement j'ai pas du faire très attention aux styles de "label" :D

}

input {
font-size: 2rem;
border: none;
font-weight: 300;
flex: 1;
}

button[type=submit] {
background-color: #FFF !important;
width: 50px;
height: 50px;

&:hover, &:focus {
background-color: #CCC !important;
}

&:after {
margin: 16px !important;
@include sprite-position($search);
@include sprite-width($search);
@include sprite-height($search);
}
}

&:before {
content: '';
position: absolute;
bottom: -6px;
left: -28px;

background: url('../images/home-clem.png');
background-size: 100%;
width: 68px;
height: 134px;

@at-root {
body.vc-clem-christmas.home .search-box::before {
background-image: url('../images/home-clem-christmas.png');
}

body.vc-clem-halloween.home .search-box::before {
background-image: url('../images/home-clem-halloween.png');
width: 160px;
left: -80px;
}
}
}

.control-group {
display: inline-flex;
width: 100%;

.controls {
width: 100%;
}

input {
padding: 0;
width: 100% !important;
}
}
}

.search-results {
.content-item {
margin-left: 0;
}
}

.search-filters {
margin-right: auto;
margin-left: auto;
max-width: 820px;
padding-left: 0;
text-align: center;
list-style: none;

li {
display: inline-flex;
padding-right: 16px;

input[type=checkbox] {
margin-top: 8px;
}
}

label {
color: white;
}
}

@media only screen and #{$media-mobile} {
.search-box {
margin: 30px 0 0!important;

padding-left: 40px;

label {
display: none;
}

&:before {
left: -46px;
}
}

.search-filters {
text-align: left;
padding-left: 10px;

li {
display: list-item;
}
}
}

@media only screen and #{$media-mobile-tablet} {
.search-box {
margin: 30px 40px 0;
}
}

4 changes: 3 additions & 1 deletion assets/scss/components/_topic-list.scss
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,9 @@
}

.highlighted {
background-color: rgba(255, 255, 100, 0.5);
background-color: rgba(255, 255, 100, 0.4);
padding: 0 3px;
border-radius: 2px;
}
}

Expand Down
4 changes: 3 additions & 1 deletion assets/scss/main.scss
Original file line number Diff line number Diff line change
Expand Up @@ -62,12 +62,12 @@
@import "components/content-item";
@import "components/editor";
@import "components/featured-item";
@import "components/home-search-box";
@import "components/markdown-help";
@import "components/mobile-menu";
@import "components/modals";
@import "components/pagination";
@import "components/pygments";
@import "components/search-box";
@import "components/tags";
@import "components/tooltips";
@import "components/topic-list";
Expand All @@ -78,9 +78,11 @@
/*------------------------
9. Pages
------------------------*/
@import "pages/flexpage";
@import "pages/home";
@import "pages/gallery";
@import "pages/api";
@import "pages/search";
@import "pages/tutorial-help";
@import "pages/tutorial-history";

Expand Down
38 changes: 38 additions & 0 deletions assets/scss/pages/_flexpage.scss
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
$content-width: 1145px;

.flexpage {
.main {
display: block;
height: auto;
width: auto;
margin: 0;
padding: 0;
}

#content {
width: 100%;
margin: 0;
padding: 0;
}

.sub-header{
display: none;
}

.flexpage-wrapper {
max-width: $content-width;
margin: 0 auto;
}

.flexpage-header {
border-bottom: solid 1px white;
background-color: #19516b; // fallback for older browser
background: #19516b radial-gradient(at top, rgba(255,255,255,0.1),rgba(0,0,0,0) 60%);
}
}

@media only screen and #{$media-mobile-tablet} {
.flexpage-wrapper {
padding: 20px 10px;
}
}
Loading