Ingested data missing after restarting docker compose #1650
Comments
|
Could you post the permission error you got for ES volume sharing? |
|
|
|
Are you using the quickstart script or manually doing |
|
I am using the quickstart script. |
|
I see. So did the script create |
|
Yes, it did.
…On Mon., Apr. 27, 2020, 18:04 Mars Lan, ***@***.***> wrote:
I see. So did the script create /tmp/datahub/elasticsearch and set the
permission to 777?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#1650 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AE32DLGBQIEDSWXMT4MQ2IDROX6OVANCNFSM4MSCXO2A>
.
|
|
Hum this is strange. I'm unable to reproduce this from my end. Are you running on Mac or Linux? Also could you try a clean checkout, clean up your docker setup and retry the quickstart script again just to be safe? |
|
I am using linux system (Ubuntu 18.04.3 LTS). I have tried after cleaning the docker containers but in vain. |
Volume is the preferred method over bind mount (https://docs.docker.com/storage/volumes/) for persistent container data. This also eliminate the need for the ugly chmod hack for elasticsearch and hopefully fix datahub-project#1650
Volume is the preferred method over bind mount (https://docs.docker.com/storage/volumes/) for persistent container data. This also eliminate the need for the ugly chmod hack for elasticsearch and hopefully fix datahub-project#1650
Volume is the preferred method over bind mount (https://docs.docker.com/storage/volumes/) for persistent container data. This also eliminates the need for the ugly chmod hack for elasticsearch and hopefully fixes datahub-project#1650
Volume is the preferred method over bind mount (https://docs.docker.com/storage/volumes/) for persistent container data. This also eliminates the need for the ugly chmod hack for elasticsearch and hopefully fixes #1650
|
@Manjeri-Rajan we have now changed the quickstart docker-compose from bind mount to named volume, which will hopefully fix this issue. Please give it a try and let us know how it goes. |
|
Thanks Mars. Will take try it. |
|
That worked. Thank you! |
|
Great to hear! |
I have installed datahub in my machine and ingested Google Big Query data using my extraction script. But I am facing the below issue.
Due to less swap memory(980 MB), my system gets stuck after a certain time after data is ingested to datahub and I have to setup the docker compose again. This clears the already ingested data and I have run to the ingestion script again.
My understanding is that the volume sharing will save the ingested data locally. But the volume sharing for mysql and neo4j is not happening.
Also the elastic search volume sharing is giving me permission error(I had to comment out that piece of code to run the docker compose).
Is there an approach to avoid ingesting the already ingested data.
The text was updated successfully, but these errors were encountered: