Firestore and Database Emulator: Initialization of an instance with a dataset #1167

gaeldestrem · 2019-03-12T16:15:45Z

It would be great to have the possibility to create an instance of the Firestore emulator with a dataset.

Currently, we're doing a batch write to initialize or reset an instance of an emulator. The issue is that it takes nearly 3s for an update of 120 operations (our testing dataset).

Would it be possible to create a Firestore emulator instance from a snapshot or to improve the performances of the emulator for large updates?

wyattades · 2019-10-12T18:54:24Z

I agree that it would be great to load a database "snapshot" on initialization. I currently run the firestore and functions emulators locally for development, but when I seed the DB with testing data, a bunch of firestore function triggers are executed. This is pretty undesirable, so are there any plans for a feature to help with loading initial data or, even better, firestore data persistence to disk?

samtstern · 2019-10-12T19:12:36Z

Thanks for raising the point about wanting to seed data without functions firing. As a stopgap measure we could definitely implement a way to "turn off" functions triggers with a special call while you load data. That could be useful for certain tests too. Would you use that? But for the long term we need import/export.

…

On Sat, Oct 12, 2019, 11:54 AM Wyatt Ades ***@***.***> wrote: I agree that it would be great to load a database "snapshot" on initialization. I currently run the firestore and functions emulators locally for development, but when I seed the DB with testing data, a bunch of firestore function triggers are executed. This is pretty undesirable, so are there any plans for a feature to help with loading initial data or, even better, firestore data persistence to disk? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1167?email_source=notifications&email_token=ACATB2VEE7BBI75HECFTE23QOIMOLA5CNFSM4G5NG5X2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBCGBZA#issuecomment-541352164>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACATB2WHUTO43JISN54J4U3QOIMOLANCNFSM4G5NG5XQ> .

wyattades · 2019-10-12T20:39:42Z

Yes that would be a good solution for now, thanks!

anantanandgupta · 2019-10-14T19:47:13Z

So there are no milestones set for this yet? but one thing I like now about this team here, it seems to be more active and responsive.

anantanandgupta · 2019-10-14T19:55:10Z

can guys please add realtime db also in scope for this along with firestore db.

samtstern · 2019-10-14T20:05:56Z

@anantanandgupta no milestones yet, but yes we try to be really open about out progress! And I have added the Database emulator to be in scope as well. For Database the problem is easier (JSON format) but we still need to help you "pause" the Cloud Functions

anantanandgupta · 2019-10-14T21:18:28Z

I know I am in good hands. But to tell you the criticality, I have designed this huge platform fully around firebase and everyone was against this a yesr back, but with my efforts i have proven till now that my decision to use firebase as the technology was not wrong. At this point it has become so big that to work on multiple features by multiple developers with a common database hosted on cloud has become impossible. So if each of us can work on a local copy of the database, it will be a lot easier and efficient.

ryanpbrewster · 2019-10-23T21:19:55Z

The ability to import data into the Firestore emulator (i.e., initialize it with "seed" data) should be going out with the next release. It will be a bit manual at first, we'll add better tooling in subsequent releases.

Looking into what would be necessary for the Realtime Database emulator.

SamyPesse · 2019-12-15T20:13:38Z

@ryanpbrewster is there any documentation on how to setup "seed" data ?

samtstern · 2019-12-16T18:17:09Z

@SamyPesse this has been implemented inside the Emulator JAR itself (as @ryanpbrewster said) but we have not actually exposed an API through firebase-tools to do import so this isn't really something you can use yet.

Would you mind sharing how you'd expect this to work? Would you prefer a flag on the firebase command, a field in firebase.json or something else?

ryanpbrewster · 2019-12-16T20:47:09Z

Importing and exporting data is currently implemented, but we haven't quite figured out how to make it easy to use these features. So keep in mind that we know these instructions are rough, and we're open to ideas about how to improve this:

Get an export

You can generate an export from your production Firestore instance, or from the local emulator.

Emulator export

Boot up the emulator and populate it with data however you'd like. Once you have your database in a state that you like, create an export:

curl "localhost:8080/emulator/v1/projects/ryanpbrewster-test:export" \
  -d '{"database": "projects/ryanpbrewster-test/databases/(default)","export_directory":"."}' \
  -H "Content-Type: application/json

which will dump the export onto your local filesystem in a directory like firestore_export_1576524691/

Production export

You'll need a Google Cloud Storage bucket. I'll use my test project's default AppEngine bucket ryanpbrewster-test.appspot.com. Run

gcloud firestore export gs://ryanpbrewster-test.appspot.com/firestore-export

and Firestore will dump a full copy of your production data into Google Cloud Storage. If you'd like, you can use the --collection-ids flag to limit the export to a specific set of collections.

Once it's done you can copy it onto your local filesystem (note that this is going to go poorly if you have a huge database):

gsutil cp -r gs://ryanpbrewster-test.appspot.com/firestore-export .

Import data

For now, the only way to import data into the emulator is via a command line flag:

java -jar ~/.cache/firebase/emulators/cloud-firestore-emulator-v1.10.2.jar \
  --seed_from_export=firestore_export_1576524691/firestore_export_1576524691.overall_export_metadata

If all goes well, the emulator will start up and have all of your data in it.

Note that the path it's expecting is the location of the .overall_export_metadata file, which should be at the top level of the export directory.

prgwar · 2019-12-25T07:55:05Z

I was able to emulate the firestore data from the export. Now If I try to start the firebase emulator suit. Firestore emulator is not working because the emulator is already running in the port. I tried even setting the FIRESTORE_EMULATOR_HOST variable its not recognizing. If the seed path can also be passed along the command while starting the emulator it will be much more helpful and reduce a lot of development time like firebase emulators:start --firestore_seed_from_export = \path\to\seed\file

srolfe26 · 2019-12-29T07:18:52Z

+1 for @prgwar 's suggestion. How can I get the emulator suite to run using this startup firestore? Functions in the emulator use production because "⚠ functions: The Cloud Firestore emulator is not running, so calls to Firestore will affect production."

samtstern · 2020-01-02T17:29:55Z

@prgwar thanks for your suggestion, we are currently working on deciding the API for specifying import/export from the firebase command. Using the jar directly is just a workaround and not something we actually expect developers to do in the long term.

@srolfe26 your issue sounds like a different one, you're running the Cloud Functions emulator but not the Firestore emulator. Try running firebase init emulators to initialize the Firestore emulator and then run firebase emulators:start again.

amkoehler · 2020-01-13T16:10:06Z

For any fellow WSL (1 or 2) users out there, you may have trouble connecting to the emulator from your browser. I added a --host 0.0.0.0 flag to the above command and that fixed the issue.

java -jar ~/.cache/firebase/emulators/cloud-firestore-emulator-v1.10.2.jar \
  --seed_from_export=firestore_export_1576524691/firestore_export_1576524691.overall_export_metadata \
  --host 0.0.0.0

spencerwhyte · 2020-02-08T22:00:41Z

Would @prgwar 's suggestion of using --firestore_seed_from_export= be acceptable?

If so, I'm willing to put up a PR. How else can I help get the ball rolling?

I would also love if the same seed_from_export functionality could be implemented on the Realtime Database emulator.

samtstern · 2020-02-11T00:19:14Z

@spencerwhyte thanks for the offer! I am actively tackling this right now, working on something that will support Firestore and RTDB and also any future emulators.

How would you feel about this general flow:

A "backup" is a folder that contains:
- A bunch of individual emulator backups in their canonical formats
- A single metadata.json file that describes them
To start the emulators with a seed you do: firebase emulators:start --import="./folder-with-backup"
To take a backup of the running emulators you open a new terminal while they are running and do firebase emulators:export

That's what I am currently working on, I think it's the most flexible thing but I am very open to feedback

spencerwhyte · 2020-02-11T06:07:41Z

@samtstern That's great to hear!

This design sounds perfect, just as long as the emulator export format / backup format is something that we can work with. If it's the same format as the backups on production, that'll do. We have a need to import a large amount of data for our integration tests without firing off our triggers. Currently we have been surviving by doing some checks that disable triggers when under test, however we'd love to avoid that complexity. This support for --import will allow us to simplify things.

We use firebase emulators:exec for our tests, so as long as it will work with exec as well, we'd be happy.

samtstern · 2020-02-11T17:20:12Z

@spencerwhyte

Yes we will definitely stick to the same formats as production whenever possible. So for Firestore we'll support the import/export format and for RTDB it will just be a giant json file.
Yes we will support emulators:exec --import

Thanks for the feedback!

samtstern · 2020-02-18T18:28:42Z

PR #1968 implements this feature (for Firestore only at the moment, Realtime Database coming soon) and will be included in the next CLI release (likely 7.14.0)

samtstern · 2020-02-26T22:34:34Z

This feature was just released in 7.14.0 ... give it a try and let us know what you think! Right now it's Firestore only but Realtime Database is up next.

shaimo · 2020-03-08T17:43:39Z

Is this documented anywhere? Also, is there a way to export the production data without a Google Cloud Storage bucket (which I don't have)?
Having hard time testing triggers locally without data replicated from the cloud datastore.

samtstern · 2020-03-10T22:11:39Z

@shaimo sorry we're a bit behind on docs but they should be out in the next few days!

To export from production you do need a Cloud Storage bucket. This is because export is a "managed" operation which means it happens on a Google server somewhere. We can't run an operation like that against your local filesystem, we need a cloud destination.

If you have a Firebase project you actually do have a default Cloud Storage bucket you can use and it includes some free quota. Just navigate over to the Storage section of the Firebase console to check it out.

manwithsteelnerves · 2020-03-14T05:55:10Z

@samtstern As the documentation may take some time, can you please give a brief on how to import/export to emulator instance?

samtstern · 2020-03-16T08:38:25Z

We now have some documentation on the new flags here:
https://firebase.google.com/docs/emulator-suite/install_and_configure#startup

But let me give you an example anyway. To use import/export you will need two terminal windows so let's call them Terminal 1 and Terminal 2

First start the emulators

Terminal 1

$ firebase emulators:start
# ...

After you have built up some data in them that you would like to save, run the export command in another terminal:

Terminal 2

$ firebase emulators:export ./mydirectory
# ...

Now you have an export in ./mydirectory. Next time you start the emulators, tell them to import from there:

Terminal 1

$ firebase emulators:start --import=./mydirectory
# ...

And that's it!

landonj · 2020-03-16T19:14:48Z

@samtstern Can we import from a production export using the firebase emulators:start --import command?
I see that emulators:start --import is looking for firebase-export-metadata.json but I have firestore-export/firestore-export.overall_export_metadata

samtstern · 2020-03-16T19:35:17Z

@landonj you can but it's a little tricky at the moment. The simplest thing to do is run emulators:export once and then modify the JSON files to point to your production export rather than the local one.

FergyKay · 2020-03-30T09:43:12Z

Anything on seeding RTDB for emulator yet?

samtstern · 2020-03-30T13:24:34Z

@FergyKay nothing yet but you can follow along here:
#2079

jQrgen · 2020-04-06T16:48:39Z

This is much needed

Ridermansb · 2020-05-31T13:37:30Z

I'm trying to do the exact same thing as @landonj ..

and I tried what @samtstern told ... but I'm receiving an error...

What I'm doing wrong ?

bdw.. the file exists!

firebase emulators:start --debug --only firestore,functions --import=/Users/ridermansb/Projects/meve.ai/backups/emulator-data

[2020-05-31T13:31:23.601Z] com.google.cloud.datastore.core.exception.DatastoreException: Failed to parse overall export metadata file
at com.google.cloud.datastore.emulator.impl.ExportImportUtil.parseOverallMetadataFile(ExportImportUtil.java:225)
at com.google.cloud.datastore.emulator.impl.ExportImportUtil.fetchEntities(ExportImportUtil.java:54)
at com.google.cloud.datastore.emulator.firestore.CloudFirestore.main(CloudFirestore.java:89)
{"metadata":{"emulator":{"name":"firestore"},"message":"com.google.cloud.datastore.core.exception.DatastoreException: Failed to parse overall export metadata file\n\tat com.google.cloud.datastore.emulator.impl.ExportImportUtil.parseOverallMetadataFile(ExportImportUtil.java:225)\n\tat com.google.cloud.datastore.emulator.impl.ExportImportUtil.fetchEntities(ExportImportUtil.java:54)\n\tat com.google.cloud.datastore.emulator.firestore.CloudFirestore.main(CloudFirestore.java:89)\n"}}
[2020-05-31T13:31:23.602Z] Caused by: java.io.FileNotFoundException: /Users/ridermansb/Projects/meve.ai/backups/emulator-data/Users/ridermansb/Projects/meve.ai/backups/2020-05-31/2020-05-31/2020-05-31T13:19:28_6168.overall_export_metadata (No such file or directory)
{"metadata":{"emulator":{"name":"firestore"},"message":"Caused by: java.io.FileNotFoundException: /Users/ridermansb/Projects/meve.ai/backups/emulator-data/Users/ridermansb/Projects/meve.ai/backups/2020-05-31/2020-05-31/2020-05-31T13:19:28_6168.overall_export_metadata (No such file or directory)\n"}}
[2020-05-31T13:31:23.603Z] at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:213)
at java.base/java.io.FileInputStream.(FileInputStream.java:155)
at com.google.cloud.datastore.emulator.impl.ExportImportUtil.parseOverallMetadataFile(ExportImportUtil.java:215)
... 2 more
{"metadata":{"emulator":{"name":"firestore"},"message":"\tat java.base/java.io.FileInputStream.open0(Native Method)\n\tat java.base/java.io.FileInputStream.open(FileInputStream.java:213)\n\tat java.base/java.io.FileInputStream.(FileInputStream.java:155)\n\tat com.google.cloud.datastore.emulator.impl.ExportImportUtil.parseOverallMetadataFile(ExportImportUtil.java:215)\n\t... 2 more\n"}}

Error: firestore: Firestore Emulator has exited with code: 1

gugahoi · 2020-06-01T11:41:53Z

@landonj you can but it's a little tricky at the moment. The simplest thing to do is run emulators:export once and then modify the JSON files to point to your production export rather than the local one.

Can you expand on this? Can I point directly to the gcloud bucket? Or must I first download the contents of the export locally? Which files do I need to modify?

The following is what I have attempted:

> firebase emulators:start
> firebase emulators:export ~/emulator-data/
> # close the emulator
> gcloud firestore export gs://xxxxx-xxxxx-xxxxx/emulator/
> gsutil cp -r gs://xxxxx-xxxxx-xxxxx/emulator ~/emulator-data/
> firebase emulators:start --only firestore --import /home/gugahoi/emulator-data/emulator/ --debug
[2020-06-01T10:04:36.492Z] > command requires scopes: ["email","openid","https://www.googleapis.com/auth/cloudplatformprojects.readonly","https://www.googleapis.com/auth/firebase","https://www.googleapis.com/auth/cloud-platform"]
[2020-06-01T10:04:36.493Z] > authorizing via signed-in user
i  emulators: Starting emulators: firestore {"metadata":{"emulator":{"name":"hub"},"message":"Starting emulators: firestore"}}
[2020-06-01T10:04:36.510Z] [hub] writing locator at /tmp/hub-flux-development.json
i  emulators: Shutting down emulators. {"metadata":{"emulator":{"name":"hub"},"message":"Shutting down emulators."}}
i  hub: Stopping emulator hub {"metadata":{"emulator":{"name":"hub"},"message":"Stopping emulator hub"}}
[2020-06-01T10:04:36.825Z] Error: ENOENT: no such file or directory, open '/home/gugahoi/emulator-data/emulator/firebase-export-metadata.json'
    at Object.openSync (fs.js:458:3)
    at Object.readFileSync (fs.js:360:35)
    at Object.<anonymous> (/home/gugahoi/.nvm/versions/node/v12.16.3/lib/node_modules/firebase-tools/lib/emulator/controller.js:183:44)
    at Generator.next (<anonymous>)
    at fulfilled (/home/gugahoi/.nvm/versions/node/v12.16.3/lib/node_modules/firebase-tools/lib/emulator/controller.js:5:58)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)

Error: An unexpected error has occurred.

As the above did not seem to work, I updated the firebase-exports-metadata.json file with the following contents:

> cat firebase-export-metadata.json
{"version":"8.4.1","firestore":{"version":"1.11.4","path":"emulator","metadata_file":"emulator/emulator.overall_export_metadata"}}

Below is the directory structure:

gugahoi:~/emulator-data > tree .
.
├── emulator
│   ├── all_namespaces
│   │   └── all_kinds
│   │       ├── all_namespaces_all_kinds.export_metadata
│   │       ├── output-0
│   │       └── output-1
│   └── emulator.overall_export_metadata
├── firebase-export-metadata.json
├── firebase-export-metadata.json.bak
└── firestore_export
    └── firestore_export.overall_export_metadata

However I still have no data in the emulator UI.

Edit

Just thought I'd add the final output from the emulator, it runs without error but without my data:

> firebase emulators:start --only firestore --import /home/gugahoi/emulator-data/ --debug
[2020-06-01T11:31:34.256Z] > command requires scopes: ["email","openid","https://www.googleapis.com/auth/cloudplatformprojects.readonly","https://www.googleapis.com/auth/firebase","https://www.googleapis.com/auth/cloud-platform"]
[2020-06-01T11:31:34.257Z] > authorizing via signed-in user
i  emulators: Starting emulators: firestore {"metadata":{"emulator":{"name":"hub"},"message":"Starting emulators: firestore"}}
[2020-06-01T11:31:34.274Z] [hub] writing locator at /tmp/hub-flux-development.json
i  firestore: Importing data from /home/gugahoi/emulator-data/emulator/emulator.overall_export_metadata {"metadata":{"emulator":{"name":"firestore"},"message":"Importing data from /home/gugahoi/emulator-data/emulator/emulator.overall_export_metadata"}}
[2020-06-01T11:31:34.302Z] Ignoring unsupported arg: projectId {"metadata":{"emulator":{"name":"firestore"},"message":"Ignoring unsupported arg: projectId"}}
[2020-06-01T11:31:34.303Z] Ignoring unsupported arg: auto_download {"metadata":{"emulator":{"name":"firestore"},"message":"Ignoring unsupported arg: auto_download"}}
[2020-06-01T11:31:34.303Z] Starting Firestore Emulator with command {"binary":"java","args":["-Duser.language=en","-jar","/home/gugahoi/.cache/firebase/emulators/cloud-firestore-emulator-v1.11.4.jar","--host","localhost","--port",8080,"--seed_from_export","/home/gugahoi/emulator-data/emulator/emulator.overall_export_metadata","--rules","/home/gugahoi/projects/flux/flux/firestore.rules"],"optionalArgs":["port","webchannel_port","host","rules","functions_emulator","seed_from_export"],"joinArgs":false} {"metadata":{"emulator":{"name":"firestore"},"message":"Starting Firestore Emulator with command {\"binary\":\"java\",\"args\":[\"-Duser.language=en\",\"-jar\",\"/home/gugahoi/.cache/firebase/emulators/cloud-firestore-emulator-v1.11.4.jar\",\"--host\",\"localhost\",\"--port\",8080,\"--seed_from_export\",\"/home/gugahoi/emulator-data/emulator/emulator.overall_export_metadata\",\"--rules\",\"/home/gugahoi/projects/flux/flux/firestore.rules\"],\"optionalArgs\":[\"port\",\"webchannel_port\",\"host\",\"rules\",\"functions_emulator\",\"seed_from_export\"],\"joinArgs\":false}"}}
i  firestore: Firestore Emulator logging to firestore-debug.log {"metadata":{"emulator":{"name":"firestore"},"message":"Firestore Emulator logging to \u001b[1mfirestore-debug.log\u001b[22m"}}
[2020-06-01T11:31:35.197Z] API endpoint: http://localhost:8080
 {"metadata":{"emulator":{"name":"firestore"},"message":"API endpoint: http://localhost:8080\n"}}
[2020-06-01T11:31:35.198Z] If you are using a library that supports the FIRESTORE_EMULATOR_HOST environment variable, run:

   export FIRESTORE_EMULATOR_HOST=localhost:8080

Dev App Server is now running.

 {"metadata":{"emulator":{"name":"firestore"},"message":"If you are using a library that supports the FIRESTORE_EMULATOR_HOST environment variable, run:\n\n   export FIRESTORE_EMULATOR_HOST=localhost:8080\n\nDev App Server is now running.\n\n"}}
[2020-06-01T11:31:35.322Z] Ignoring unsupported arg: auto_download {"metadata":{"emulator":{"name":"ui"},"message":"Ignoring unsupported arg: auto_download"}}
[2020-06-01T11:31:35.322Z] Ignoring unsupported arg: port {"metadata":{"emulator":{"name":"ui"},"message":"Ignoring unsupported arg: port"}}
[2020-06-01T11:31:35.323Z] Starting Emulator UI with command {"binary":"node","args":["/home/gugahoi/.cache/firebase/emulators/ui-v1.0.0/server.bundle.js"],"optionalArgs":[],"joinArgs":false} {"metadata":{"emulator":{"name":"ui"},"message":"Starting Emulator UI with command {\"binary\":\"node\",\"args\":[\"/home/gugahoi/.cache/firebase/emulators/ui-v1.0.0/server.bundle.js\"],\"optionalArgs\":[],\"joinArgs\":false}"}}
i  ui: Emulator UI logging to ui-debug.log {"metadata":{"emulator":{"name":"ui"},"message":"Emulator UI logging to \u001b[1mui-debug.log\u001b[22m"}}
[2020-06-01T11:31:35.395Z] Web / API server started at http://localhost:4000
 {"metadata":{"emulator":{"name":"ui"},"message":"Web / API server started at http://localhost:4000\n"}}

┌───────────────────────────────────────────────────────────────────────┐
│ ✔  All emulators ready! View status and logs at http://localhost:4000 │
└───────────────────────────────────────────────────────────────────────┘

┌───────────┬────────────────┬─────────────────────────────────┐
│ Emulator  │ Host:Port      │ View in Emulator UI             │
├───────────┼────────────────┼─────────────────────────────────┤
│ Firestore │ localhost:8080 │ http://localhost:4000/firestore │
└───────────┴────────────────┴─────────────────────────────────┘
  Other reserved ports: 4400, 4500

Issues? Report them at https://github.com/firebase/firebase-tools/issues and attach the *-debug.log files.

bkuri · 2020-06-07T23:12:10Z

@gugahoi your post helped me come up with a solution: https://gist.github.com/bkuri/36cef71bcf30a5416b1c353cb81f38ed

It's a small bash script that seeds your cloud storage emulator instance. All it needs is the project name and a destination path to save exported data to. ie: ./seed-storage.sh my-project /tmp/export.

BTW, the above script depends on two very common tools (lsof and screen) in addition to the usual suspects (firebase, gcloud and gsutil), so make sure you have them installed.

Hope this helps someone. 👍

manwithsteelnerves · 2020-06-08T08:12:56Z

Anyway to avoid triggering of firestore functions on importing an existing database?

gugahoi · 2020-06-15T08:23:36Z

@bkuri thanks for that script, I guess there is something odd going on with my setup/installation as I am still unable to get any data to show up in the firestore emulator console.

metafaniel · 2020-06-24T04:40:31Z

Thanks to @gugahoi I was able to import a collection from production into the local emulator.
To do so I did the following steps:

Go to my local Firebase project path.
Start the emulators using: firebase emulators:start
Create manually some mockup data using the GUI at http://localhost:4000/firestore using the buttons provided: + Start Collection and + Add Document.
Export this data locally using: emulators:export ./mydirectory
About the project data located at Firebase Database / Cloud Firestore, I exported a single collection like this: gcloud firestore export gs://my-project-bucket-id.appspot.com --collection-ids=myCollection The export is now located under Firebase Storage in a folder with a timestamp as name (I didn't use a prefix for my test)
Download this folder to local drive with: gsutil cp -r gs://my-project-bucket-id.appspot.com/myCollection ./production_data_export NOTE: I did this in a Windows environment... gsutil will throw this error: "OSError: The filename, directory name, or volume label syntax is incorrect" if the folder has invalid characters for a folder name in Windows (i.e. colons) or this error: "OSError: Invalid argument.9.0 B]" if an inner file in the folder has invalid characters too. To be able to download the export locally, rename these with a valid Windows name (i.e. removing the colons) like this: gsutil mv gs://my-project-bucket-id.appspot.com/2020-05-22T02:01:06_86152 gs://my-project-bucket-id.appspot.com/myCollection
Once downloaded, imitate the local export structure renaming the folder to firestore_export and copying the firebase-export-metadata.json file from the local export folder. Just to be visual, here's the structure I got:

$ tree .
.
├── local_data_export
│   ├── firebase-export-metadata.json
│   └── firestore_export
│       ├── all_namespaces
│       │   └── all_kinds
│       │       ├── all_namespaces_all_kinds.export_metadata
│       │       └── output-0
│       └── firestore_export.overall_export_metadata
└── production_data_export
    ├── firebase-export-metadata.json
    └── firestore_export
        ├── all_namespaces
        │   └── kind_myCollection
        │       ├── all_namespaces_kind_myCollection.export_metadata
        │       ├── output-0
        │       └── output-1
        └── firestore_export.overall_export_metadata

8 directories, 9 files

Finally, start the local emulator pointing to this production data to be imported: firebase emulators:start --import=./mock_up_data/production_data_export/
You should see the imported data at: http://localhost:4000/firestore/

I wish these instructions were clear enough to be helpful. I hope this will help someone while we wait for a more robust solution from the Firebase folks.

gugahoi · 2020-06-24T04:59:32Z

Just as a brief update: when I posted the instructions from the comment above, I was doing it in WSL2 and they didn't seem to work but the same steps in Ubuntu seem to be fine.I suspect there might be something in WSL2 interfering.

nvanhoesel · 2020-11-18T08:53:36Z

I want to do this the other way around i want to import my emulator data into production, I need to add 7000 google sheet rows to my firestore and i was able to do that by editing a function however when trying to do it in production my memory limit gets exceeded also when something goes wrong inside the function i might end up with duplicate data. I noticed the export import and i noticed that i can make an export of my local emulator firestore data. But how about importing my local backup into production ?

samtstern · 2020-11-20T11:54:36Z

@nvanhoesel the import/export formats are the same in production and in the emulators. But yours is an uncommon use case so you'll need to do some work yourself to get the emulator export into Google Cloud Storage so that Firestore can import it properrly.

cedvdb · 2022-02-08T20:38:54Z

Is there a way to reset it to initial imported data while it is running, like an endpoint that can be called from code ? How do people even run integration tests without this ?

DiegoGonzalezCruz · 2023-10-13T16:10:32Z

Is this still working? I'm getting errors exit code 1.
If I import just a few collections , it works fine.

mbleigh assigned ryanpbrewster Mar 12, 2019

bkendall added the emulators: firestore label Mar 12, 2019

samtstern added the emulator-suite label May 16, 2019

samtstern added the type: feature request label Jul 17, 2019

samtstern mentioned this issue Oct 14, 2019

In emulators, realtime database, firestore write (via emulator functions) is showing success but can't see data in the firebase console #1658

Closed

samtstern changed the title ~~Firestore Emulator: Initialization of an instance with a dataset~~ Firestore and Database Emulator: Initialization of an instance with a dataset Oct 14, 2019

samtstern added the emulators: database label Oct 14, 2019

ryanpbrewster removed their assignment Jan 16, 2020

samtstern self-assigned this Jan 28, 2020

samtstern mentioned this issue Feb 11, 2020

Import/export support for emulators #1968

Merged

2 tasks

samtstern closed this as completed in #1968 Feb 18, 2020

samtstern mentioned this issue Mar 30, 2020

[Tracking] Add import/export support to RTDB emulator #2079

Closed

snyk-bot mentioned this issue Apr 4, 2020

[Snyk] Upgrade firebase-tools from 7.14.0 to 7.15.0 adwd/saito-sensei#159

Open

prescottprue mentioned this issue Apr 24, 2020

Import Prod Firestore Data Into Emulator #2175

Closed

hoangofHUST mentioned this issue May 7, 2020

Firebase emulator export doesnt work #2218

Closed

adiazarya100 mentioned this issue May 13, 2020

Write back from local web application to the local emulator firestore database is not working #2237

Closed

jkomoros mentioned this issue Jun 20, 2020

Support local emulator development jkomoros/card-web#324

Open

Firestore and Database Emulator: Initialization of an instance with a dataset #1167

Firestore and Database Emulator: Initialization of an instance with a dataset #1167

Comments

gaeldestrem commented Mar 12, 2019

wyattades commented Oct 12, 2019

samtstern commented Oct 12, 2019 via email

wyattades commented Oct 12, 2019

anantanandgupta commented Oct 14, 2019

anantanandgupta commented Oct 14, 2019

samtstern commented Oct 14, 2019

anantanandgupta commented Oct 14, 2019 • edited Loading

ryanpbrewster commented Oct 23, 2019

SamyPesse commented Dec 15, 2019

samtstern commented Dec 16, 2019

ryanpbrewster commented Dec 16, 2019

Get an export

Emulator export

Production export

Import data

prgwar commented Dec 25, 2019

srolfe26 commented Dec 29, 2019

samtstern commented Jan 2, 2020

amkoehler commented Jan 13, 2020

spencerwhyte commented Feb 8, 2020

samtstern commented Feb 11, 2020

spencerwhyte commented Feb 11, 2020 • edited Loading

samtstern commented Feb 11, 2020

samtstern commented Feb 18, 2020

samtstern commented Feb 26, 2020

shaimo commented Mar 8, 2020

samtstern commented Mar 10, 2020

manwithsteelnerves commented Mar 14, 2020 • edited Loading

samtstern commented Mar 16, 2020 • edited Loading

landonj commented Mar 16, 2020

samtstern commented Mar 16, 2020

FergyKay commented Mar 30, 2020

samtstern commented Mar 30, 2020 • edited Loading

jQrgen commented Apr 6, 2020

Ridermansb commented May 31, 2020 • edited Loading

gugahoi commented Jun 1, 2020 • edited Loading

Edit

bkuri commented Jun 7, 2020

manwithsteelnerves commented Jun 8, 2020

gugahoi commented Jun 15, 2020

metafaniel commented Jun 24, 2020 • edited Loading

gugahoi commented Jun 24, 2020

nvanhoesel commented Nov 18, 2020

samtstern commented Nov 20, 2020

cedvdb commented Feb 8, 2022 • edited Loading

DiegoGonzalezCruz commented Oct 13, 2023

anantanandgupta commented Oct 14, 2019 •

edited

Loading

spencerwhyte commented Feb 11, 2020 •

edited

Loading

manwithsteelnerves commented Mar 14, 2020 •

edited

Loading

samtstern commented Mar 16, 2020 •

edited

Loading

samtstern commented Mar 30, 2020 •

edited

Loading

Ridermansb commented May 31, 2020 •

edited

Loading

gugahoi commented Jun 1, 2020 •

edited

Loading

metafaniel commented Jun 24, 2020 •

edited

Loading

cedvdb commented Feb 8, 2022 •

edited

Loading