Skip to content

Commit

Permalink
fix (WIP): Use File.stream!
Browse files Browse the repository at this point in the history
This seems to avoid memory issues, but means that we can't stream lines
in from stdin.
  • Loading branch information
arkadyan committed Oct 3, 2023
1 parent 46b2f45 commit 5a72a3d
Showing 1 changed file with 10 additions and 14 deletions.
24 changes: 10 additions & 14 deletions scripts/sanitize_db_dump.exs
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,18 @@ defmodule SanitizeDbDump do
NOTE: Currently we are "cheating" by relying on the knowledge that email addresses and phone numbers come one after the other in our users table definition. This lets us limit where we look for 10-digit numbers and avoid timestamps and other items that aren't actually phone numbers but look the same.
To dump from production, sanitize, and load into dev locally:
Run like:
```
PGPASSWORD=<PROD PASSWORD> pg_dump --host=alerts-concierge-prod.cw84s0ixvuei.us-east-1.rds.amazonaws.com --port=5432 --username=alerts_concierge --dbname=alerts_concierge_prod --table=users --table=trips --table=subscriptions --no-owner --data-only | elixir ./scripts/sanitize_db_dump.exs | psql -d alert_concierge_dev
elixir ./scripts/sanitize_db_dump.exs /path/to/input_file.sql
```
"""

def run() do
:stdio
|> IO.stream(:line)
@chars ~c(abcdefghijklmnopqrstuvwxyz0123456789)

def run(file) do
file
|> File.stream!()
|> Stream.map(&sanitize_an_email_addresses_followed_by_a_phone_number/1)
|> Stream.map(&sanitize_an_email_addresses_followed_by_null/1)
|> Enum.each(&IO.write(&1))
Expand All @@ -32,16 +34,10 @@ defmodule SanitizeDbDump do
end

defp mock_email() do
chars = String.split("abcdefghijklmnopqrstuvwxyz0123456789", "")

address =
Enum.reduce(1..10, [], fn _i, acc ->
[Enum.random(chars) | acc]
end)
|> Enum.join("")

address = for _ <- 1..10, into: "", do: <<Enum.random(@chars)>>
"#{address}@example.com"
end
end

SanitizeDbDump.run()
file = System.argv() |> List.first()
SanitizeDbDump.run(file)

0 comments on commit 5a72a3d

Please sign in to comment.