Add Import#json, Export#json, and Export#email

NEWS.md

@@ -2,6 +2,29 @@ Unreleased
 
 - Drops support for Ruby 3.0, since it is EOL.
 - Drops support for Rails 6.
+- Adds `Import#json`, `Export#json`, and `Export#email` (#34).
+
+ The `#json` interface for importing and exporting JSON have been designed to work the same way they already work for the CSV interfaces. For example:
+
+ ```ruby
+ json_string = [
+ {
+ email: "george@vandelay_industries.com",
+ password: "bosco"
+ }
+ ].to_json
+
+ result = ArtVandelay::Import.new(:users).json(json_string, attributes: {email_address: :email, passcode: :password})
+ ```
+
+ `ArtVandelay::Export#email_csv` has been changed to a more-generic `ArtVandelay::Export#email` method that takes a new `:format` option. The new option defaults to `:csv` but can also be used with `:json`. Since the old `#email_csv` method no longer exists, you'll need to update your application code accordingly. For example:
+
+ ```diff
+ -ArtVandelay::Export.email_csv(to: my_email_address)
+ +ArtVandelay::Export.email(to: my_email_address)
+ ```
+
+ *Benjamin Wil*
 
 0.2.0 (June 15, 2023)
 

README.md

@@ -4,14 +4,14 @@ Demo](https://github.com/thoughtbot/art_vandelay/actions/workflows/ci.yml/badge.
 
 Art Vandelay is an importer/exporter for Rails 7.0 and higher.
 
-Have you ever been on a project where, out of nowhere, someone asks you to send them a CSV of data? You think to yourself, “Ok, cool. No big deal. Just gimme five minutes”, but then that five minutes turns into a few hours. Art Vandelay can help. 
+Have you ever been on a project where, out of nowhere, someone asks you to send them a CSV of data? You think to yourself, “Ok, cool. No big deal. Just gimme five minutes”, but then that five minutes turns into a few hours. Art Vandelay can help.
 
 **At a high level, here’s what Art Vandelay can do:**
 
 - 🕶 Automatically [filters out sensitive information](#%EF%B8%8F-configuration).
 - 🔁 Export data [in batches](#exporting-in-batches).
-- 📧 [Email](#artvandelayexportemail_csv) exported data.
-- 📥 [Import data](#-importing) from a CSV.
+- 📧 [Email](#artvandelayexportemail) exported data.
+- 📥 [Import data](#-importing) from a CSV or JSON file.
 
 ## ✅ Installation
 
@@ -48,6 +48,8 @@ end
 
 ### 📤 Exporting
 
+Art Vandelay supports exporting CSVs and JSON files.
+
 ```ruby
 ArtVandelay::Export.new(records, export_sensitive_data: false, attributes: [], in_batches_of: ArtVandelay.in_batches_of)
 ```
@@ -57,7 +59,7 @@ ArtVandelay::Export.new(records, export_sensitive_data: false, attributes: [], i
 |`records`|An [Active Record Relation](https://api.rubyonrails.org/classes/ActiveRecord/Relation.html) or an instance of an Active Record. E.g. `User.all`, `User.first`, `User.where(...)`, `User.find_by`|
 |`export_sensitive_data`|Export sensitive data. Defaults to `false`. Can be configured with `ArtVandelay.filtered_attributes`.|
 |`attributes`|An array attributes to export. Default to all.|
-|`in_batches_of`|The number of records that will be exported into each CSV. Defaults to 10,000. Can be configured with `ArtVandelay.in_batches_of`|
+|`in_batches_of`|The number of records that will be exported into each file. Defaults to 10,000. Can be configured with `ArtVandelay.in_batches_of`|
 
 #### ArtVandelay::Export#csv
 
@@ -74,6 +76,21 @@ csv = csv_exports.first.to_a
 # => [["id", "email", "password", "created_at", "updated_at"], ["1", "[email protected]", "[FILTERED]", "2022-10-25 09:20:28 UTC", "2022-10-25 09:20:28 UTC"]]
 ```
 
+#### ArtVandelay::Export#json
+
+Returns an instance of `ArtVandelay::Export::Result`.
+
+```ruby
+result = ArtVandelay::Export.new(User.all).json
+# => #<ArtVandelay::Export::Result>
+
+json_exports = result.json_exports
+# => [#<CSV::Table>, #<CSV::Table>, ...]
+
+json = JSON.parse(json_exports.first)
+# => [{"id"=>1, "email"=>"[email protected]", "password"=>"[FILTERED]", "created_at"=>"2022-10-25 09:20:28.123Z", "updated_at"=>"2022-10-25 09:20:28.123Z"}]
+```
+
 ##### Exporting Sensitive Data
 
 ```ruby
@@ -104,12 +121,12 @@ csv_size = result.csv_exports.first.size
 # => 100
 ```
 
-#### ArtVandelay::Export#email_csv
+#### ArtVandelay::Export#email
 
-Emails the recipient(s) CSV exports as attachments.
+Emails the recipient(s) exports as attachments.
 
 ```ruby
-email_csv(to:, from: ArtVandelay.from_address, subject: "#{model_name} export", body: "#{model_name} export")
+email(to:, from: ArtVandelay.from_address, subject: "#{model_name} export", body: "#{model_name} export")
 ```
 
 |Argument|Description|
@@ -118,17 +135,19 @@ email_csv(to:, from: ArtVandelay.from_address, subject: "#{model_name} export",
 |`from`|The email address of the sender.|
 |`subject`|The email subject. Defaults to the following pattern: "User export"|
 |`body`|The email body. Defaults to the following pattern: "User export"|
+|`format`|The format of the export file. Either `:csv` or `:json`.|
 
 ```ruby
 ArtVandelay::Export
  .new(User.where.not(confirmed: nil))
- .email_csv(
+ .email(
  to: ["george@vandelay_industries.com", "kel_varnsen@vandelay_industries.com"],
  from: "noreply@vandelay_industries.com",
  subject: "List of confirmed users",
- body: "Here's an export of all confirmed users in our database."
+ body: "Here's an export of all confirmed users in our database.",
+ format: :json
  )
-# => ActionMailer::Base#mail: processed outbound mail in... 
+# => ActionMailer::Base#mail: processed outbound mail in...
 ```
 
 ### 📥 Importing
@@ -179,38 +198,78 @@ csv(csv_string, **options)
 |`csv_string`|Data in the form of a CSV string.|
 |`**options`|A hash of options. Available options are `headers:` and `attributes:`|
 
-#### Options
+##### Options
 
 |Option|Description|
 |------|-----------|
 |`headers:`|The CSV headers. Use when the supplied CSV string does not have headers.|
 |`attributes:`|The attributes the headers should map to. Useful if the headers do not match the model's attributes.|
 
-##### Rolling back if a record fails to save
+##### Setting headers
 
 ```ruby
 csv_string = CSV.generate do |csv|
- csv << ["email", "password"]
  csv << ["george@vandelay_industries.com", "bosco"]
- csv << ["kel_varnsen@vandelay_industries.com", nil]
 end
 
-result = ArtVandelay::Import.new(:users, rollback: true).csv(csv_string)
-# => rollback transaction
+result = ArtVandelay::Import.new(:users).csv(csv_string, headers: [:email, :password])
+# => #<ArtVandelay::Import::Result>
 ```
 
-##### Setting headers
+#### ArtVandelay::Import#json
+
+Imports records from the supplied JSON. Returns an instance of `ArtVandelay::Import::Result`.
+
+```ruby
+json_string = [
+ {
+ email: "george@vandelay_industries.com",
+ password: "bosco"
+ },
+ {
+ email: "kel_varnsen@vanderlay_industries.com",
+ password: nil
+ }
+].to_json
+
+result = ArtVandelay::Import.new(:users).json(json_string)
+# => #<ArtVandelay::Import::Result>
+
+result.rows_accepted
+# => [{:row=>[{"email"=>"george@vandelay_industries.com", "password"=>"bosco"}], :id=>1}]
+
+result.rows_rejected
+# => [{:row=>[{"email"=>"kel_varnsen@vandelay_industries.com", "password"=>nil}], :errors=>{:password=>["can't be blank"]}}]
+```
+
+```ruby
+json(json_string, **options)
+```
+
+##### Options
+
+|Option|Description|
+|------|-----------|
+|`attributes:`|The attributes the JSON object keys should map to. Useful if the headers do not match the model's attributes.|
+
+#### Rolling back if a record fails to save
+
+`ArtVandelay::Import.new` supports a `:rollback` keyword argument. It imports all rows as a single transaction and does not persist any records if one record fails due to an exception.
 
 ```ruby
 csv_string = CSV.generate do |csv|
+ csv << ["email", "password"]
  csv << ["george@vandelay_industries.com", "bosco"]
+ csv << ["kel_varnsen@vandelay_industries.com", nil]
 end
 
-result = ArtVandelay::Import.new(:users).csv(csv_string, headers: [:email, :password])
-# => #<ArtVandelay::Import::Result>
+result = ArtVandelay::Import.new(:users, rollback: true).csv(csv_string)
+# => rollback transaction
 ```
 
-##### Mapping custom headers
+#### Mapping custom headers
+
+Both `ArtVandelay::Import#csv` and `#json` support an `:attributes` keyword argument. This lets you map fields in the import document to your Active Record model's attributes.
 
 ```ruby
 csv_string = CSV.generate do |csv|
@@ -222,7 +281,9 @@ result = ArtVandelay::Import.new(:users).csv(csv_string, attributes: {email_addr
 # => #<ArtVandelay::Import::Result>
 ```
 
-##### Stripping whitespace
+#### Stripping whitespace
+
+`ArtVandelay::Import.new` supports a `:strip` keyword argument to strip whitespace from values in the import document.
 
 ```ruby
 csv_string = CSV.generate do |csv|

lib/art_vandelay.rb

@@ -16,10 +16,10 @@ class Error < StandardError
 
  class Export
  class Result
- attr_reader :csv_exports
+ attr_reader :exports
 
- def initialize(csv_exports)
- @csv_exports = csv_exports
+ def initialize(exports)
+ @exports = exports
  end
  end
 
@@ -45,19 +45,42 @@ def csv
  Result.new(csv_exports)
  end
 
- def email_csv(to:, from: ArtVandelay.from_address, subject: "#{model_name} export", body: "#{model_name} export")
+ def json
+ json_exports = []
+
+ if records.is_a?(ActiveRecord::Relation)
+ records.in_batches(of: in_batches_of) do |relation|
+ json_exports << relation
+ .map { |record| row(record.attributes, format: :hash) }
+ .to_json
+ end
+ elsif records.is_a?(ActiveRecord::Base)
+ json_exports << [row(records.attributes, format: :hash)].to_json
+ end
+
+ Result.new(json_exports)
+ end
+
+ def email(
+ to:,
+ from: ArtVandelay.from_address,
+ subject: "#{model_name} export",
+ body: "#{model_name} export",
+ format: :csv
+ )
  if from.nil?
  raise ArtVandelay::Error, "missing keyword: :from. Alternatively, set a value on ArtVandelay.from_address"
  end
 
  mailer = ActionMailer::Base.mail(to: to, from: from, subject: subject, body: body)
- csv_exports = csv.csv_exports
+ exports = public_send(format).exports
 
- csv_exports.each.with_index(1) do |csv, index|
- if csv_exports.one?
- mailer.attachments[file_name] = csv
+ exports.each.with_index(1) do |export, index|
+ if exports.one?
+ mailer.attachments[file_name(format: format)] = export
  else
- mailer.attachments[file_name(suffix: "-#{index}")] = csv
+ file = file_name(suffix: "-#{index}", format: format)
+ mailer.attachments[file] = export
  end
  end
 
@@ -70,18 +93,25 @@ def email_csv(to:, from: ArtVandelay.from_address, subject: "#{model_name} expor
 
  def file_name(**options)
  options = options.symbolize_keys
+ format = options[:format]
  suffix = options[:suffix]
  prefix = model_name.downcase
  timestamp = Time.current.in_time_zone("UTC").strftime("%Y-%m-%d-%H-%M-%S-UTC")
 
- "#{prefix}-export-#{timestamp}#{suffix}.csv"
+ "#{prefix}-export-#{timestamp}#{suffix}.#{format}"
  end
 
- def filtered_values(attributes)
- if export_sensitive_data
- ActiveSupport::ParameterFilter.new([]).filter(attributes).values
- else
- ActiveSupport::ParameterFilter.new(ArtVandelay.filtered_attributes).filter(attributes).values
+ def filtered_values(attributes, format:)
+ attributes =
+ if export_sensitive_data
+ ActiveSupport::ParameterFilter.new([]).filter(attributes)
+ else
+ ActiveSupport::ParameterFilter.new(ArtVandelay.filtered_attributes).filter(attributes)
+ end
+
+ case format
+ when :hash then attributes
+ when :array then attributes.values
  end
  end
 
@@ -116,11 +146,11 @@ def model_name
  records.model_name.name
  end
 
- def row(attributes)
+ def row(attributes, format: :array)
  if self.attributes.any?
- filtered_values(attributes.slice(*standardized_attributes))
+ filtered_values(attributes.slice(*standardized_attributes), format:)
  else
- filtered_values(attributes)
+ filtered_values(attributes, format:)
  end
  end
 
@@ -163,6 +193,20 @@ def csv(csv_string, **options)
  end
  end
 
+ def json(json_string, **options)
+ options = options.symbolize_keys
+ attributes = options[:attributes] || {}
+ array = JSON.parse(json_string)
+
+ if rollback
+ active_record.transaction do
+ parse_json_data(array, attributes, raise_on_error: true)
+ end
+ else
+ parse_json_data(array, attributes)
+ end
+ end
+
  private
 
  attr_reader :model_name, :rollback, :strip
@@ -191,6 +235,24 @@ def build_params(row, attributes)
  end
  end
 
+ def parse_json_data(array, attributes, **options)
+ raise_on_error = options[:raise_on_error] || false
+ result = Result.new(rows_accepted: [], rows_rejected: [])
+
+ array.each do |entry|
+ params = build_params(entry, attributes)
+ record = active_record.new(params)
+
+ if raise_on_error ? record.save! : record.save
+ result.rows_accepted << {row: entry, id: record.id}
+ else
+ result.rows_rejected << {row: entry, errors: record.errors.messages}
+ end
+ end
+
+ result
+ end
+
  def parse_rows(rows, attributes, **options)
  options = options.symbolize_keys
  raise_on_error = options[:raise_on_error] || false