Issues with Spoof File Checks

[ericchernuka]

Hi,
I just upgraded to Paperclip 4.0 and now I'm getting an error about spoofed_media_type. I found the helper for:

do_not_validate_attachment_file_type :push_certificate

But I still receive error the error message. Does Paperclip::MediaTypeSpoofDetector.using(adapter, value.original_filename).spoofed? not take into account the do_not_validate?

[jyurek]

No, the spoof detector is not affected by that helper, only the enforced check for content type/filename validations.

What is the filename and type of file that you're uploading? What is the output of file --mime-type for this file?

[ericchernuka]

The filename is "ios_push_notification_certificate.pem" and the type that set on the form uploader is "application/x-x509-ca-cert". When I run file --mime-type it returns ios_push_notification_certificate.pem: text/plain

[jyurek]

Well, that's annoying. The file command completely fails us here. I think letting do_not_validate_attachment_file_type bypass this might be the thing to do. I'll let you know when I add that in. Sorry about the inconvenience.

[ericchernuka]

No problem. I thought I could get around it by registering a new mime-type for the lookup but wasn't able to have that work either.

Thanks for the quickly reply and looking into this. Thought I was going nuts.

[schristm]

I've got the same problem here. I'm wondering if there's a workaround for the time being until there's an update. For now, I'll just roll back to the previous version. Overall though, this looks like a nice release. Looking forward to testing it out more. Thanks for producing this gem.

[glebtv]

This also messes with files assigned with uri-open for me (same error)

[glebtv]

Workaround:

require 'paperclip/media_type_spoof_detector'
module Paperclip
  class MediaTypeSpoofDetector
    def spoofed?
      false
    end
  end
end

[mkarklins]

@glebtv is right.
I just encountered the same issue ( Paperclip 4.1.0 ). ( used the exact same workaround )
Passing a URI as paperclip image creates a temp file with a filename e.g. 'open-uri20140210-81840-d9erdb'. Note that it doesn't have an extenstion - guess paperclip doesn't like that anymore.

[jyurek]

Nope, it doesn't. But that sounds like something we can fix. I assume the URL will have an extension, so we can tack one on to the Tempfile name.

[chrismhilton]

I have come across the same problem attempting to upload a word .doc file which is resulting in the spoofed error message as the file utility returns null for the mime-type. This is on an Ubuntu 12.04.4 LTS (GNU/Linux 3.8.0-29-generic x86_64) server with version is 5.09 of the file utility and its magic.mgc file from 2011.

These are the relevant rails application log entries:

I, [2014-02-11T13:54:01.212664 #2215]  INFO -- :   Parameters: {"utf8"=>"✓", "authenticity_token"=>"*****", "attachment"=>{"attachable_id"=>"133", "attachable_type"=>"Member", "name"=>"doc", "description"=>"", "attachment"=>#<ActionDispatch::Http::UploadedFile:0x007fb27d422840 @tempfile=#<Tempfile:/tmp/RackMultipart20140211-2215-12kq7vg>, @original_filename="CRM Reports.doc", @content_type="application/msword", @headers="Content-Disposition: form-data; name=\"attachment[attachment]\"; filename=\"CRM Reports.doc\"\r\nContent-Type: application/msword\r\n">}, "commit"=>"Create Attachment"}
I, [2014-02-11T13:54:01.251359 #2215]  INFO -- : Command :: file -b --mime-type '/tmp/965e167d5e49be81f12458cd73908c8720140211-2215-qg25m9'
I, [2014-02-11T13:54:01.261300 #2215]  INFO -- : [paperclip] Content Type Spoof: Filename CRM_Reports.doc (["application/msword", "application/word", "application/x-msword", "application/x-word", "text/plain"]), content type discovered from file command: . See documentation to allow this combination.

For reference, I can upload the same file in development on my mac running mavericks with version is 5.04 of the file utility.

[attenzione]

I also have issue with LinkedIn avatars (probably other networks have the same)
i try to set user avatar from LinkedIn auth data.
for example link to my avatar, there is no correct file name, so console output is

Command :: file -b --mime-type '/var/folders/36/x84hhxs12p976ft_2zh9d7fh0000gn/T/5bee0e2ae651d1f03d8279bf1325360920140213-78899-5f5vji'
[paperclip] Content Type Spoof: Filename 0_R84qR_HfdTSmRmE3RkdIRhVfIFU7UDE3JG2wRhdKqT7xkWgTBQH4v8gYLoRK4owDV_VoqbUB66UF ([]), content type discovered from file command: image/jpeg.

and model is not saved

[rossshannon]

The issue also affects Facebook avatars captured from OAuth authentication. The image comes back as part of the JSON response as an URL of the form http://graph.facebook.com/10000000/picture?type=large (note no extension), which then fails validation.

[djcp]

@chrismhilton - can you send me an example of a file that gives a null mime-type? I cannot replicate the condition where file would return nothing. I'm on debian testing, BTW. djcp at thoughtbot dot com

[teeparham]

Here's an attachment that was previously saved from gravatar. It originally had no extension. When I reprocess it, it fails spoof detection:

(note: the trailing dot is part of the url)
https://dcluasumhcbm6.cloudfront.net/images/6635/original139a7916496355bb264ef5c0a15944db.

[paperclip] copying images/6635/original359f43a6098ab57d0627650dc0265bf9. to local file /var/folders/gg/76wnz1s57gl_whn0xqb4hz5m0000gn/T/596a5011c51bd5107971bf1b98dad9eb20140218-4537-bxfcel
[AWS S3 200 0.675437 0 retries] get_object(:bucket_name=>"xxx",:key=>"images/6635/original359f43a6098ab57d0627650dc0265bf9.")  
[paperclip] saving images/6635/medium359f43a6098ab57d0627650dc0265bf9.
[AWS S3 200 1.063706 0 retries] put_object(:acl=>:public_read,:bucket_name=>"xxx",:cache_control=>"public,max-age=31536000",:content_length=>14778,:content_type=>"image/jpeg",:data=>Paperclip::FileAdapter: 7eafe68436d86220a2f116db76bcb31e20140218-4537-17g095b20140218-4537-4dm8jo,:expires=>"Wed, 18 Feb 2015 16:54:32 GMT",:key=>"images/6635/medium359f43a6098ab57d0627650dc0265bf9.")  
Command :: file -b --mime-type '/var/folders/gg/76wnz1s57gl_whn0xqb4hz5m0000gn/T/596a5011c51bd5107971bf1b98dad9eb20140218-4537-bxfcel'
[paperclip] Content Type Spoof: Filename open-uri20121207-25901-1brx8vl ([]), content type discovered from file command: image/jpeg. See documentation to allow this combination.

[agius]

Also hits when there's a difference between production / dev dbs... I have an image attachment for companies which uses different s3 buckets for production vs dev. When I copy the production db to dev, paperclip tries to download the image from the production s3 bucket and doesn't have access. So it fails with the same error message. Frustrating.

[browntiger]

An additional note: on CentOS 5.10 (at least) it seems the file command doesn't have a --mime-type parameter, just --mime, which causes spoof detection to fail.
A workaround is:

module Paperclip
  class MediaTypeSpoofDetector
    def type_from_file_command
      begin
        Paperclip.run("file", "-b --mime :file", :file => @file.path)
      rescue Cocaine::CommandLineError
        ""
      end
    end
  end
end

[will-r]

I can confirm (after @chrismhilton) that file version 5.09 will not give a mime type for word files. I too am using Ubuntu 12.04 so it could be local to that distribution, but for now I have disabled the checks.

It is also a nuisance that the anti-spoof validation runs on every save, as it means an unchanged file has to be pulled down from S3. Could it be applied only when the file has changed? Happy to submit a pr.

[jyurek]

There is a change on master (8b5289b) that only checks for spoofing when the file changes. That should go out relatively soon. @browntiger Could you open another ticket about that, so we can keep separate causes for the same problem contained? Thanks.

[browntiger]

@jyurek, Thanks, I opened ticket #1451 for the problem.

[guyisra]

I confirm spoofed problems with facebook image URI

[lozandier]

Where's the documentation to re-enable setting and changing attachments to an object?

I'm having issues as well with the latest version of Paperclip as far as normal pngs being seen as invalid png files on Windows.

However, it seems more controlled and has to do more of not being able to remove the restriction of assigning an attachment to an object via the command line (for testing purposes, as well as for mocking,stubbing, and extending factories).

[asecondwill]

im getting an error: "Validation failed: Datafile has an extension that does not match its contents" when saving an xml file through a rake task fetching from FTP server. the same file can be attached to the model through active admin. running file sc.xml --mime-type gives application/xml. Is this the same issue? im using ruby 2.0.0 and rails 4.0.2 .

[asecondwill]

@glebtv 's workaround got me rolling for now. (thanks!) and probably confirms its the same issue i guess.

[ays0110]

Sorry for the noobish question, but where do I put this workaround? In my model that has the paperclip attachment? As a helper somewhere?

[asecondwill]

@ays0110 i put it at the top of the rake file that was having problems. that might not be best practice, but there it is. worked for me. might help you. i'd try it a few places and see if it works. then think about where is best for it. for me its a bit hacky, so as near to the need as possible. hth

[rossshannon]

@asecondwill /config/initializers/paperclip.rb is the natural place for it.

[asecondwill]

@rossshannon ah ok, yeah that makes sense. thank you.

[ptduc]

The content not match.. :(
" Avatar has an extension that does not match its contents"
What's up?

[djcp]

@lozandier - Your problem is probably that you don't have the file command available on $PATH, or it doesn't work properly. This should work OK on a POSIX-compatible OS like linux or OSX. Please open up a separate ticket if you can't get this to work after installing file and confirming it works and is correctly on your path.

[dkonayuki]

I still have this 'Content Type Spoof' error with the lastest paperclip patch!

[thebucknerlife]

@kenn's tip above works for me!

[dkonayuki]

I get it working now. I did exactly the opposite of @kenn's tip. It's like this:

user.avatar = open(avatar_url)

[ryankc33]

@dkonayuki that worked for me as well for omniauth with paperclip 4.1.1

url = auth.info.image
user.avatar = open(url)

note: adding :image_size => 'SIZE' as an option for omniauth breaks both methods :/

[justinko]

@chrismhilton I'm on ubuntu 12.04 LTS as well. You need to set the path to the file binary in Cocaine. I added this to config/initializers/cocaine.rb: Cocaine::CommandLine.path = '/usr/bin'

[justinko]

@chrismhilton actually, you should use Paperclip.options[:command_path] = '/usr/bin'

[yashazgar]

Hi
I am getting a similar error
"Attachment translation missing:
en.activerecord.errors.models.submission_detail.attributes.attachment.spoofed_media_type"

When I upload files (word, excel) with Japanese content in it

I tried adding 'do_not_validate_attachment_file_type' but no use.

I do not want to override the paperclip module

Please help me to fix.

[ozvillafan]

If you are using ruby on windows (like me, sadly) then note that there is no "file" command on windows for paperclip to check spoofing. As such, you always get a message like this in your rails server:

[paperclip] Content Type Spoof: Filename upload_test.csv (["text/csv", "text/comma-separated-values"]), content type discovered from file command: . See documentation to allow this combination.

Thankfully, you can add the file command using the correct Gnuwin32 open source package. Make sure you set your environment variables "path" to include the installed bin directory and close all windows terminals to pick up the change (this includes any IDE's, such as RubyMine) then restart your rails server.

[mehulkar]

Getting the same issue with .plist files. file --mime-type returns text/plain and it fails spoof detection

[abrambailey]

Is there a workaround to allow you to use something like:

url = auth.info.image
user.avatar = open(url)

... and specify image_size?

[lozandier]

@ozvillafan I recommend using Ubuntu or another well-received version of
Linux, unless you cannot reliably backup or recover your particular Windows
Parition with a Windows X disc (or if you want to boot by default to
Windows using Window's bootloader). Ubuntu auto-partitioning command often
messes up and it's usually a better idea to do it yourself for the /,
swapfile, and the optional /home partition

On Tue, Sep 2, 2014 at 3:49 AM, ozvillafan [email protected] wrote:

If you are using ruby on windows (like me, sadly) then note that there is
no "file" command on windows for paperclip to check spoofing. As such, you
always get a message like this in your rails server:

[paperclip] Content Type Spoof: Filename upload_test.csv (["text/csv",
"text/comma-separated-values"]), content type discovered from file command:
. See documentation to allow this combination.

Thankfully, you can add the file command using the correct Gnuwin32 open
source package. Make sure you set your environment variables "path" to
include the installed bin directory and close all windows terminals to pick
up the change (this includes any IDE's, such as RubyMine) then restart your
rails server.

—
Reply to this email directly or view it on GitHub
#1429 (comment)
.

Kevin Lozandier
[email protected] [email protected]

[scanales]

@mehulkar were you able to pass the spoof detection? I'm getting errors when uploading images via api as base64 string and then decoding and saving to paperclip. File is image/jpg and detection is application/octet-stream

[mehulkar]

@scanales we basically did a mass override and turned off the feature using @glebtv's workaround since we can trust our users:

require 'paperclip/media_type_spoof_detector'
module Paperclip
  class MediaTypeSpoofDetector
    def spoofed?
      false
    end
  end
end

[Kagetsuki]

Same error, using latest paperclip gem relese, on Ubuntu 14.04. Confirming that workaround from @glebtv does work.

[scanales]

@mehulkar I've tried that but it's not working for me, I'm on CentOS. I get the following error:

Command :: file -b --mime '/tmp/53d9d5a46c76ace548ff62a0385485a420141016-13087-gmpqhm.jpeg'
Command :: identify -format '%wx%h,%[exif:orientation]' '/tmp/53d9d5a46c76ace548ff62a0385485a420141016-13087-gmpqhm.jpeg[0]' 2>/dev/null
[paperclip] An error was received while processing: #<Paperclip::Errors::NotIdentifiedByImageMagickError: Paperclip::Errors::NotIdentifiedByImageMagickError>

This is the code I added to the initializer:

require 'paperclip/media_type_spoof_detector'         

Paperclip::MediaTypeSpoofDetector.class_eval do
    def spoofed?
      false
    end
    def type_from_file_command      
        begin       
          Paperclip.run("file", "-b --mime :file", :file => @file.path)
        rescue Cocaine::CommandLineError
          ""
        end       
    end      
end

[Marthyn]

require 'paperclip/media_type_spoof_detector'

module Paperclip
  class MediaTypeSpoofDetector
    def type_from_file_command
      begin
        Paperclip.run("file", "-b --mime :file", :file => @file.path)
      rescue

      end
    end
  end
end

seems to work when file is uploaded on windows

[mjodko]

As an improvement to @glebtv's workaround you might want to try this:

# config/initializers/paperclip.rb
require 'paperclip/media_type_spoof_detector'

module Paperclip
  class MediaTypeSpoofDetector
    old_spoofed = instance_method(:spoofed?)

    define_method(:spoofed?) do
      if supplied_file_content_types.count > 0
        old_spoofed.bind(self).()
      else
        false
      end
    end
  end
end

[mach-kernel]

@glebtv @mjodko thank you for this. Giggling that this is in production on our servers right now.

[rystraum]

Hopefully this replaces the monkey patching: #2378

[kirushanths]

No need for monkey patch.

Paperclip has content-type checking and media-type checking. Both seem to do the same thing using file -b --mime.

Media-type checking uses the global config of content_type_mappings. However, sometimes you want per-model exceptions. Paperclip provides the option for it but not well documented. I got it from the source here:
https://github.com/thoughtbot/paperclip/blob/v4.3/lib/paperclip/has_attached_file.rb#L83

You can do the following to control media-type and content-type checking:

class PaperclipModel < ActiveRecord::Base

   has_attached_file :attachment, { validate_media_type: false }

  validates_attachment :attachment, {
     # tweak as desired
     content_type: { content_type: ["text/csv", "text/plain", Paperclip::ContentTypeDetector::SENSIBLE_DEFAULT] }
  }
end

[tomfast]

Using the latest version of Paperclip (5.1.0) resolved the issue for me. I'm suspecting

• Add default content_type_detector to UploadedFileAdapter (Default to Paperclip::ContentTypeDetector #2270)

did the trick for me.

Source: https://github.com/thoughtbot/paperclip/blob/master/NEWS for 5.1.0

[dradux]

Standard Alpine Linux (in docker) does not come with the 'file' command so you may want to verify you have it to start with.

• hope its of use to others

[kennabila]

Standard Alpine Linux (in docker) does not come with the 'file' command so you may want to verify you have it to start with.

• hope its of use to others

I got the same problem on Alpine (Docker), the problem solved by adding this on my Dockerfile:
RUN apk add file

Thanks!

[sunil-smartbear]

Workaround:

require 'paperclip/media_type_spoof_detector'
module Paperclip
  class MediaTypeSpoofDetector
    def spoofed?
      false
    end
  end
end

This is the perfect solution!!!
Thanks