Friday, January 4, 2013

[Salesforce / Apex] POST Mutipart/form-data with HttpRequest

17/10/2014: the solution has been improved. Datails at the end of the post.

Grandma says you cannot post a Mutipart/form-data using an HttpRequest in APEX?
Well, if she says this now you can tell her this is no more true!

All comes from a CloudSpokes challenge (here is the link)...at the time of starting the challenge I was absolutely sure I would have ended up the challenge in less than a day: http gets/posts  are not a big problem in APEX...well so it seemed.

To complete the challenge you had to make 4 REST calls (login, book a new upload, upload the file, set permissions): during testing the last step always failed.

This was the first time I jumped in front of this issue.

If you don't want to know what I did, go directly here.
The first thing I noted was that you cannot send a base64 encoded file to a server expecting a binary file...It wans't that obvious to me, because I've never struggled with file encoding.

The first code was something like this:
 
public static HTTPResponse uploadFile(Attachmnet file)
 {
  String boundary = '__boundary__xxx';
  String header = '--'+boundary+'\n';
     + 'Content-Disposition: form-data; name="data"; filename="'+file.name
     +'"\nContent-Type: application/octet-stream\n\n';

  String footer = '\n--'+boundary+'--';
  
  String body = EncodingUtil.base64Encode(file.Body); //encodes the blob into a base64 encoded String
  
  body = header + body + footer;
  
  HttpRequest req = new HttpRequest();
  req.setHeader('Content-Type','multipart/form-data; boundary='+boundary);
  req.setMethod('POST');
  req.setEndpoint('http://posttestserver.com/post.php?dir=what_a_wonderful_post');   //COOL site to test form uploads
  req.setBody(body);
  req.setTimeout(60000);
  req.setHeader('Content-Length',String.valueof(body.length()));
  
  Http http = new Http();
      return http.send(req);
 }

Then I was all "Eureka! An encoded string cannot be understood if the server needs a binary", so the only thing to do is to make a concatenation of header + file.Body.toString() + footer! This works only if the Blob comes from a text file (i.e. TXT, XML or CSV files): in these cases you don't have any problem...but with binary data all you have is the error:

Blob is not a valid UTF-8 string

I had to find another way.


Searching the web for "uploading binary data using apex" I found those bad links:
  • http://success.salesforce.com/ideaView?id=08730000000Kr80AAC
  • http://boards.developerforce.com/t5/Apex-Code-Development/Image-upload-using-multipart-form-data/td-p/243335
  • http://boards.developerforce.com/t5/Apex-Code-Development/sending-a-non-ascii-file-via-Http-POST/td-p/116662

That leaded to the block of comments you can see in the challenge's dashboard.

I didn't give up anyway. I had all data needed to send the request so I knew the solution was out there.

First thing was to understand if there was a way to merge Blobs types: it is not possible in APEX if you don't have the original data (in that case you use String concatenation or List of Integers concatenation, if you have bynary data in form of intergers list).
So I came up with the idea to merge header, body and footer using base64 encoded version, something like this:

String encoded = EncodingUtil.base64Encode(Blob.valueOf(header))+EncodingUtil.base64Encode(file.Body)+EncodingUtil.base64Encode(Blob.valueOf(footer));
 req.setBodyAsBlob(EncodingUtil.base64Decode(encoded));

I found that sometimes it worked (after a bit I understood that that times I was extremely lucky!!).

Debugging and searching the web (see this post for example) I came to know that a base64 encoded String could have padding characters because the base64 encoding is done using chunks of 3 bytes (see Google for details), and if data is not multiple of 3 bytes this padding in needed.

So I decided to remove the trailing "=" from each encoded chunck of the body request and paste them together. But it's not the proper way to play with encoded base64 strings, as removing trailing padding needs a reencoding of the original data.

The idea was to remove in some way, without messing with the encoded strings, all trailing padding "=".

For the header string it was simple, because it was simple text and I could have added some blank spaces to get an encoded string without "=". That's:

 
  String boundary = '__boundary__xxx';
  String header = '--'+boundary+'\n';
     + 'Content-Disposition: form-data; name="data"; filename="'+file.name
     +'"\nContent-Type: application/octet-stream';

  String headerEncoded = EncodingUtil.base64Encode(Blob.valueOf(header+'\n\n'));
  //this ensures no trailing "=" padding
  while(headerEncoded.endsWith('='))
  {
   header+=' ';
   headerEncoded = EncodingUtil.base64Encode(Blob.valueOf(header+'\n\n'));
  }

So in practice I add extra spaces before the "\n\n" ending characters till I have an encoded string without padding.

The Blob file is the main problem. I need the unencoded data to get the needed trailing, so I need a String value of the body: even if with that String how can I change the file to avoid the "=" ? As this data can be anything (form txt files to encoded zips), it is not so simple to add some padding character to avoid the "=" padding (not clear I know)...

If the encoded body doesn't contain any trailing "=", now the problem is over, the sum of the encoded header, body and footer works.

The problem is the last 4 bytes of the encoded body. That is from the 0th byte to the N-4th byte of the file I have no problem, becase it is an encoded version without "=" trailing.

How do I encode those last 4 bytes merging them with the footer?

I discovered that the HttpRequest class has a strange behavior: the setBodyAsBlob() and getBody() are complementary for the use I need. That is the following code doens't throw a "Blob is not a valid UTF-8 string" exception:

   Blob body = file.body;
   HttpRequest tmp = new HttpRequest();
   tmp.setBodyAsBlob(body);
   String bodyString = tmp.getBody();
   System.debug('## Output body:'+bodyString );

The result is a messing sequence of characters. Are they properly encoded? Yes they are, this is a kind of test:

Blob decoded4Bytes = EncodingUtil.base64Decode('AA==');
System.debug('FIRST ENCODING: '+EncodingUtil.base64Encode(decoded4Bytes));
HttpRequest tmp = new HttpRequest();
tmp.setBodyAsBlob(decoded4Bytes);
System.debug('LAST ENCODING: '+EncodingUtil.base64Encode(tmp.getBodyAsBlob()));

Using different kind of random encoded data (other that "AA==") the results of encoding, blobbing, httpRequesting (??!!), is always the same.
This is what i needed:

  1. decode the last 4 bytes in blob
  2. append it into an HttpRequest using the "setBodyAsBlob()"
  3. get the body as string with "getBody()"
  4. merge this string with the footer
  5. base64 encode the resulting string
  6. merge the base64 encoding of header, file body (from 0 to N-4th byte), previous merged string
  7. base64 unencoding the resulting string
  8. here you are the Blob you needed!
This is the resulting code:

public static HTTPResponse uploadFile(Attachmnet file)
{
  String boundary = '__boundary__xxx';
  String header = '--'+boundary+'\n';
  body += 'Content-Disposition: form-data; name="data"; filename="'+file.name
    +'"\nContent-Type: application/octet-stream';

  String footer = '\n--'+boundary+'--';
  
  // no trailing padding on header by adding ' ' before the last "\n\n" characters
  String headerEncoded = EncodingUtil.base64Encode(Blob.valueOf(header+'\n\n'));
  //this ensures no trailing "=" padding
  while(headerEncoded.endsWith('='))
  {
   header+=' ';
   headerEncoded = EncodingUtil.base64Encode(Blob.valueOf(header+'\n\n'));
  }
  //base64 encoded body
  String bodyEncoded = EncodingUtil.base64Encode(file.body);
  //base64 encoded footer
  String footerEncoded = EncodingUtil.base64Encode(Blob.valueOf(footer));
  
  Blob bodyBlob = null;
  //last encoded body bytes
  String last4Bytes = bodyEncoded.substring(bodyEncoded.length()-4,bodyEncoded.length());
  //if the last 4 bytes encoded base64 ends with the padding character (= or ==) then re-encode those bytes with the footer
  //to ensure the padding is added only at the end of the body
  if(last4Bytes.endsWith('='))
  {
   Blob decoded4Bytes = EncodingUtil.base64Decode(last4Bytes);
   HttpRequest tmp = new HttpRequest();
   tmp.setBodyAsBlob(decoded4Bytes);
   String last4BytesFooter = tmp.getBody()+footer;   
   bodyBlob = EncodingUtil.base64Decode(headerEncoded+bodyEncoded.substring(0,bodyEncoded.length()-4)+EncodingUtil.base64Encode(Blob.valueOf(last4BytesFooter)));
  }
  else
  {
   bodyBlob = EncodingUtil.base64Decode(headerEncoded+bodyEncoded+footerEncoded);
  }
  
  if(bodyBlob.size()>3000000)
  { 
   //this a "public class CustomException extends Exception{}"
   throw new CustomException('File size limit is 3 MBytes');
  }
  
  HttpRequest req = new HttpRequest();
  req.setHeader('Content-Type','multipart/form-data; boundary='+boundary);
  req.setMethod('POST');
  req.setEndpoint('http://posttestserver.com/post.php?dir=watchdox');  
  req.setBodyAsBlob(bodyBlob);
  req.setTimeout(60000);
  req.setHeader('Content-Length',String.valueof(req.getBodyAsBlob().size()));
  Http http = new Http();
  HTTPResponse res = http.send(req);
  return res;
}

I tested it with different kind of files, dimensions and it always worked. I'd like to know your thoughts.
See ya!

UPDATE

See this improvement to my solution. I'll add the content right here:
public static void uploadFile(Blob file_body, String file_name, String reqEndPoint){
      // Repost of code  with fix for file corruption issue
      // Orignal code postings and explanations
      // http://enreeco.blogspot.in/2013/01/salesforce-apex-post-mutipartform-data.html
      // http://salesforce.stackexchange.com/questions/24108/post-multipart-without-base64-encoding-the-body
      // Additional changes commented GW: that fix issue with occasional corruption of files
      String boundary = '----------------------------741e90d31eff';
      String header = '--'+boundary+'\nContent-Disposition: form-data; name="file"; filename="'+file_name+'";\nContent-Type: application/octet-stream';
      // GW: Do not prepend footer with \r\n, you'll see why in a moment
      // String footer = '\r\n--'+boundary+'--'; 
      String footer = '--'+boundary+'--';             
      String headerEncoded = EncodingUtil.base64Encode(Blob.valueOf(header+'\r\n\r\n'));
      while(headerEncoded.endsWith('='))
      {
       header+=' ';
       headerEncoded = EncodingUtil.base64Encode(Blob.valueOf(header+'\r\n\r\n'));
      }
      String bodyEncoded = EncodingUtil.base64Encode(file_body);
      // GW: Do not encode footer yet
      // String footerEncoded = EncodingUtil.base64Encode(Blob.valueOf(footer));

      Blob bodyBlob = null;
      String last4Bytes = bodyEncoded.substring(bodyEncoded.length()-4,bodyEncoded.length());

      // GW: Replacing this entire section
      /*
      if(last4Bytes.endsWith('='))
      {
           Blob decoded4Bytes = EncodingUtil.base64Decode(last4Bytes);
           HttpRequest tmp = new HttpRequest();
           tmp.setBodyAsBlob(decoded4Bytes);
           String last4BytesFooter = tmp.getBody()+footer;   
           bodyBlob = EncodingUtil.base64Decode(headerEncoded+bodyEncoded.substring(0,bodyEncoded.length()-4)+EncodingUtil.base64Encode(Blob.valueOf(last4BytesFooter)));
      }
      else
      {
            bodyBlob = EncodingUtil.base64Decode(headerEncoded+bodyEncoded+footerEncoded);
      }
      */
     // GW: replacement section to get rid of padding without corrupting data
     if(last4Bytes.endsWith('==')) {
        // The '==' sequence indicates that the last group contained only one 8 bit byte
        // 8 digit binary representation of CR is 00001101
        // 8 digit binary representation of LF is 00001010
        // Stitch them together and then from the right split them into 6 bit chunks
        // 0000110100001010 becomes 0000 110100 001010
        // Note the first 4 bits 0000 are identical to the padding used to encode the
        // second original 6 bit chunk, this is handy it means we can hard code the response in
        // The decimal values of 110100 001010 are 52 10
        // The base64 mapping values of 52 10 are 0 K
        // See http://en.wikipedia.org/wiki/Base64 for base64 mapping table
        // Therefore, we replace == with 0K
        // Note: if using \n\n instead of \r\n replace == with 'oK'
        last4Bytes = last4Bytes.substring(0,2) + '0K';
        bodyEncoded = bodyEncoded.substring(0,bodyEncoded.length()-4) + last4Bytes;
        // We have appended the \r\n to the Blob, so leave footer as it is.
        String footerEncoded = EncodingUtil.base64Encode(Blob.valueOf(footer));
        bodyBlob = EncodingUtil.base64Decode(headerEncoded+bodyEncoded+footerEncoded);
      } else if(last4Bytes.endsWith('=')) {
        // '=' indicates that encoded data already contained two out of 3x 8 bit bytes
        // We replace final 8 bit byte with a CR e.g. \r
        // 8 digit binary representation of CR is 00001101
        // Ignore the first 2 bits of 00 001101 they have already been used up as padding
        // for the existing data.
        // The Decimal value of 001101 is 13
        // The base64 value of 13 is N
        // Therefore, we replace = with N
        // Note: if using \n instead of \r replace = with 'K'
        last4Bytes = last4Bytes.substring(0,3) + 'N';
        bodyEncoded = bodyEncoded.substring(0,bodyEncoded.length()-4) + last4Bytes;
        // We have appended the CR e.g. \r, still need to prepend the line feed to the footer
        footer = '\n' + footer;
        String footerEncoded = EncodingUtil.base64Encode(Blob.valueOf(footer));
        bodyBlob = EncodingUtil.base64Decode(headerEncoded+bodyEncoded+footerEncoded);              
      } else {
        // Prepend the CR LF to the footer
        footer = '\r\n' + footer;
        String footerEncoded = EncodingUtil.base64Encode(Blob.valueOf(footer));
        bodyBlob = EncodingUtil.base64Decode(headerEncoded+bodyEncoded+footerEncoded);  
      }

      HttpRequest req = new HttpRequest();
      req.setHeader('Content-Type','multipart/form-data; boundary='+boundary);
      req.setMethod('POST');
      req.setEndpoint(reqEndPoint);
      req.setBodyAsBlob(bodyBlob);
      req.setTimeout(120000);

      Http http = new Http();
      HTTPResponse res = http.send(req);
}   

37 comments:

  1. Nice work. You did what what many thought to be impossible. Very impressive.

    ReplyDelete
  2. Great job, yes sir! It is a breakthrough. Will have been hard work.

    ReplyDelete
  3. Hi Guys thanks, I abused of my mind doing this :)

    ReplyDelete
  4. Hi,
    I am currently working on similar task. I have one small question regarding the final code that is pasted above.

    body+='Content-Disposition: ....

    where you have used this body variable in your code. i.e Do I need to change the line no.19 like the blow.

    String bodyEncoded = body+EncodingUtil.base64Encode(file.body);

    Please advice.

    Thanks,
    Indy.

    ReplyDelete
  5. Hi Indy, it's a typo.
    Line 5 is "header += ..." and not "body += ..." (actually you can concatenate the string in line 5 with the string in line 4).
    The code has been modified before posting to the blog, that's why the typo.
    Hope this can help.
    Enrico

    ReplyDelete
  6. Hi I am trying to use the same code for BOX.com upload and getting Bad Request error, Any thought why it not working for box.com.

    ReplyDelete
  7. You should check the debug and try to figure out why the sending body is not in the expected format of what Box.com expectes...no more hints sorry!

    ReplyDelete
  8. This comment has been removed by the author.

    ReplyDelete
    Replies
    1. It work fine with DOC,PDF,PNG but not with DOCX from Mac,

      I am not able to understand why encoding not working for DOCX, Any luck why its happen with these file format.

      Can you help me to solve this for DOCX ?

      Delete
  9. This comment has been removed by the author.

    ReplyDelete
    Replies
    1. by Using the debug log Encoding and Decoding is fine.

      Delete
  10. After week spend no 100% solution GOT any where. one space header+=' '; corrupting the image file with extra space in binary.

    Validated in http://www.base64decode.org/. Seems need to look more work around with this code.


    ReplyDelete
  11. I am having prolem:
    Failed to parse multipart/form-data content: Stream ended unexpectedly

    Can anybody help me?

    ReplyDelete
  12. This solution does not work and will not work. Take any file and open in any binary viewer. Checkout the last one byte or two bytes of the file. If the last byte is non-UTF character e.g 'DF', the above solution will miserably fail. Reason being, when you do tmp.setBodyAsBlob(decoded4Bytes) and then tmp.getBody(), no doubt it doesnt raise UTF8 exception, but what it does is, it will replace any unsupported UTF character with combination of three bytes, something like EF,.GH FD, and this changes the content of your binary file.

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
  13. Another thing, had this tmp.setBodyAsBlob(decoded4Bytes) and then tmp.getBody() trick worked, you wouldn't have required to do all these things, like taking last 4 encoded bytes, decoding and then using them. You could have simply set the entire File's data using setBodyAsBlob and retrieve it using getBody, which wont raise.

    In all I would say, good try but certainly doesn't work with every fail. This also explains above queries like why this solution didnt work for DOCX files, because DOCX files are zip file based formats and gets corrupted when their content changes. Your algorithm would be changing the last bytes to 'EF GH FD'

    ReplyDelete
  14. Are there any alternative way how to send HTTP post multipart/form-data contentType request???

    ReplyDelete
    Replies
    1. Currently there doesn't exist a way to do that. Above solution can work for a small subset of files but not all.

      Delete
  15. This is example for java. But i use upload files throw web form with javascript. Anybody know, how send filenames to restful APEX services?

    ReplyDelete
  16. Hi I am use the same code for BOX.com upload file and getting StatusCode=400 and Bad Request error and saw in debug log file format is "--1376307429601
    Content-Disposition: form-data; name="fileName" ; filename="@fileName.txt ;parent_id=788878787"
    Content-Type:text/plain
    Test box
    ----1376307429601-- "
    Please help where i done the mistake and some hits

    ReplyDelete
  17. Hi Onkar

    were u able to upload docx file on box ?

    Anyone else worked on Salesforce-box integration?

    ReplyDelete
  18. Hi @enreeco, @gagan and @onkar.

    I was also having the same requirement and used the above code. However as mentioned in few comments above the code works sometimes and not all the times. Th reason is that when we convert the last 4 bytes of encoded part to string using the HttpRequest class, it corrupts the data. The data length of the returned data is not the same and so the complete image data gets corrupted which is though successfully accepted as a valid request by Salesforce but rejected while processing the same.

    Did you get any luck for any developments on this after that?

    Kamal

    ReplyDelete
  19. Hi Enrico Murru,

    i am facing some problem while sending the multipart/form-data request through apex. i have gone through this blog but i cannot figure out why my code is failing. i want to send two file in the request but somehow second parameter i.e. second file is not getting sent.

    https://salesforce.stackexchange.com/questions/251125/multipart-form-data-no-mapping-found-for-http-request-for-uri

    can you please help me out in this?

    ReplyDelete
  20. This comment has been removed by the author.

    ReplyDelete
  21. I use this code to send a file from Salesforce to M-files, it worked well as long as the file size is less than 12000000, I know that this is a Salesforce Limit, do you know if there is any workaround.

    Thanks

    ReplyDelete
    Replies
    1. Hi gosfa, refer tp https://blog.enree.co/2013/01/salesforce-apex-post-mutipartform-data.html, the new blog's house :)
      Unfortunately there is no way to workaround this limit

      Delete
  22. Hi I have similar challenge , I am sending file from salesforce to external server where text format seems fine , other than text all type files are corrupted.

    ReplyDelete
  23. Can SomeOne Please help me with the code. how to upload a file from local to the field of the salesforce custom object."In Lightning".

    ReplyDelete
    Replies
    1. You can refer this
      http://sfdcmonkey.com/2017/09/25/file-upload-lightning-component/

      Delete
  24. This comment has been removed by the author.

    ReplyDelete
  25. Hi,

    I am trying to upload a text file (also tried PDF, etc) to Salesforce.
    Text file contains 'hello world'.

    I am opening it in binary mode:

    with open(filen, "rb") as f:
    bodyEncoded = base64.b64encode(f.read())

    Then after all the code you have I am composing multi-part message for SalesForce:

    ------------------------------741e90d31eff
    Content-Disposition: form-data; name="entity_content";
    Content-Type: application/json;

    {"ContentDocumentId": "0699E000000lKbLQAU", "ReasonForChange": "Large file upload", "PathOnClient": "hello_world.txt"}

    ------------------------------741e90d31eff
    Content-Disposition: form-data; name="VersionData"; filename="hello_world.txt";
    Content-Type: application/octet-stream

    b'aGVsbG8gd29ybGQK'
    ------------------------------741e90d31eff--

    After uploading it, when I open uploaded file I just see its content as:
    b'aGVsbG8gd29ybGQK'

    I have tried also a pdf, but again, original file and uploaded-downloaded file md5 checksums are different.

    Could you help to understand what I might be doing wrong?

    ReplyDelete
  26. Hi I am using the same to send file to upload as transient document in adobe echosign. But I am getting bad request error.

    ReplyDelete
  27. Hi. Your code seems to work but Im getting the following error in my backend
    java.lang.IllegalArgumentException: Invalid content disposition format
    at org.springframework.http.ContentDisposition.parse(ContentDisposition.java:327) ~[spring-web-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]

    My backend works fine if I use Postman to upload the file it works. But when I use your class it does not. Could you help to identify what is the problem?

    ReplyDelete
  28. Hello I used your code to send a PDF via callout to a company that parses out the fields. It sends the PDF but sometimes it turns it green and most of the file is all messed up. The lines are everywhere and most of it is unreadable. Some of it is. Can anyone please help me fix this??? No idea why it's going it.

    ReplyDelete
  29. Hello, the code for http requests works for me but it fails for https could you help me?

    ReplyDelete