AppExchange:


	Blog		Projects & Toolkits		Documents		Discussion Boards		Join ADN

Note: The information in this article applies to .NET Framework 1.1 ONLY.

SOAP messages generated by both the sforce API client and the sforce API service can become very large. This can have adverse performance implications due to increased transmission and reception periods over the network. In the case of mobile devices on slow connections, this issue is magnified. Because increasing client performance capabilities is often much cheaper and easier than increasing network bandwidth, performing pre- and post-processing of SOAP messages to reduce their size is a reasonable solution.

The sforce service fully supports the HTTP 1.1 content encoding specification for use of compression. To date, the .Net framework generated proxy clients to not fully support this at the HTTP transport level. This requires us to use extensions to implement compression/decompression in .Net.

By using compression on the client and the server, the size of large SOAP messages transmitted across the wire can be reduced by up to 90%. This means that a 100K SOAP message can be compressed to 10K. A call that may have spent 10 seconds on the network will spend nearer 2 seconds on the network.

For the majority of SOAP messages that you generate to the sforce service, you will probably not need to use compression. For batch operations in particular, compressing the out-bound message can provide very good performance gains. Requesting that the response be compressed will provide significant gains for queries that return large amounts of data.

Using Compression is contingent on the ability to control the HTTP headers (HTTP protocol @ w3c) that are generated with your SOAP messages. Most web services support some kind of compression and will send compressed data to the client if the client has indicated that it can handle compressed data. The client indicates this to the service by setting an HTTP header called Accept-Encoding. This header tells the service what kind of message encoding the client can accept. You typically see this value set to "gzip". If you use this setting when sending you SOAP request to sforce, the message response will be compressed. If the server does compress the data, it will add an HTTP header called Content-Encoding with a value of "gzip" depending on the Accept-Encoding value sent with the request.

If you have set the Accept-Encoding header on your request, you should examine the HTTP header on the response to verify the encoding. Look for "gzip" in the Content-Encoding http header.

Now that we know how to let the client and the service know whether or not we want and can handle compression, we need to know how and when to compress and decompress the messages. Regardless of what language you are using for your implementation (VB .Net or C#) the process is the same.

When you create the parameters that represent the data being sent to a web service you are creating objects that are typed using your language of choice. Your message parameters are then serialized in to a SOAP message. The SOAP message is XML that can be represented as a string and reliably transmitted over the web. You will want to compress your message after serialization.

Upon receipt of the SOAP response from the service, you will have a stream of compressed data. Your application will attempt to deserialize it and convert it to the appropriate DOM objects that are represented by the XML contained in the response. Before this can happen you need de-compress the data back to XML. You will want to decompress the stream before deserialization.

The illustration below shows the process flow for client and server side compression. GetRequestStream and GetResponseStream are overridable methods on the WebRequest object that the generated SOAP proxy derives from.

The rest of this tech note will address the .Net implementation for compressing SOAP messages using wrappers for the System.Net.WebRequest class, System.Net.WebResponse class and the generated proxy class.

This technote previously espoused using Soap extensions to implement compression. Although that method works, it had some pretty serious drawbacks, the worst being that any changes made to the generated proxy were lost if you had to refresh your WSDL file.

Instead of using the Soap extensions, we will explain a more reliable and stable method that takes advantage of object oriented practices.

Wrapping the Generated Proxy Client

The generated proxy in .Net is an extension of the System.Web.Services.Protocols.SoapHttpClientProtocol class. This class exposes the GetWebRequest method. This method is our opportunity to add compression functionality to the Soap client. To allow refreshing of the generated soap client, we will extend the soap client using a new class called GzipSforceService. In this sample we are using soapGzip as the namespace and have added a web reference to the salesforce.com wsdl and called it sforce. The result is that the generated proxy client has a fully qualified name of soapGzip.sforce.SforceService. The declaration of our wrapper class is shown below:

using System;

namespace soapGzip
{

    class GzipSforceService : sforce.SforceService
    {
  }
}

Instead of declaring our proxy client as

sforce.SforceService binding = new sforce.SforceService();

we will declare the client as

soapGzip.GzipSforceService binding = new soapGzip.GzipSforceService();

This gives us a level of indirection on the generated proxy client that will allow us to replace or refresh the WSDL file without having to rewrite code changes to the generated client.

Adding a couple of properties to our wrapper class will give us control over compression on a request by request basis.

private bool _acceptZippedResponse;
private bool _sendZippedRequest;
// Property to toggle compression on and off, set to true to enable
   sending compressed requests, false to disable.
     public bool SendZippedRequest
        {
            set { _sendZippedRequest = value;}
            get { return _sendZippedRequest;}
        }
        /// Property to toggle compression on and off, set to true to enable
        receiving compressed responses, false to disable.
        public bool AcceptZippedResponse
        {
            set { _acceptZippedResponse = value;}
            get { return _acceptZippedResponse;}
        }

To indicate that we would like to have our next request compressed we would set the property on the binding object

Binding.SendZippedRequest = true;

Binding.AcceptZippedResponse = true;

The default constructor for our wrapper class takes no parameters. This will cause the default behavior of no inbound or outbound compression to occur. For convenience, I have included a second constructor that accepts two boolean values to initialize the compression behavior at instantiation.

public GzipSforceService(bool acceptZippedResponse, bool sendZippedRequest)
        {
            _acceptZippedResponse = acceptZippedResponse;
            _sendZippedRequest = sendZippedRequest;
        }

The final piece of our wrapper class is where we hijack the WebRequest object by overriding the GetWebRequest method and returning our wrapped WebRequest object.

protected override System.Net.WebRequest GetWebRequest(Uri uri)
    {
        return new GzipWebRequest(base.GetWebRequest(uri),
         SendZippedRequest, AcceptZippedResponse);
    }

Our version of the WebRequest constructor takes the two values that indicate whether or not we send or can receive a compressed message in addition to passing the standard WebRequest object obtained from our base class.

The final version of the wrapped generated proxy class is show below.

using System;

namespace soapGzip
{

    class GzipSforceService : sforce.SforceService
    {
        private bool _acceptZippedResponse;
        private bool _sendZippedRequest;
        /// Property to toggle compression on and off,
        set to true to enable
        sending compressed requests, false to disable.
        public bool SendZippedRequest
        {
            set { _sendZippedRequest = value;}
            get { return _sendZippedRequest;}
        }
        /// Property to toggle compression on and off,
        set to true to enable
        receiving compressed responses, false to disable.
        public bool AcceptZippedResponse
        {
            set { _acceptZippedResponse = value;}
            get { return _acceptZippedResponse;}
        }

        /// This constructor will create a request that
        has it's initial compression attributes set to the
        parameter values.public GzipSforceService
        (bool acceptZippedResponse,
        bool sendZippedRequest)
        {
            _acceptZippedResponse = acceptZippedResponse;
            _sendZippedRequest = sendZippedRequest;
        }

        /// This constructor will create a request
        that has compression off by default (can be enabled
         later).
        public GzipSforceService()
        {
            _acceptZippedResponse = false;
            _sendZippedRequest = false;
        }

        protected override System.Net.WebRequest
        GetWebRequest(Uri uri)
        {
            return new GzipWebRequest(base.GetWebRequest(uri),
             SendZippedRequest, AcceptZippedResponse);
        }
    }
}

Wrapping the WebRequest Object

The WebRequest object is what handles the communication between the client and the service. At some point this object obtains a network stream to send to the service. It is at this point that we want to compression the data, just prior to sending over the wire. Before that happens though, we will need to let the wrapped WebRequest know whether or not we want compression to happen.

The following snippet shows the constructors for our wrapped WebRequest. The first constructor just calls the second supply values of false for both inbound and outbound compression. The second takes two boolean parameters that indicate what kind of compression we want (inbound, outbound or both) and the WebRequest object that was obtained by the generated client proxy.

using System;
using System.Net;
using System.IO;

namespace soapGzip
{
    public class GzipWebRequest : WebRequest
    {
        internal const string GZIP = "gzip";
        private bool gzipRequest;
        private WebRequest wr;

        public GzipWebRequest(WebRequest wrappedRequest) : this
        (wrappedRequest, false, true)
        {
        }

        public GzipWebRequest
        (WebRequest wrappedRequest, bool compressRequest,
         bool acceptCompressedResponse)
        {
            wr = wrappedRequest;
            gzipRequest = compressRequest;
            if(this.gzipRequest)
            wr.Headers["Content-Encoding"] = GZIP;
            if(acceptCompressedResponse)
            wr.Headers["Accept-Encoding"] = GZIP;
        }

We have defined a string constant that contains the http header value to use for the encoding headers. These headers indicate that we are sending a compressed message and can accept a compressed message. We are also saving the value of the compressRequest parameter for use later when we deliver the network stream.

When implementing the WebRequest object, there are a number of methods that must be overridden. In these cases we will just pass the values on from the WebRequest that was passed into our constructor.

// most of these just delegate to the contained WebRequest
    public override string Method
    {
        get { return wr.Method; }
        set { wr.Method = value; }
    }

    public override Uri RequestUri
    {
        get { return wr.RequestUri; }
    }

    public override WebHeaderCollection Headers
    {
        get { return wr.Headers; }
        set { wr.Headers = value; }
    }

    public override long ContentLength
    {
        get { return wr.ContentLength; }
        set { wr.ContentLength = value; }
    }

    public override string ContentType
    {
        get { return wr.ContentType; }
        set { wr.ContentType = value; }
    }

    public override ICredentials Credentials
    {
        get { return wr.Credentials; }
        set { wr.Credentials = value; }
    }

    public override bool PreAuthenticate
    {
        get { return wr.PreAuthenticate; }
        set { wr.PreAuthenticate = value; }
    }

Because we are going to intercept the network stream we need a variable to hold our interim compressed stream.

	private Stream request_stream = null;

The next three overrides actually represent the two available options when using the web services client. The generated client actually provides synchronous and asynchronous invocation of the web service methods. The synchronous version looks like the actual name of the web service method. The asynchronous consists of a begin and end method and in most implementations, is not used on the client. The WebRequest object behaves similarly.

    public override System.IO.Stream GetRequestStream()
    {
        return WrappedRequestStream(wr.GetRequestStream());
    }

    public override IAsyncResult BeginGetRequestStream
    (AsyncCallback callback, object state)
    {
        return wr.BeginGetRequestStream (callback, state);
    }

    public override System.IO.Stream EndGetRequestStream
    (IAsyncResult asyncResult)
    {
        return WrappedRequestStream(wr.EndGetRequestStream
        (asyncResult));
    }

When returning the request stream from our wrapped WebRequest, we will first pass it though a function called WrappedRequestStream as shown above. This function will do our compression for us.

    private Stream WrappedRequestStream(Stream requestStream)
    {
        if ( request_stream == null )
        {
            request_stream = requestStream;
            if(this.gzipRequest)
                request_stream =
    new ICSharpCode.SharpZipLib.GZip.GZipOutputStream
    (request_stream);
        }
        return request_stream;
    }

It is in this function that we re-examine the Boolean value saved in the second constructor. Based on this value we will compress the original stream and then return it from our function.

To be able to decompress the response we need to be able to control the WebResponse object that contains the response. This object is provided by the WebRequest objects GetWebResponse method. Below is the overridden GetWebResponse method. It too can be invoked synchronously or asynchronously.

    public override WebResponse GetResponse()
    {
        return new GzipWebResponse(wr.GetResponse ());
    }

    public override IAsyncResult BeginGetResponse
    (AsyncCallback callback, object state)
    {
    return wr.BeginGetResponse (callback, state);
    }

    public override WebResponse EndGetResponse(IAsyncResult asyncResult)
    {
    return new GzipWebResponse(wr.EndGetResponse (asyncResult));
    }

Notice that we are returning an object called GzipWebResponse for the GetResponse function. The GzipWebResponse object will be defined further down.

The final wrapped WebRequest class is shown below.


using System;
using System.Net;
using System.IO;

namespace soapGzip
{
    /// 
    /// This class wraps an existing WebRequest, and will handle
    compressing the request, and decompressing the response as needed.
    ///
    /// To use this with a Soap Client, create a new class that
    derives from the WSDL generated class and override GetWebRequest,
    /// its implementation should simply be
    ///		return new GzipWebRequest(base.GetWebRequest(uri));
    /// or if you want to compress the request message as well as
    the response, do this
    ///     return new GzipWebRequest(base.GetWebRequest(uri),
     true, true);
    ///
    /// Then when using the web service, remember to create instances
     of the derived class, rather than the generated class
    /// 
    public class GzipWebRequest : WebRequest
    {
        internal const string GZIP = "gzip";
        private bool gzipRequest;
        private WebRequest wr;


        /// 
        /// This constructor will send an uncompressed request,
         and indicate that we can accept a compressed response.
        /// You should be able to use this anywhere to get
        automatic support for handling compressed responses

        public GzipWebRequest(WebRequest wrappedRequest) :
        this(wrappedRequest, false, true)
        {
        }

        /// 

        /// This constructor allows to indicate if you want
        to compress the request, and if you want to indicate
         that you can handled a compressed response
        /// 
        /// The WebRequest we're
        wrapping.
        /// if true, we will gzip
        the request message.
        /// if true,
        we will indicate that we can handle a gzip'd response,
        and decode it if we get a gziped response.
        public GzipWebRequest(WebRequest wrappedRequest, bool
        compressRequest, bool acceptCompressedResponse)
        {
            this.wr = wrappedRequest;
            this.gzipRequest = compressRequest;
            if(this.gzipRequest)
                wr.Headers["Content-Encoding"] = GZIP;
            if(acceptCompressedResponse)
                wr.Headers["Accept-Encoding"] = GZIP;
        }

        #region These are straight pass-through delegates
        to the contained webrequest
        // most of these just delegate to the contained
        WebRequest
        public override string Method
        {
            get { return wr.Method; }
            set { wr.Method = value; }
        }

        public override Uri RequestUri
        {
            get { return wr.RequestUri; }
        }

        public override WebHeaderCollection Headers
        {
            get { return wr.Headers; }
            set { wr.Headers = value; }
        }

        public override long ContentLength
        {
            get { return wr.ContentLength; }
            set { wr.ContentLength = value; }
        }

        public override string ContentType
        {
            get { return wr.ContentType; }
            set { wr.ContentType = value; }
        }

        public override ICredentials Credentials
        {
            get { return wr.Credentials; }
            set { wr.Credentials = value; }
        }

        public override bool PreAuthenticate
        {
            get { return wr.PreAuthenticate; }
            set { wr.PreAuthenticate = value; }
        }

        #endregion

        private Stream request_stream = null;

        public override System.IO.Stream GetRequestStream()
        {
            return WrappedRequestStream(wr.GetRequestStream());
        }

        public override IAsyncResult BeginGetRequestStream
        (AsyncCallback callback, object state)
        {
            return wr.BeginGetRequestStream (callback, state);
        }

        public override System.IO.Stream EndGetRequestStream
        (IAsyncResult asyncResult)
        {
            return WrappedRequestStream
            (wr.EndGetRequestStream (asyncResult));
        }

        /// 
        /// helper function that wraps the request
        stream in a GzipOutputStream,
        /// if we're going to be compressing the request
        /// 
        /// 
        /// 
        private Stream WrappedRequestStream(Stream requestStream)
        {
            if ( request_stream == null )
            {
                request_stream = requestStream;
                if(this.gzipRequest)
                    request_stream =
        new ICSharpCode.SharpZipLib.GZip.GZipOutputStream(request_stream);
            }
            return request_stream;
        }

        public override WebResponse GetResponse()
        {
            return new GzipWebResponse(wr.GetResponse ());
        }

        public override IAsyncResult BeginGetResponse
        (AsyncCallback callback, object state)
        {
            return wr.BeginGetResponse (callback, state);
        }

        public override WebResponse EndGetResponse
        (IAsyncResult asyncResult)
        {
            return new GzipWebResponse(wr.EndGetResponse
            (asyncResult));
        }
    }
}

Wrapping the WebResponse Object

If we have successfully request that the service return our SOAP response in a compressed format, then we will need to decompress it prior to allowing .Net to deserialize the data into the generated objects. The process is similar to and simpler than the WebRequest object.

The constructor and variable declaration is show below.

    /// 
    /// This is an implementation of WebResponse
    that delegates to another WebResponse implementation.
    /// It will automatically insert a GzipInputStream
    into the ResponseStream, if the response
    /// indicates that its gzip compressed.
    /// 
    public class GzipWebResponse : WebResponse
    {
        private WebResponse wr;
        private Stream response_stream = null;

        internal GzipWebResponse(WebResponse wrapped)
        {
            this.wr = wrapped;
        }

The constructor takes the response supplied by the generated client proxy. We save this off into a local variable. Like the wrapped web request, we also declare a stream object for handling the network stream.

At some point in the process the generated client proxy will want the data returned from the web service call. To get this the GetResponseStream function is called. It is at this point that we will want to inspect the http headers to determine if the server compressed the response, and if so, decompress the stream into a new stream to pass back from the function.

    public override Stream GetResponseStream()
    {
        if ( response_stream == null )
        {
            response_stream = wr.GetResponseStream();
            if ( string.Compare(Headers["Content-Encoding"],
             "gzip", true) == 0 )
    response_stream =
    new ICSharpCode.SharpZipLib.GZip.GZipInputStream(response_stream);
        }
        return response_stream;
    }

Again, as with the WebRequest object, there are a few methods that MUST be overridden and are shown below.


    // these all delegate to the contained WebResponse
    public override long ContentLength
    {
        get { return wr.ContentLength; }
        set { wr.ContentLength = value; }
    }

    public override string ContentType
    {
        get { return wr.ContentType; }
        set { wr.ContentType = value; }
    }

    public override Uri ResponseUri
    {
        get { return wr.ResponseUri; }
    }

    public override WebHeaderCollection Headers
    {
        get { return wr.Headers; }
    }

The full listing of the wrapped web response is shown below.

using System;
using System.Net;
using System.IO;

namespace soapGzip
{
    /// 
    /// This is an implementation of WebResponse
    that delegates to another WebResponse implementation.
    /// It will automatically insert a GzipInputStream
    into the ResponseStream, if the response
    /// indicates that its gzip compressed.
    /// 
    public class GzipWebResponse : WebResponse
    {
        private WebResponse wr;
        private Stream response_stream = null;

        internal GzipWebResponse(WebResponse wrapped)
        {
            this.wr = wrapped;
        }

        /// 
        /// Wrap the returned stream in a gzip uncompressor
         if needed
        /// 
        /// 
        public override Stream GetResponseStream()
        {
            if ( response_stream == null )
            {
                response_stream = wr.GetResponseStream();
                if
( string.Compare(Headers["Content-Encoding"], "gzip", true) == 0 )
                    response_stream =
new ICSharpCode.SharpZipLib.GZip.GZipInputStream(response_stream);
            }
            return response_stream;
        }

        // these all delegate to the contained WebResponse
        public override long ContentLength
        {
            get { return wr.ContentLength; }
            set { wr.ContentLength = value; }
        }

        public override string ContentType
        {
            get { return wr.ContentType; }
            set { wr.ContentType = value; }
        }

        public override Uri ResponseUri
        {
            get { return wr.ResponseUri; }
        }

        public override WebHeaderCollection Headers
        {
            get { return wr.Headers; }
        }
    }
}

By wrapping the generated proxy in a new class that overrides the GetWebRequest method, we can return an extended version of the WebRequest class in which we handle the compression. This class uses System.Net.WebRequest and System.Net.WebResponse to communicate with a web service.

Summary

While this pattern on the surface may seem complex, especially to those of us who migrated to .Net from good old Visual Basic, it represents a best practice in when working with generated code modules. The ability to regenerate you client proxy is especially valuable when working with the salesforce.com Enterprise WSDL.

Also, you may have noticed a couple of objects in the ICSharpCode namespace. These are reliable compression libraries from our friends at www.icsharpcode.net.

For a complete working sample of this code, download the .Net SOQL Explorer from http://sforce.sourceforge.net.

Privacy Statement \| Security Statement
© Copyright 2000-2007 salesforce.com, inc. Customer Relationship Management (CRM) • All rights reserved Various trademarks held by their respective owners.