Web fundamentals, Part 2: HTTP Basics

Published by atollkuci on October 10, 2008 under .NET, Software Development, Web Fundamentals

As defined in the “HTTP RFC“, HTTP (Hypertext Transfer Protocol) is “an application-level protocol for distributed, collaborative, hypermedia information systems. It is a generic, stateless, object-oriented protocol which can be used for any tasks, such as name servers and distributed object management systems, through extension of its request methods. A feature of HTTP is the typing and negotiation of data representation, allowing systems to be built independently of the data being transferred.” That’s the long official definition, while the practical most important aspect is that it is a stateless protocol. What this means is that each request/response is totally independent from any other request. Basically we always have a server, a client and the HTTP protocol that transmit the data between them. Since the protocol is stateless if we want to store state we need to pass it with each request/response. In order to understand how this can be done we need to look in more details of how the HTTP protocol works, but to give you an idea, we have three options:

Use cookies - That’s how the magic of “session” works in some platforms.
Use URL parameters (GET request) - An alternative way to support “session”.
Use form hidden fields - That’s how the ASP.NET web forms and ViewState concept work.

Each HTTP request/response is compromised of two parts: the header and the body. While the header is required, the body is optional.They are separated by an empty line.
Now let’s have a more closer look on how a HTTP request/response looks like.

HTTP Request

One of the best ways to understand the HTTP protocol is to use a tool that captures HTTP request/response headers. If you’re using Firefox, there’s an add-on called Live HTTP Headers that does a nice job. A request for this blog results in the following HTTP request header:

GET / HTTP/1.1Host: blog.tollkuci.comUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3 (.NET CLR 3.5.30729)Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8Accept-Language: en-us,sq;q=0.7,en;q=0.3Accept-Encoding: gzip,deflateAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7Keep-Alive: 300Connection: keep-aliveCookie: __utma=192489849.890738818.1223636002.1223636002.1223636002.1; __utmb=192489849;X-lori-time-1: 1223637168885If-Modified-Since: Fri, 10 Oct 2008 11:09:44 GMT

As you can see a HTTP request header is just a list of name/value pairs separated but new line. While the HTTP header has a big list of possible fields, a typical request such as the one above contains the most useful ones and a good understanding of these fields it’s more that enough for most practical reasons. Now let’s analyze each field of the above request in detail:

The first line of the HTTP header indicates the type of the request (in this case GET), the path of the request (in this case /, meaning the default page) and the version of the protocol to be used (in this case HTTP 1.1).
The Host field indicates the server when the request should be made.
User-Agent: Informative fields with details about the user agent (that is the browser) that is making the request. In this case it includes details about browser (Firefox), Windows and .NET. The server receiving the request may make decisions based on this field, for example render different versions for different browsers.
Accept: Specify the media type that the response should have. In this case it is specified that text/html and application/xhtml+xml are the preferred response types. However if that’s not possible application/xml can be used and if that’s not possible any media type can be returned. The order above is specified by the quality parameter (q). Since text/html and application/xhtml+xml do not have a quality parameter it default to 1.
Accept-Encoding: Similar to Accept, but it restricts the encoding of the response. In this case it means that the browser can accept gzip and deflate encodings. If the server also supports gzip or deflate encodings it can compress the response before sending it to the browser, thus saving bandwidth.
Accept-Charset: Again, similar to Accept, but it defines the characters sets that are allowed for the response. In this case “ISO-8859-1″ is the preferred character set, then UTF-8 and then any other character set.
Keep-Alive and Connection specify underlying TCP connection details which are mostly irrelevant. Basically they are used to specify if the underlying connection should be closed for each request/response or should be alive for a specified interval.
Cookie: This is one of the most important fields. It include the list of valid cookies that the user agent (browser) has for this domain. Cookies are at the heart of higher-level concepts such as “Session”. The basic idea is that the browser keeps track of valid cookies for a domain and send them with each request. I’ll talk with much more details for cookies in a future article.
X-lori-time-1: This is a custom header sent by one of the add-ons I use in Firefox. I included it here to show that it is possible to include customer headers in a HTTP request.
If-Modified-Since: It is used for caching purposes and tells the server to not send the data, but to send a special response (304, Not Modified) if the resource is not modified since the specified date. In this case the user agent (browser) will use the cached copy of the resource.

If the above case, since the request is a GET request there’s no message body. In a POST request, the request body will include the name and values of the fields in the form.

HTTP Response

HTTP response also includes a header and a body. The response header for the above request is as below:

HTTP/1.x 200 OKCache-Control: no-cache, must-revalidate, max-age=0Connection: closeDate: Fri, 10 Oct 2008 11:12:50 GMTPragma: no-cacheContent-Type: text/html; charset=UTF-8Content-Encoding: gzipExpires: Wed, 11 Jan 1984 05:00:00 GMTLast-Modified: Fri, 10 Oct 2008 11:12:49 GMTVary: Accept-EncodingServer: WWW Server/1.1X-Powered-By: ASP.NET, PHP/5.2.0X-Pingback: http://blog.tollkuci.com/xmlrpc.phpStatus: 200 OK

Let’s examine each field in detail:

The first line include details of the protocol used and the status of the response (200 OK means that the request is fulfilled successfully).
Cache-Control: Used to define the behavior of caching for the returned resource. In this case, no-cache means that the resource should not be cached anywhere, that the client must re-validate the request and that max-age of caching is 0. End result of all this is that the page should not be cached. There are several other options that can be used to fine tune caching. We’ll have the chance to talk more of caching in a future article.
Connection: As explained for the Request header specify details for the underlying TCP connection.
Date: Date and time when the request was processed.
Pragma: Generic field that can be used for specific implementations. In this case it specifies that the page should not be cached.
Content-Type: Specified the media type of the response body.
Content-Encoding: Encoding of the response body. Since the browser supported gzip encoding and the server can generate gzip response it has compressed the response using gzip.
Expires: Date and time when the resource should expire from the cache. In this case a past date is used to indicate that the page should not be cached.
Last-Modified: Date and time when the request was last modified.
Vary: Specify the mechanism that was used to generate the response. In this case it means that Accept-Encoding was used to generate the response as gzip.
Server: Details about the server that is sending the response.
X-Powered-By: Customer header used the return details about the server that is sending the response.
X-Pingback: Again customer header used to specify the pingback url.
Status: Must important field that includes the response status code and name. 200 OK means that the request was fulfilled successfully. Other common response statuses are:

301 Moved Permanently - indicates that the resource was moved permanently to a new URL.
302 Moved Temporarily - indicates that the resources was moved temporarily to a new URL.
401 Unauthorized - indicates that the request requires user authentication.
403 Forbidden - indicates that the credentials specified with the request are not valid and the request is forbidden.
404 Not Found - the most well know status. Indicates that the specified resource cannot be found.
500 Internal Server Error - the server is unable to fulfill the request.

For a list of all header fields and response codes, you can always refer to the HTTP RFC.

Conclusions

This was just an overview of the basics of the HTTP protocol. There are lots of resource on the web that go down on details for each of the above mentioned topics, by my idea was to give you only the most important concepts. The most important thing to remember from this article is the stateless nature of HTTP and that each request/response is independent from any other. Since in most application it is necessary to store the state somehow, mechanism like url parameters, cookies or hidden form fields are used to maintain this state. We’ll look at them in more details in this series. As a final note, use a tool (such as the Live HTTP Headers add-on) to experiment with HTTP headers. Also don’t be afraid to “play” with the HTTP protocol by just using telnet to connect with any host (telnet tollkuci.com 80) and to start typing the request header manually.

Rescheduling…

Published by atollkuci on September 21, 2008 under .NET, Software Development, mISV

As it happens often in software development, I had to reschedule the release date of the “Website Analyzer”. A combination of several factors made it impossible to release the first public beta in October 1st. Instead I’m planning to have a private beta in October 1st and the first public beta in November 1st. However during this time the product has taken shape. I have finished the parsing and crawling modules, so given a starting point it is able to crawl all the other resources based on the constraints specified by the user. Also the module that parse the HTML and create a simple DOM is finished. However still there’s a lot to be done, which includes:

Developing some of the plugins planned for the beta release. The list will include at least broken links, missing titles, meta tags, page rank, alexa rank, gzip compression and page size information.
Developing the website. I have decided to use the ASP.NET MVC framework for the website combined with some AJAX stuff but I’m still looking for a nice design.
Improve unit testing and code coverage of the product.
All the marketing stuff. There are many questions mark here, but I hope they will have soon an answer.

Also, I’m planning to write some articles for the basis of HTTP, HTML, CSS and JavaScript. There are so many frameworks out there now that abstract away the low level stuff, but in the end to build a website you need a good understanding of the above concepts. I have the feeling that few developers have a good understanding of these concepts and a better understanding would help them a lot.

Microsoft Summit in Tirana

Published by atollkuci on January 29, 2008 under .NET, Software Development, mISV

Today, I participated in the first Microsoft Summit held in Tirana, representing AESS Group together with two colleagues. As I have mentioned before, I’m co-founder and Chief Technology Officer of AESS Group, which is one of the best software companies in Albania. It was founded in summer 2005 and was the first Albanian company to achieve the Microsoft Certified Partner status in the country. All our developers have the Microsoft Certified Professional status, with two of them being Microsoft Certified Application Developers and me Microsoft Certified Solution Developer.

As this was the first Microsoft Summit here, it was mainly a general overview of what Microsoft is doing and focused on licensing Microsoft products. However we hope that the increased presence of Microsoft in the country will bring benefits to us as a software company because people “will get used” to pay for licensing software which currently is very difficult.

I got several pictures, but since I forgot my digital camera I used the phone and as you can see the quality is poor (sorry for that). I choose the best two to put them in the site.

Microsoft Summit - Presentation

Picture taken during the presentation.

Microsoft Summit - Coffee break

Picture taken during the coffee break between the sessions.

Outlook vs Lookout

Published by atollkuci on January 17, 2008 under .NET, Software Development, mISV

It’s been quite a lot of time since my last post, but I’ve been very busy these days. I’m planning to write at least one article each week, mostly related with software development & mISVs, but don’t know if I’ll have enough time to do it. At least I’ll try…
Anyway, I’m writing this article on my Dell Inspiron 9400 laptop running Vista Ultimate x64. I generally love Vista and one of the things I like more is it’s build it search functionality. I’m also running Office 2007 and using Outlook as my email client which also has built-in search capabilities. Searching in Vista is pretty reliable, but unfortunately that’s not the case with outlook. As illustrated in the print screen below, when I search by one the first name of one of my contacts in the All Mail folder, it returns 7 results in about 20 seconds and I know that there are more items.
Outlook print screen

I had heard for a tool called Lookout that works great, but didn’t work with Outlook 2007. A few days before I run across a post that explained how to run Lookout with Outlook 2007 and I decided to try it. After installing it and letting index my emails the same search returned 84 results in only 1.2 seconds. Pretty impressive!

Lookout print screen

I find it very hard to believe that Microsoft cannot make searching in outlook better than an add-in developed by a single person. Until Microsoft fix it, I’ll be happy with lookout

P.S For some reasons lookout currently crashes Outlook when it tries to index new emails. So I disabled automatic indexing of new emails and will manually run the indexing procedure.

Sorting KeyedCollection<K, T>

Published by atollkuci on October 24, 2007 under .NET

Today I had to sort a keyed collection. I have a EntityCollection<T> generic class that inherits from KeyedCollection<PrimaryKey, T> where PrimaryKey is a class that encapsulate a database primary key. At first I checked the .NET class library to see if there’s anything available. It turns out that there’s no sorting capability available for the KeyedCollection<K, T> class unlike the List<T> class which has a Sort() method.
After googling around I found out that the important thing is to use the base class Items collection for the sort by casting it from IList to a List<T> which does support sorting. So I added a Sort method to my EntityCollection<T> like this:
/// <summary> /// Sort the collection using the specified criteria. /// </summary> /// <param name="sortableItems">Items to use for sorting.</param> public void Sort(params SortingCriteria[] sortableItems) { List<T> list = base.Items as List<T>; if (list != null) { ObjectComparer<T> comparer = new ObjectComparer<T>(sortableItems); list.Sort(comparer); } }
After casting the Items collection I create an instance of the ObjectComparer class that inherits from Comparer and is used to compare two items. It looks like this:
class ObjectComparer<K> : Comparer<K> { List<SortingCriteria> sortableItems;


            public ObjectComparer(params SortingCriteria[] sortableItems)

            {

                this.sortableItems = new List<SortingCriteria>();

                foreach (SortingCriteria item in sortableItems)

                {

                    this.sortableItems.Add(item);

                }

            }
            public override int Compare(K x, K y)

            {

                if (x == null && y == null)

                {

                    return 0;

                }

                else if (x == null)

                {

                    return -1;

                }
                foreach (SortingCriteria item in sortableItems)

                {

                    PropertyInfo propInfo = typeof(K).GetProperty(item.PropertyName);

                    IComparable xComparer = propInfo.GetValue(x, null) as IComparable;

                    object yValue = propInfo.GetValue(y, null);
                    if (xComparer == null)

                    {

                        throw new ArgumentOutOfRangeException("Object does not support IComparable interface");

                    }

int result = xComparer.CompareTo(yValue); if (result < 0) { return (item.Order == SortingOrder.Asceding) ? result : -result; } if (result > 0) { return (item.Order == SortingOrder.Asceding) ? result : -result; } // If equal continue and compare the next property } return 0; } }
SortingCriteria is a simple class that encapsulate the name of the property and the sorting order as below:
/// <summary> /// Represent a sorting criteria. /// </summary> public class SortingCriteria { private string propertyName; private SortingOrder order;


        /// <summary>

        /// Create a new sorting criteria, for the specified property name. Sorting order will be ascending.

        /// </summary>

        /// <param name="propertyName">Name of the property to use for sorting.</param>

        public SortingCriteria(string propertyName)

            : this(propertyName, SortingOrder.Asceding)

        {

        }
        /// <summary>

        /// Create a new sorting criteria, for the specified property name in the specified order.

        /// </summary>

        /// <param name="propertyName">Name of the property to use for sorting.</param>

        /// <param name="order">Sorting order to use.</param>

        public SortingCriteria(string propertyName, SortingOrder order)

        {

            this.propertyName = propertyName;

            this.order = order;

        }
        /// <summary>

        /// Gets the name of the property.

        /// </summary>

        public string PropertyName

        {

            get

            {

                return this.propertyName;

            }

        }

/// <summary> /// Gets the sorting order. /// </summary> public SortingOrder Order { get { return order; } } }
Finally it uses the Sort() method of the List<T> to do the actual sorting. I added three overloads of the sort method to make it easier to use:
/// <summary> /// Sort the collection by the specified property, using the specified order. /// </summary> /// <param name="propertyName">Name of the property to use for sorting.</param> /// <param name="order">Sorting order.</param> public void Sort(string propertyName, SortingOrder order) { Sort(new SortingCriteria(propertyName, order)); }


        /// <summary>

        /// Sort the collection by the specified property, in ascending order.

        /// </summary>

        /// <param name="propertyName">Name of the property to use for sorting.</param>

        public void Sort(string propertyName)

        {

            Sort(propertyName, SortingOrder.Asceding);

        }

/// <summary> /// Sort the collection using the specified properties in ascending order. /// </summary> /// <param name="propertyNames">Name of properties to use for sorting.</param> public void Sort(params string[] propertyNames) { SortingCriteria[] items = new SortingCriteria[propertyNames.Length]; for (int i = 0; i < items.Length; i++) { items[i] = new SortingCriteria(propertyNames[i]); } Sort(items); }
This allows the client to use code like:
EntityCollection<Order> orders = Order.GetAll(); orders.Sort(”CustomerName”);
to sort them by the CustomerName property in ascending order, or:
EntityCollection<Order> orders = Order.GetAll(); orders.Sort(”CustomerName”, SortingOrder.Descending);
to sort them in descending order by the CustomerName property, or:
EntityCollection<Order> orders = Order.GetAll(); orders.Sort(”Country”, “CustomerName”);
to sort them by country and then by CustomerName in ascending order.

Archive for the '.NET' Category

Web fundamentals, Part 2: HTTP Basics

HTTP Request

HTTP Response

Conclusions

Rescheduling…

Microsoft Summit in Tirana

Outlook vs Lookout

Sorting KeyedCollection<K, T>

Search

Archives

Categories

Pages