tollkuci.com

As defined in the “HTTP RFC“, HTTP (Hypertext Transfer Protocol) is “an application-level protocol for distributed, collaborative, hypermedia information systems. It is a generic, stateless, object-oriented protocol which can be used for any tasks, such as name servers and distributed object management systems, through extension of its request methods. A feature of HTTP is the typing and negotiation of data representation, allowing systems to be built independently of the data being transferred.” That’s the long official definition, while the practical most important aspect is that it is a stateless protocol. What this means is that each request/response is totally independent from any other request. Basically we always have a server, a client and the HTTP protocol that transmit the data between them. Since the protocol is stateless if we want to store state we need to pass it with each request/response. In order to understand how this can be done we need to look in more details of how the HTTP protocol works, but to give you an idea, we have three options:

1. Use cookies - That’s how the magic of “session” works in some platforms.
2. Use URL parameters (GET request) - An alternative way to support “session”.
3. Use form hidden fields - That’s how the ASP.NET web forms and ViewState concept work.

Each HTTP request/response is compromised of two parts: the header and the body. While the header is required, the body is optional.They are separated by an empty line.
Now let’s have a more closer look on how a HTTP request/response looks like.

HTTP Request

One of the best ways to understand the HTTP protocol is to use a tool that captures HTTP request/response headers. If you’re using Firefox, there’s an add-on called Live HTTP Headers that does a nice job. A request for this blog results in the following HTTP request header:

GET / HTTP/1.1
Host: blog.tollkuci.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3 (.NET CLR 3.5.30729)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,sq;q=0.7,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Cookie: __utma=192489849.890738818.1223636002.1223636002.1223636002.1; __utmb=192489849;
X-lori-time-1: 1223637168885
If-Modified-Since: Fri, 10 Oct 2008 11:09:44 GMT

As you can see a HTTP request header is just a list of name/value pairs separated but new line. While the HTTP header has a big list of possible fields, a typical request such as the one above contains the most useful ones and a good understanding of these fields it’s more that enough for most practical reasons. Now let’s analyze each field of the above request in detail:

• The first line of the HTTP header indicates the type of the request (in this case GET), the path of the request (in this case /, meaning the default page) and the version of the protocol to be used (in this case HTTP 1.1).
• The Host field indicates the server when the request should be made.
• User-Agent: Informative fields with details about the user agent (that is the browser) that is making the request. In this case it includes details about browser (Firefox), Windows and .NET. The server receiving the request may make decisions based on this field, for example render different versions for different browsers.
• Accept: Specify the media type that the response should have. In this case it is specified that text/html and application/xhtml+xml are the preferred response types. However if that’s not possible application/xml can be used and if that’s not possible any media type can be returned. The order above is specified by the quality parameter (q). Since text/html and application/xhtml+xml do not have a quality parameter it default to 1.
• Accept-Encoding: Similar to Accept, but it restricts the encoding of the response. In this case it means that the browser can accept gzip and deflate encodings. If the server also supports gzip or deflate encodings it can compress the response before sending it to the browser, thus saving bandwidth.
• Accept-Charset: Again, similar to Accept, but it defines the characters sets that are allowed for the response. In this case “ISO-8859-1″ is the preferred character set, then UTF-8 and then any other character set.
• Keep-Alive and Connection specify underlying TCP connection details which are mostly irrelevant. Basically they are used to specify if the underlying connection should be closed for each request/response or should be alive for a specified interval.
• Cookie: This is one of the most important fields. It include the list of valid cookies that the user agent (browser) has for this domain. Cookies are at the heart of higher-level concepts such as “Session”. The basic idea is that the browser keeps track of valid cookies for a domain and send them with each request. I’ll talk with much more details for cookies in a future article.
• X-lori-time-1: This is a custom header sent by one of the add-ons I use in Firefox. I included it here to show that it is possible to include customer headers in a HTTP request.
• If-Modified-Since: It is used for caching purposes and tells the server to not send the data, but to send a special response (304, Not Modified) if the resource is not modified since the specified date. In this case the user agent (browser) will use the cached copy of the resource.

If the above case, since the request is a GET request there’s no message body. In a POST request, the request body will include the name and values of the fields in the form.

HTTP Response

HTTP response also includes a header and a body. The response header for the above request is as below:

HTTP/1.x 200 OK
Cache-Control: no-cache, must-revalidate, max-age=0
Connection: close
Date: Fri, 10 Oct 2008 11:12:50 GMT
Pragma: no-cache
Content-Type: text/html; charset=UTF-8
Content-Encoding: gzip
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Last-Modified: Fri, 10 Oct 2008 11:12:49 GMT
Vary: Accept-Encoding
Server: WWW Server/1.1
X-Powered-By: ASP.NET, PHP/5.2.0
X-Pingback: http://blog.tollkuci.com/xmlrpc.php
Status: 200 OK

Let’s examine each field in detail:

• The first line include details of the protocol used and the status of the response (200 OK means that the request is fulfilled successfully).
• Cache-Control: Used to define the behavior of caching for the returned resource. In this case, no-cache means that the resource should not be cached anywhere, that the client must re-validate the request and that max-age of caching is 0. End result of all this is that the page should not be cached. There are several other options that can be used to fine tune caching. We’ll have the chance to talk more of caching in a future article.
• Connection: As explained for the Request header specify details for the underlying TCP connection.
• Date: Date and time when the request was processed.
• Pragma: Generic field that can be used for specific implementations. In this case it specifies that the page should not be cached.
• Content-Type: Specified the media type of the response body.
• Content-Encoding: Encoding of the response body. Since the browser supported gzip encoding and the server can generate gzip response it has compressed the response using gzip.
• Expires: Date and time when the resource should expire from the cache. In this case a past date is used to indicate that the page should not be cached.
• Last-Modified: Date and time when the request was last modified.
• Vary: Specify the mechanism that was used to generate the response. In this case it means that Accept-Encoding was used to generate the response as gzip.
• Server: Details about the server that is sending the response.
• X-Powered-By: Customer header used the return details about the server that is sending the response.
• X-Pingback: Again customer header used to specify the pingback url.
• Status: Must important field that includes the response status code and name. 200 OK means that the request was fulfilled successfully. Other common response statuses are:
• 301 Moved Permanently - indicates that the resource was moved permanently to a new URL.
• 302 Moved Temporarily - indicates that the resources was moved temporarily to a new URL.
• 401 Unauthorized - indicates that the request requires user authentication.
• 403 Forbidden - indicates that the credentials specified with the request are not valid and the request is forbidden.
• 404 Not Found - the most well know status. Indicates that the specified resource cannot be found.
• 500 Internal Server Error - the server is unable to fulfill the request.

For a list of all header fields and response codes, you can always refer to the HTTP RFC.

Conclusions

This was just an overview of the basics of the HTTP protocol. There are lots of resource on the web that go down on details for each of the above mentioned topics, by my idea was to give you only the most important concepts. The most important thing to remember from this article is the stateless nature of HTTP and that each request/response is independent from any other. Since in most application it is necessary to store the state somehow, mechanism like url parameters, cookies or hidden form fields are used to maintain this state. We’ll look at them in more details in this series. As a final note, use a tool (such as the Live HTTP Headers add-on) to experiment with HTTP headers. Also don’t be afraid to “play” with the HTTP protocol by just using telnet to connect with any host (telnet tollkuci.com 80) and to start typing the request header manually.

Although I had the first contact with the web relatively late (about 8 years) ago, I still remember generating tables is plain ASP by combining server-side JavaScript with HTML code. Web development has matured a lot since then with many new frameworks, patterns, technologies, etc that try to hide all the low-level details and offer high-level constructs. All this is fine as long as it works as expected. However, as you know, that’s not always the case and when it’s broken knowing the low-level stuff helps a lot. I have the feeling that new developers get used to the high-level concepts and are “scared” by the basics of web such as HTTP, HTML, CSS and JavaScript. With little effort to learn this basic concepts, you are in a much better position to understand how all fits together.

The last couple of years, especially in the Microsoft & Java World, the focus has been on hiding the details as much as possible. ASP.NET web forms tried to create a model similar to desktop applications, hiding all the interaction between the client and the server. It worked well for basic scenarios, but in the long term it proved far from an ideal solution. Microsoft realized this and is now adopting the well know MVC pattern in the form of ASP.NET MVC (Check out Scott Guthrie’s blog at http://weblogs.asp.net/Scottgu/ for a lot of useful information on ASP.NET MVC). Also a lot of other open source frameworks from Ruby on Rails to Spring use a similar pattern. What all of them have in common is the push for separating the View (HTML) from the Model and Controller. This has the side effect that developers (designers) need to be familiar with raw HTML and CSS to create nice looking sites. For this and other reasons I decided to write a series of posts dedicated to Web Fundamentals. This will include topics such as:

• The basics of HTTP protocol, HTTP headers, difference between POST & GET, SSL, etc.
• HTML & DOM: It’s all about tags.
• How CSS can be used to style a website.
• Using JavaScript to manipulate the DOM.
• The “magic” of server side session and cookies.
• The new “sexy” technology called AJAX.

The plan is to have one new article each week, time permitting. I hope you’ll enjoy and learn from them…

Check out the first article Web fundamentals, Part 2: HTTP Basics.

As it happens often in software development, I had to reschedule the release date of the “Website Analyzer”. A combination of several factors made it impossible to release the first public beta in October 1st. Instead I’m planning to have a private beta in October 1st and the first public beta in November 1st. However during this time the product has taken shape. I have finished the parsing and crawling modules, so given a starting point it is able to crawl all the other resources based on the constraints specified by the user. Also the module that parse the HTML and create a simple DOM is finished. However still there’s a lot to be done, which includes:

• Developing some of the plugins planned for the beta release. The list will include at least broken links, missing titles, meta tags, page rank, alexa rank, gzip compression and page size information.
• Developing the website. I have decided to use the ASP.NET MVC framework for the website combined with some AJAX stuff but I’m still looking for a nice design.
• Improve unit testing and code coverage of the product.
• All the marketing stuff. There are many questions mark here, but I hope they will have soon an answer.

Also, I’m planning to write some articles for the basis of HTTP, HTML, CSS and JavaScript. There are so many frameworks out there now that abstract away the low level stuff, but in the end to build a website you need a good understanding of the above concepts. I have the feeling that few developers have a good understanding of these concepts and a better understanding would help them a lot.

As you know many things do not go according to the plan. In this case the main thing did not go according to the plan and this is the product idea. As I said in the previous post I had envisioned it as a way to extract useful information (email, phone, fax, etc) from unstructured data (internet, email, etc). It turns out that the majority of the customers for this kind of products are SPAM companies, that is companies that send SPAM emails. Associating the first product of my mISV with SPAM is not such a good idea, so I decided to change the product. This is one of the biggest advantages of being a mISV: you can react quickly to change. No need for long meetings end endless discussions.

But what product I’m going to built? I will stick to the October 1st, 2008 release date so it must have the same attributes as the first one. For this reason I decided to stay in the same area, but a different product. It will be called “Tollkuci Website Analyzer” and a you can guess will analyze a website and offer advice for problems and improvements that can be made. Until now I have decided on a few things:

• It will be a web product, so users can directly check their website, without downloading anything.
• It will not analyze a single page, but the full website while allowing the users to limit the scope.
• It will be free for a limited number of pages, perhaps 100 pages.
• For big websites the result can take a considerable amount of time. So the users will submit the site and receive an email when the analysis is finished.

I’m still in the process of defining the actual features it will have, but for a overall idea they fall into these categories:

• General - broken links, missing titles, html & css standards, etc
• SEO - meta keywords, meta description, Google page rank, use of keywords in page content, etc.
• Performance - gzip compression, number of images, total page size, time to first byte, time to load, etc.
• Security - NoBot protection for forms, possibility of SQL injection attacks, etc
• ASP.NET specific - usage of output cashing, size of view state, etc.

Of course for the first verson only a few of these will be implemented, but I’m thinking to focus on ASP.NET specific features. Also I’m planning that for each problem reported there will be a thorough explanation of how it can be solved and also the possibility to offer consulting to the users through email, phone, remote access, etc.

Once I’ll finalise the list of features for the first version I’ll write my next article, so keep visiting

As I said in the previous post I’m going to release the first product of my mISV in October 1st, 2008. After evaluating lots of ideas I decided to build an “email extractor” or “email spider”. The more general idea is to extract useful data from different sources. This can be:

• Extract email addresses from web.
• Extract email addresses from local documents.
• Extract email addresses from outlook files.
• Extract phone/fax numbers from web.
• etc, etc

This can get pretty complicated and will require a lot of efforts, so for the first version I’ll focus on extracting email addresses from the web. But why this product? The two most important reasons are:

• I have already built a simple version in a few days and I know that it’s possible to built it in the timeframe I have planned.
• There’s definitely a market for it. There are at least 5-6 competitors in the same niche and based on their site activity they seem to be doing well.

But what I will do differently to differentiate from the competition? What I will do better? Why will people buy my product? Well, to tell the truth I have no idea (yet). I think it’s almost impossible to answer these questions before building the product. You have to start working, have some users, get some feedback, improve the product and repeat the cycle. If you improve continuously, you’ll be successful. The key to succeed is not to have a killer idea (at least for 99.99% of us), but is to listen to customers and improve all the time. As said by one of the blogers I read frequently (http://www.followsteph.com/2008/06/26/the-secret-to-success-can-be-summed-up-in-one-word-perseverance/) the key is preservance. To be successful the product may end up very different to how I’m envisioning now, but we’ll know this after some time (I hope not too much time:)).

As I wrote in my last post I moved the blog from the default site www.tollkuci.com to blog.tollkuci.com (Currently www.tollkuci.com redirect to blog.tollkuci.com but this will change later). The reason for this change is because I want to use www.tollkuci.com as the official site of my newly born mISV “Tollkuci Software”. Yes, that’s right, a new mISV on the market and based on my knowledge that would be the first mISV in Albania. I wanted to do it since some time and finally decided to go on and do it. There will be a lot of work to do but I have made some important decisions:

* I have decided on the product I’m going to build.
* I have set the deadline for going live and that will be October 1, 2008.
* I’m almost done selecting the name for the product.
* I have done some basic research and I’m sure there’s enough market for it.

I have started some preparatory work, but the official start date is July 1, 2008 which leaves 3 months for going live. Considering that I will work on it in my “spare” time that’s not a lot of time, but my plan is to spend about 20% of my time. That would be about 120 hours in total and I believe it’s possible to built a simple product and have it ready for sale in 120 hours (others have done it in less time, like 1 week or 1 month part time). Lately, I have tried to adapt myself to the SCRUM methodology and I’m planning to use it for this product. There will be 3 sprints in total and I have already planned the work for the first one. The good thing about SCRUM is that if you cannot implement everything you drop features but you still have a product by the end of the sprint. I’m confident I will have a product ready for sale by October 1, 2008 (thought it may not sale).
In my next post, I’ll write more about the product I’m going to built. Also, I’m planning to write much more often here so stay tuned…

Finally, I’m starting to write again in my site. As you may notice, I have changed the blog address from the homepage http://www.tollkuci.com to http://blog.tollkuci.com. I have some plans on how to use http://www.tollkuci.com for some other purpose…more on this in the next days.

In the meantime, football fans should know that Germany has made once more in the semi-final of the Euro 2008. In the last 4 big competitions, Germany has made it at least to the semifinals in three occasions, and this during a period when German football has been considerd in crisis. I’m cheering for them to go all the way and win it:)

Today, I participated in the first Microsoft Summit held in Tirana, representing AESS Group together with two colleagues. As I have mentioned before, I’m co-founder and Chief Technology Officer of AESS Group, which is one of the best software companies in Albania. It was founded in summer 2005 and was the first Albanian company to achieve the Microsoft Certified Partner status in the country. All our developers have the Microsoft Certified Professional status, with two of them being Microsoft Certified Application Developers and me Microsoft Certified Solution Developer.

As this was the first Microsoft Summit here, it was mainly a general overview of what Microsoft is doing and focused on licensing Microsoft products. However we hope that the increased presence of Microsoft in the country will bring benefits to us as a software company because people “will get used” to pay for licensing software which currently is very difficult.

I got several pictures, but since I forgot my digital camera I used the phone and as you can see the quality is poor (sorry for that). I choose the best two to put them in the site.

Picture taken during the presentation.

Picture taken during the coffee break between the sessions.

It’s been quite a lot of time since my last post, but I’ve been very busy these days. I’m planning to write at least one article each week, mostly related with software development & mISVs, but don’t know if I’ll have enough time to do it. At least I’ll try…
Anyway, I’m writing this article on my Dell Inspiron 9400 laptop running Vista Ultimate x64. I generally love Vista and one of the things I like more is it’s build it search functionality. I’m also running Office 2007 and using Outlook as my email client which also has built-in search capabilities. Searching in Vista is pretty reliable, but unfortunately that’s not the case with outlook. As illustrated in the print screen below, when I search by one the first name of one of my contacts in the All Mail folder, it returns 7 results in about 20 seconds and I know that there are more items.

I had heard for a tool called Lookout that works great, but didn’t work with Outlook 2007. A few days before I run across a post that explained how to run Lookout with Outlook 2007 and I decided to try it. After installing it and letting index my emails the same search returned 84 results in only 1.2 seconds. Pretty impressive!

I find it very hard to believe that Microsoft cannot make searching in outlook better than an add-in developed by a single person. Until Microsoft fix it, I’ll be happy with lookout

P.S For some reasons lookout currently crashes Outlook when it tries to index new emails. So I disabled automatic indexing of new emails and will manually run the indexing procedure.

I was trying to build an itinerary for a planned trip across Europe using Live Search maps. I go to http://maps.live.com, click on the directions link, enter Tirana in the Start box and Thesaloniki in the End box. Everything is fine and the route with all the steps is displayed. However the distance is in miles!!! I check for a way to get the distance in km, but cannot find it. Let’s try the help, still no way. How can they make such a stupid mistake?!!! Do they think that everybody in the World is using miles? Do they know that the international standard is km? Seems they don’t, which I find very very strange. In the end, at least 20% of the people that visit live maps use it to get directions and more than half of them are not from USA, so they’re loosing at least 10% of their customers because of this stupid decision/mistake…
What they could do better:

• Since I’m getting directions in Europe, they should default to km.
• and in any case they should offer the option to switch between miles and km. Not everybody use miles for God’s sake.