Downloading web resources with http.stream - basics - Revision history

Polluks: /* Introduction */ HTTPS

2019-02-17T20:02:08Z

‎Introduction: HTTPS

Krashan: POST is supported now.

2010-12-29T11:34:14Z

POST is supported now.

Krashan: /* Length of data */ More example code and external examples.

2010-08-28T18:12:20Z

‎Length of data: More example code and external examples.

Krashan: /* Length of data */ style

2010-08-28T15:48:31Z

‎Length of data: style

Krashan: /* Introduction */ Added note about HTTP compression.

2010-08-28T15:26:47Z

‎Introduction: Added note about HTTP compression.

Krashan: /* Length of data */ Added second example. Length clarifications.

2010-08-28T15:25:58Z

‎Length of data: Added second example. Length clarifications.

Krashan: /* Length of data */ style++;

2010-08-27T19:46:19Z

‎Length of data: style++;

Krashan: /* Length of data */ Part of contents (to be continued).

2010-08-27T19:45:16Z

‎Length of data: Part of contents (to be continued).

Krashan: New section started.

2010-08-27T18:53:17Z

New section started.

Krashan: /* Minimal example */ Link to example works.

2010-08-27T15:47:38Z

‎Minimal example: Link to example works.

@@ Line 13: / Line 13: @@
 * Easy protocol debugging via MediaLogger.
 The class has some disadvantages however. Some of them may be removed in future versions:
 * No support for persistent connections.
 * No support for HTTP compression.

@@ Line 1: / Line 1: @@
 ''Grzegorz Kraszewski''
 ==Introduction==
-The ''http.stream'' class is one of Reggae stream classes, in other words data sources. In a chain of Reggae objects, a ''http.stream'' instance will be always the first object, having only one, output port. A ''http.stream'' object may be also used standalone, not connected to anything, just to retrieve any data resource reachable via HTTP protocol and particularly its GET request. From this point of view, ''http.stream'' is just embeddable HTTP/1.1 client with simple yet powerful API. A brief list of its features is given below:
+The ''http.stream'' class is one of Reggae stream classes, in other words data sources. In a chain of Reggae objects, a ''http.stream'' instance will be always the first object, having only one, output port. A ''http.stream'' object may be also used standalone, not connected to anything, just to retrieve any data resource reachable via HTTP protocol and particularly its GET and POST requests. From this point of view, ''http.stream'' is just embeddable HTTP/1.1 client with simple yet powerful API. A brief list of its features is given below:
 * Socket API encapsulation. ''http.stream'' completely isolates application (and its programmer) from ''bsdsocket.library'' and TCP/IP stack. Only very basic knowledge of TCP/IP is needed to use ''http.stream'' with success.
 * Unlike ''bsdsocket.library'' base instances, ''http.stream'' objects may be shared between processes (with the only exception that object must be disposed by proces which created it).
@@ Line 13: / Line 13: @@
 * Easy protocol debugging via MediaLogger.
 The class has some disadvantages however. Some of them may be removed in future versions:
 * No support for persistent connections.
 * No support for HTTP compression.

@@ Line 53: / Line 53: @@
 '''The server sends dynamically generated data'''
-Data are usually generated by some server-side script, written in PHP or other language. In this case server does not know the length ''a priori'' so it switches to HTTP chunked transfer mode. The ''http.stream'' object handles it automatically, and reports 0 as ''MMA_StreamLength'', which means that the length is unknown. The only way to process such data is downloading it in blocks in a loop until the object reports MMERR_END_OF_DATA error code.
+Data are usually generated by some server-side script, written in PHP or other language. In this case server does not know the length ''a priori'' so it switches to HTTP chunked transfer mode. The ''http.stream'' object handles it automatically, and reports 0 as ''MMA_StreamLength'', which means that the length is unknown. The only way to process such data is downloading it in blocks in a loop until the object reports MMERR_END_OF_DATA error code. The loop code may look like this:
 It is important that [http://www.ietf.org/rfc/rfc2616.txt RFC 2616], the HTTP specification, does not specify, that static files '''must''' be served without chunked transfer. On the other hand the server is not forced to use chunked transfer for dynamically generated contents. Assumption that server will not use chunks for a file just because the file is static one, may fail. Then, the safe way is to use download loop always and treat ''MMA_StreamLength'' as a hint only.

← Older revision		Revision as of 15:48, 28 August 2010
Line 56:		Line 56:


−	It is important that [http://www.ietf.org/rfc/rfc2616.txt RFC 2616], the HTTP specification, does not specify, that static files '''must''' be served without chunked transfer. On the other hand the server is not forced to use chunked transfer for dynamically generated contents. Assumption that server will not use chunks for a file just because it is static one, may fail. Then, the safe way is to use download loop always and treat ''MMA_StreamLength'' as a hint only.	+	It is important that [http://www.ietf.org/rfc/rfc2616.txt RFC 2616], the HTTP specification, does not specify, that static files '''must''' be served without chunked transfer. On the other hand the server is not forced to use chunked transfer for dynamically generated contents. Assumption that server will not use chunks for a file just because the file is static one, may fail. Then, the safe way is to use download loop always and treat ''MMA_StreamLength'' as a hint only.

@@ Line 15: / Line 15: @@
 * No support for POST requests.
 * No support for persistent connections.
 * Making connection, sending request and receiving response header is done in the constructor, so it is synchronous to the application. Any network delay in constructor blocks the application until timeout or other error is reached. It can be worked around by putting all the network operation on a subprocess.

@@ Line 33: / Line 33: @@
 Usefulness of the above example is limited. It downloads only predefined amount of data (or less, if the resource turns out to be shorter). Usually we want to download all the data and this implies getting the length of it somehow. A few scenarios are possible:
-* The length of data is known before downloading. This is the easiest, but the most rare case. It can be handled exactly as in the example from the previous section &ndash; a statically sized buffer and single ''MMM_Pull()'' call.
-* The server sends a static file. Then it knows the size and passes it in the response header (''Content-Length'' field). The ''http.stream'' object extracts it automatically. Then data length may be obtained by getting ''MMA_StreamLength'' attribute. It means that the length is known before data downloading, so a buffer may be allocated dynamically.
+'''The length of data is known before downloading'''
-* The server sends dynamically generated data. Data are usually generated by some server-side script, written in PHP or other language. In this case server does not know the length ''a priori'' so it switches to HTTP chunked transfer mode. The ''http.stream'' object handles it automatically, and reports 0 as ''MMA_StreamLength'', which means that the length is unknown. The only way to process such data is downloading it in blocks in a loop until the object reports MMERR_END_OF_DATA error code.

← Older revision		Revision as of 19:45, 27 August 2010
Line 31:		Line 31:
	We assume here, ''http.stream'' class has been loaded previously with ''OpenLibrary()'' (see [[Reggae_tutorial:_Accessing_Reggae_in_applications#Opening_and_closing_individual_classes\|Opening and closing individual classes]]). The code will download first 7465 bytes of MorphZone main page (HTML code), assuming there will be no error. This assumption is rather risky, because a network operation can fail for numerous reasons. Then we will be calling method on the NULL pointer and disposing NULL later, which can even lead to application crash. For this reason ''http.stream'' offers a few ways for handling errors. They will be discussed later, for now a minimal error handling is checking ''NewObject()'' result against NULL. This is used in a [http://krashan.ppa.pl/reggae/library/http_simple.c simple example] downloading the first 1000 bytes of a resource specified in the commandline and dumping them into the console. Note that using this program for binary resources (like images) may result in rather weird output... I recommend running this example along with MediaLogger, to learn ''http.stream'' protocol debugging features.		We assume here, ''http.stream'' class has been loaded previously with ''OpenLibrary()'' (see [[Reggae_tutorial:_Accessing_Reggae_in_applications#Opening_and_closing_individual_classes\|Opening and closing individual classes]]). The code will download first 7465 bytes of MorphZone main page (HTML code), assuming there will be no error. This assumption is rather risky, because a network operation can fail for numerous reasons. Then we will be calling method on the NULL pointer and disposing NULL later, which can even lead to application crash. For this reason ''http.stream'' offers a few ways for handling errors. They will be discussed later, for now a minimal error handling is checking ''NewObject()'' result against NULL. This is used in a [http://krashan.ppa.pl/reggae/library/http_simple.c simple example] downloading the first 1000 bytes of a resource specified in the commandline and dumping them into the console. Note that using this program for binary resources (like images) may result in rather weird output... I recommend running this example along with MediaLogger, to learn ''http.stream'' protocol debugging features.
	==Length of data==		==Length of data==
		+	Usefulness of the above example is limited. It downloads only predefined amount of data (or less, if the resource turns out to be shorter). Usually we want to download all the data and this implies getting the length of it somehow. A few scenarios are possible:
		+
		+	* The length of data is known before downloading. This is the easiest, but the most rare case. It can be handled exactly as in the example from the previous section – a statically sized buffer and single ''MMM_Pull()'' call.
		+	* The server sends a static file. Then it knows the size and passes it in the response header (''Content-Length'' field). The ''http.stream'' object extracts it automatically. Then data length may be obtained by getting ''MMA_StreamLength'' attribute. It means that the length is known before data downloading, so a buffer may be allocated dynamically.
		+	* The server sends a dynamically generated data. Data are usually generated by some server-side script, written in PHP or other language. In this case server does not know the length ''a priori'' so it switches to HTTP chunked transfer mode. The ''http.stream'' object handles it automatically, and reports 0 as ''MMA_StreamLength'', which means that the length is unknown. The only way to process such data is downloading it in blocks in a loop until the object reports MMERR_END_OF_DATA error code.

← Older revision		Revision as of 18:53, 27 August 2010
Line 30:		Line 30:

	We assume here, ''http.stream'' class has been loaded previously with ''OpenLibrary()'' (see [[Reggae_tutorial:_Accessing_Reggae_in_applications#Opening_and_closing_individual_classes\|Opening and closing individual classes]]). The code will download first 7465 bytes of MorphZone main page (HTML code), assuming there will be no error. This assumption is rather risky, because a network operation can fail for numerous reasons. Then we will be calling method on the NULL pointer and disposing NULL later, which can even lead to application crash. For this reason ''http.stream'' offers a few ways for handling errors. They will be discussed later, for now a minimal error handling is checking ''NewObject()'' result against NULL. This is used in a [http://krashan.ppa.pl/reggae/library/http_simple.c simple example] downloading the first 1000 bytes of a resource specified in the commandline and dumping them into the console. Note that using this program for binary resources (like images) may result in rather weird output... I recommend running this example along with MediaLogger, to learn ''http.stream'' protocol debugging features.		We assume here, ''http.stream'' class has been loaded previously with ''OpenLibrary()'' (see [[Reggae_tutorial:_Accessing_Reggae_in_applications#Opening_and_closing_individual_classes\|Opening and closing individual classes]]). The code will download first 7465 bytes of MorphZone main page (HTML code), assuming there will be no error. This assumption is rather risky, because a network operation can fail for numerous reasons. Then we will be calling method on the NULL pointer and disposing NULL later, which can even lead to application crash. For this reason ''http.stream'' offers a few ways for handling errors. They will be discussed later, for now a minimal error handling is checking ''NewObject()'' result against NULL. This is used in a [http://krashan.ppa.pl/reggae/library/http_simple.c simple example] downloading the first 1000 bytes of a resource specified in the commandline and dumping them into the console. Note that using this program for binary resources (like images) may result in rather weird output... I recommend running this example along with MediaLogger, to learn ''http.stream'' protocol debugging features.
		+	==Length of data==

← Older revision		Revision as of 15:47, 27 August 2010
Line 29:		Line 29:


−	We assume here, ''http.stream'' class has been loaded previously with ''OpenLibrary()'' (see [[Reggae_tutorial:_Accessing_Reggae_in_applications#Opening_and_closing_individual_classes\|Opening and closing individual classes]]). The code will download first 7465 bytes of MorphZone main page (HTML code), assuming there will be no error. This assumption is rather risky, because a network operation can fail for numerous reasons. Then we will be calling method on the NULL pointer and disposing NULL later, which can even lead to application crash. For this reason ''http.stream'' offers a few ways for handling errors. They will be discussed later, for now a minimal error handling is checking ''NewObject()'' result against NULL. This is used in a [http://krashan.ppa.pl/~~sjjsjh~~ simple example] downloading the first 1000 bytes of a resource specified in the commandline and dumping them into the console. Note that using this program for binary resources (like images) may result in rather weird output... I recommend running this example along with MediaLogger, to learn ''http.stream'' protocol debugging features.	+	We assume here, ''http.stream'' class has been loaded previously with ''OpenLibrary()'' (see [[Reggae_tutorial:_Accessing_Reggae_in_applications#Opening_and_closing_individual_classes\|Opening and closing individual classes]]). The code will download first 7465 bytes of MorphZone main page (HTML code), assuming there will be no error. This assumption is rather risky, because a network operation can fail for numerous reasons. Then we will be calling method on the NULL pointer and disposing NULL later, which can even lead to application crash. For this reason ''http.stream'' offers a few ways for handling errors. They will be discussed later, for now a minimal error handling is checking ''NewObject()'' result against NULL. This is used in a [http://krashan.ppa.pl/reggae/library/http_simple.c simple example] downloading the first 1000 bytes of a resource specified in the commandline and dumping them into the console. Note that using this program for binary resources (like images) may result in rather weird output... I recommend running this example along with MediaLogger, to learn ''http.stream'' protocol debugging features.