Reggae and XPK

Grzegorz Kraszewski

Introduction

XPK is a modular system for data compression. It consists of the main library, xpkmaster.library, and sublibraries stored in LIBS:Compressors/ directory. The main library provides API, while sublibraries provide compression algorithms. XPK has been designed for classic AmigaOS, then it has been ported to MorphOS and is now a native system component with 5 sublibs delivered with the system. Old sublibraries, compiled for M680x0 processors may be installed and used as well. XPK package provides also a set of shell based tools to compress, decompress and manipulate the data.

While XPK tools work only for files, the API itself supports generalized stream concept, which fits Reggae nicely. Reggae interfaces to XPK with a demultiplexer class named xpk.demuxer. The class features not only interface, but also automatic recognition of streams compressed with XPK. It also hides some XPK API problems, like forcing application to decompress whole data chunks.

How Reggae uses xpk.demuxer automatically

The xpk.demuxer class has been designed to be transparent for application in most common cases. It means, when Reggae decoding tree is created with MediaNewObject(), a programmer does not need to care if data are compressed with XPK, or not. In case data are compressed, the first pass of format detection and demultiplexer matching will match xpk.demuxer. Then, xpk.demuxer object announces raw data stream on its output, which will cause the second pass of format detection, on decompressed data this time.

Let's assume a stream containing XPK compressed PCM WAVE audio file. Reggae will cascade xpk.demuxer with wave.demuxer, then add audiopcm.decoder as usual. The file will be then decoded the same way as not compressed one. There is no change in application code, all this is fully automatic. It is worth noting, that XPK support in MediaNewObject() is 100% transparent and all applications, even those written before xpk.demuxer release will gain XPK support with no changes in code or recompilation.

Example 1 (see a frame on the left) loads an XPK packed BMP image (It is a Reggae logo, 512 × 512 pixels, 24-bit) from a file and displays it in a window using picture.output. As BMP format has no compression and the image has big areas of solid colors, the compression ratio is very good (98.48% of gain, from 786 486 to 11 992 bytes). The file is compressed with GZIP packer. What is interesting in this code, it really does not matter if the file reggae512.bmp.xpk is compressed or not. One can try to decompress the file leaving the original name. The .xpk extension does not matter, as Reggae does not rely on extension in format recognition code. The example will load and display uncompressed image with no single character changed in the code. This is how transparency of XPK support works.

A few words about the code. The main() function opens needed libraries and creates a Reggae decoding tree starting from file stream of given name. There is MMT_PICTURE filter added to avoid loading files not being pictures. Then display_picture() function gets picture dimensions and opens a window of needed size. Finally blit_picture() creates picture.output object, connects it with processing tree, sets destinaion RastPort to the window one and specifies offsets, so the image is displayed inside window borders. MMM_Play() method causes the picture to be drawn. Note that picture.output does it in strips, so memory footprint is kept low. Function wait_for_closing() is nothing more than a plain Intuition message handling loop, which exits after window close gadget is clicked.

Compressed data in memory

Instead of loading image from a file, one can convert compressed data into link object or C source and then link it with executable. This is useful for things like program logo or some specific gadget imagery. Example 2 program demonstrates this technique. The code is almost identical to example 1 (that is why the code is not repeated here). The first, obvious difference is addition of large table containing compressed data. The table has been generated with BinToC tool. The second difference is data source specification for MediaNewObject(). Stream type is now memory.stream. Instead of passing stream name, stream handle is specified, which for memory streams is just address of the data. Memory stream needs also length specification (as there is no such thing as stream end) with MMA_StreamLength attribute. The rest of code is unchanged. XPK decompression is fully automatic, as in the previous example.

Using xpk.demuxer for raw data

Sometimes it may be feasible to compress some raw data with XPK. For example storing images as BMP files has a disadvantage of inverted scanline order, which forces Reggae to buffer the whole image in memory. Both example programs above allocate 786 432 bytes for image buffer. It may be avoided by storing the image as raw data with proper top-to-bottom scanline order (in fact it may be avoided also by using some sane image format with strong internal compression like PNG, so the whole XPK game is not worth playing...). The class may be also used for any data not to be decoded by Reggae, just as a comfortable wrapper over XPK API. One may be surprised that XpkRead() and XpkSeek() are not counterparts of Read() and Seek() respectively. XpkRead() forces to read decompressed data in big chunks, size of chunk depends on packing method used and varies typically from 32 to 256 kB. XpkSeek() seeks to the chunk boundary and returns position in the next uncompressed chunk yet to be read.

Using xpk.demuxer as a standalone component, one can avoid these XPK API obstacles. The class provides MMM_Pull() and MMM_Seek() methods with single byte resolution, all buffering needed is handled internally. The example 3 illustrates the usage of xpk.demuxer for raw data. The example is a fully featured file fragments cutter. Because of this the source code is a bit longer, but still compact. Because we can't use MediaNewObject() automation here, file.stream and xpk.demuxer classes have to be opened manually with OpenLibrary() in Main(). Then, after parsing commandline arguments, perform_action() creates Reggae processing pipeline by hand. Stream object and xpk.demuxer instance are created and then connected together with MediaConnectTagList() call. When the pipeline is ready and an output file is opened with usual Open(), pump_data() function is called for every fragment. The pump_data() function could consist of a single MMM_Seek(), MMM_Pull and Write(), but data copying has been programmed in a loop with 64 kB buffer. Then cutting large data chunks is not memory demanding. Thanks to wrapping XPK with xpk.demuxer, data can be pulled with arbitrary amounts every time independently of any previous MMM_Seek() or compressor natural chunk size.

In this example XPK support is transparent as well, as uncompressed files are handled just fine. Note however that this is done other way than in previous examples. When Reggae creates a processing tree automatically with MediaNewObject(), an xpk.demuxer object is created only if datastream is compressed. In the last example, xpk.demuxer instance is always created and data always go through it. Then XPK itself is instructed to pass uncompressed data through. This feature of XPK may cause some troubles however. When xpkmaster.library detects uncompressed data and is requested to pass them through, it wants to know the length of data first. Then, if a stream object connected to an xpk.demuxer instance reports unknown stream length, connecting stream and demuxer with MediaConnect() will fail. Most of stream classes report known stream length. At the time of writing, the only stream able to return 0 (unknown) length is http.stream when using chunked transfer (see RFC 2616 for details).

#define __NOLIBBASE__
#define USE_INLINE_STDARG

#include <proto/exec.h>
#include <proto/intuition.h>
#include <proto/multimedia.h>
#include <classes/multimedia/video.h>

extern struct Library *SysBase, *DOSBase;
struct Library *IntuitionBase, *MultimediaBase;


void wait_for_closing(struct Window *window)
{
  BOOL running = TRUE;
  struct IntuiMessage *imsg;

  while (running)
  {
    WaitPort(window->UserPort);

    while (imsg = (struct IntuiMessage*)GetMsg(window->UserPort))
    {
      if (imsg->Class == IDCMP_CLOSEWINDOW) running = FALSE;
      ReplyMsg(&imsg-;>ExecMessage);
    }
  }
}


void blit_picture(Object *picture, struct Window *window)
{
  Object *pic_output;
  struct Library *picture_output_base;

  if (picture_output_base = OpenLibrary("multimedia/picture.output", 51))
  {
    if (pic_output = NewObject(NULL, "picture.output",
      MMA_Video_DestOffsetX, window->BorderLeft,
      MMA_Video_DestOffsetY, window->BorderTop,
      MMA_Video_RastPort, (ULONG)window->RPort,
    TAG_END))
    {
      if (MediaConnectTagList(picture, 0, pic_output, 0, TAG_END))
      {
        DoMethod(pic_output, MMM_Play);
      }
      DisposeObject(pic_output);
    }
    CloseLibrary(picture_output_base);
  }
}


void display_picture(Object *picture)
{
  LONG width, height;

  width = MediaGetPort(picture, 0, MMA_Video_Width);
  height = MediaGetPort(picture, 0, MMA_Video_Height);

  if (width && height)
  {
    struct Window *window;

    if (window = OpenWindowTags(NULL,
      WA_InnerWidth, width,
      WA_InnerHeight, height,
      WA_Title, (ULONG)"Reggae and XPK tutorial, example 1",
      WA_DragBar, TRUE,
      WA_DepthGadget, TRUE,
      WA_CloseGadget, TRUE,
      WA_IDCMP, IDCMP_CLOSEWINDOW,
    TAG_END))
    {
      blit_picture(picture, window);
      wait_for_closing(window);
      CloseWindow(window);
    }
  }
}


int main(void)
{
  if (IntuitionBase = OpenLibrary("intuition.library", 50))
  {
    if (MultimediaBase = OpenLibrary("multimedia/multimedia.class", 52))
    {
      Object *picture;

      if (picture = MediaNewObjectTags(
        MMA_StreamType, (ULONG)"file.stream",
        MMA_StreamName, (ULONG)"reggae512.bmp.xpk",
        MMA_MediaType, MMT_PICTURE,
      TAG_END))
      {
        display_picture(picture);
        DisposeObject(picture);
      }
      CloseLibrary(MultimediaBase);
    }
    CloseLibrary(IntuitionBase);
  }
  return 0;
}