Any packers with support of overlapping src/dest?

[Nienn Heskil]

Not sure what this feature is called. Basically you give it the dest address for the unpacked data in RAM, and also the address where the compressed block will be located (or possibly the end bound). Then at the time of packing, if it finds that at some point dest catches up with src, it only packs the data until that point, and stores the rest of it unpacked. Very useful for large blocks loaded towards #FFFF and uncompressed down.

[Einar Saukas]

It's not exactly the same thing but...

When you compress with ZX7, it tells you the minimum "delta" you need when overlapping compressed and uncompressed areas. It's typically quite small, for instance when delta=2 you can decompress almost everything in-place, then simply poke the last 2 bytes manually. And it also allows compressing backwards.

[Nienn Heskil]

I'm confused about why you'd want to POKE anything as long as you maintain that delta distance (and if you don't, I suppose the file won't decompress correctly in the first place?). Did you mean something like: decompress as closely as the delta allows, then LDDR the unpacked data to its correct location if needed. That is an option but, as you say, it's not quite the same thing, and it needs extra memory to work.

[Einar Saukas]

Suppose you want to decompress exactly 16384 bytes of data from address 32768 to address 49151. Suppose that it gave you "delta=2" when you compressed it. Then you have the following options:

• Load compressed data ending at address 49153, then decompress it into addresses 32768 to 49151. You will be using 2 extra bytes beyond the destination area.

• Compress first 16382 bytes only, load compressed data ending at address 49151, then decompress it into addresses 32768 to 49149. Afterwards poke the last 2 values to addresses 49150 and 49151. You won't use any address outside the destination area.

• Load compressed data starting at address 32766, then decompress it backwards into addresses 49151 to 32768. You will need to use 2 extra bytes behind the destination area.

• Compress last 16382 bytes only, load compressed data starting at address 32768, then decompress it backwards into addresses 49151 to 32770. Afterwards poke the last 2 values to addresses 32768 and 32769. You won't use any address outside the destination area.

I don't recommend decompressing everything into the wrong place then using LDDR, because it's slower and requires more instructions.

[Nienn Heskil]

• Compress first 16382 bytes only, load compressed data ending at address 49151, then decompress it into addresses 32768 to 49149. Afterwards poke the last 2 values to addresses 49150 and 49151. You won't use any address outside the destination area.

That seems to imply quite some work beyond doing a 'simple POKE' (including, interestingly, running the packer twice). But I see what you mean.

[Einar Saukas]

• Compress first 16382 bytes only, load compressed data ending at address 49151, then decompress it into addresses 32768 to 49149. Afterwards poke the last 2 values to addresses 49150 and 49151. You won't use any address outside the destination area.

That seems to imply quite some work beyond doing a 'simple POKE' (including, interestingly, running the packer twice). But I see what you mean.

Now it's me that don't see what you mean!

To pack: compress everything except last 2 bytes (i.e. addresses 32768 to 49149), then just take note of last 2 bytes (addresses 49150 and 49151).

To unpack:

Code: Select all

ld hl,nn        ; compressed block located such that it ends at address 49151
ld de,32768     ; destination
call dzx7       ; decompress from 32768 to 49149
ld hl,nn        ; last 2 bytes
ld (49150),hl   ; poke addresses 49150 and 49151

Notice this solution is only needed if you really cannot afford to spend 2 bytes (or whatever is the value of "delta") before or after your destination area. If you can, the 1st and 3rd solutions I mentioned are even simpler.

[Nienn Heskil]

I'm mainly looking at it from the point of view of automating this sort of thing for a loader maker system (similar to this one by Zup, but exporting to several tape/disk formats at once directly from sjasmplus). The loading process is:
1. Screen, loaded to RAMTOP (#5E00), decompressed to #4000.
2. 128K pages, loaded to RAMTOP, decompressed to somewhere >=#C000 as needed.
3. Final 128K page+main code as a single block, loaded to (#10000-length) and decompressed down as needed. -Here this sort of problem can arise.

It's possible to tell beforehand that packed and unpacked data are going to be overlapped in the memory, but that doesn't tell you whether the data would decompress correctly. So in my case (p.3 above), if there's overlapping, I assume the first thing to do is find whether (packed_start+packed_length - unpacked_start+unpacked_length) =< delta. If it is, all is fine and one can use the packed result as is. If not, the file needs to be re-compressed with length (unpacked_length-delta). In that case you also need to move the 'unpacked delta bytes' together with their length to the beginning of the packed file, mark it in some way, make changes to the unpacker to support this sort of thing, and so on. That's quite a number of operations. You might see why I sometimes favor the LDDR approach.

[TomD]

Following this with interest as I'm in the middle of writing a universal tape loader myself. This was brought about due to a new 128k project I'm working on which requires memory page 7. I found this problematic as the 128k buffers are also located there so I created a simple m/c loader to get around it. Am now looking at packing the memory pages to reduce the load size, hence the interest. I've got a backwards compressor working (using my own zxsc compressor) but as you both stated it sometimes fails due to the overlap and I need to now work on the delta. So a WiP.

One thing I did want to mention is that for disk based loaders, for my Z80onDSK utility I built a de-compressor which works straight from disk avoiding the need to stage the compression in memory first. This slows down the loading but as it is disk it loads pretty fast anyway and I found it was the simplest solution especially as memory page 7 is even more of a pain on +3 models due to 3DOS.

TomD

[Dr beep]

I wrote MULTITASK.

It can compress and swap 3 16K games on a 48K ZX Spectrum.

To swap games you use an intrupt, compress 16K to 32K, select another 16K bank and rhe programm swaps compressed code there and decompresses 16K to 32K and runs again. The part which you compress/uncompress is always fully useable.

[zup]

Well, I guess you looked into sna2zx7. Well, it may help you but keep in mind that it inherites some complexities of previous utilities (sna2transtape) that were experimental and do many things they should not (forcing 48k mode, storing 128k programs).

sna2zx7 makes no assumptions over the snapshot (like real transfer interfaces), so...
- It needs to restore every CPU register.
- It needs to save every byte of memory (spoiler: it doesn't, and note that it checks if zx7 data is bigger than uncompressed data).
- It corrupts some lines on screen, because it is the only "safe" place to do so (=because any other place may be useful data, even if it is zero filled).
- To avoid overruning compressed data, it "reserves" 256 bytes that can not be compressed, and are overwritten on the last block load (NOTE: in fact, the last compressed block overwrites unused parts of the loader up to the last 128 bytes. I guess that 128 bytes are enough to ensure data not overrunning the compressed block).
- Because of this restrictions, it may reject some sna that won't work (stack or PC in the same areas as the Phoenix routines, or if data is going to overrun the compressed data).
- I was going to "reserve" 512 or 768 bytes so I could put a turbo loading routine (making loads even faster) but I got lazy...

How can it be different for you?
- Well, most assembled programs are going to be launched using RANDOMIZE USR. So you won't need to use a complex routine to store every CPU register, just jump into that address (well, except SP and maybe BC). No need to trash your loading screen.
- I don't know how is the output of that assembler, but I guess you won't need to save every bit of memory. There will be plenty of memory "holes", so maybe you should parse your assembler output to find where are that holes and put your loader there (i.e.: if the assembled output goes from 40000 to 60000, you have plenty of space above 60000 to avoid overruns).
- Same for pages 1,3,4,6 and 7. No need to load anything if the assembler didn't put anything there.

In most cases I don't think you'll need to worry about overruns. Except the more complex games (like Los Amores de Brunilda) there are few games that requires every byte of RAM, so you'll be able to find those "holes" and use them.

[TomD]

It took me a while but my Z80onDSK utility restore every byte of memory, minus the screen, from Z80 snapshots, plus all the registers, interrupt state etc... This has been especially useful for some of the more recent All-In-One hacks that have been released for some of the tape only based multi-loaders. The is mostly achieved by having a disk based decompressor, moving memory around and using the screen area as a buffer which does corrupt it at the end of the load.

TomD

[Einar Saukas]

I'm mainly looking at it from the point of view of automating this sort of thing for a loader maker system (similar to this one by Zup, but exporting to several tape/disk formats at once directly from sjasmplus). The loading process is:
1. Screen, loaded to RAMTOP (#5E00), decompressed to #4000.
2. 128K pages, loaded to RAMTOP, decompressed to somewhere >=#C000 as needed.
3. Final 128K page+main code as a single block, loaded to (#10000-length) and decompressed down as needed. -Here this sort of problem can arise.

It's possible to tell beforehand that packed and unpacked data are going to be overlapped in the memory, but that doesn't tell you whether the data would decompress correctly. So in my case (p.3 above), if there's overlapping, I assume the first thing to do is find whether (packed_start+packed_length - unpacked_start+unpacked_length) =< delta. If it is, all is fine and one can use the packed result as is. If not, the file needs to be re-compressed with length (unpacked_length-delta). In that case you also need to move the 'unpacked delta bytes' together with their length to the beginning of the packed file, mark it in some way, make changes to the unpacker to support this sort of thing, and so on. That's quite a number of operations. You might see why I sometimes favor the LDDR approach.

Perhaps you are overcomplicating this problem...

Your description suggests that you want to unpack the last block to the top of memory. For instance, suppose you want to decompress your last block into memory addresses START=$5B08 to END=$FFFF. This is how you do it:

• Compress this last block using the backwards option in ZX7. Suppose you got DELTA=3 and PACKED_SIZE=$7123.

• Load the compressed block to address START-DELTA = $5B08-3 = $5B05. Therefore the last address of the compressed block will be START-DELTA+PACKED_SIZE-1 = $5B08-3+$7123-1 = $CC27

• Decompress the memory block using ZX7 backwards version, as follows:
  

  Code: Select all

  ld hl,$CC27
  ld de,$FFFF
  call dzx7_back

That's all.

[Nienn Heskil]

Some interesting insights there guys.

My loader maker works similarly to sjasmplus's built in TAP exporter, but with compression, more formats and a few other features. The data blocks come with autocompiled headers loaded into SP that control what to do with the blocks (where they are unpacked etc), and where to jump next. Each header ends with a DEFW LOAD. In this way, it's possible to append several unrelated programs that are compiled separately to the same disk/tape image, for instance game intro + the game itself. When one program exits, the loading continues, assuming it keeps the SP and all that. This can be further hacked so that it can even serve as a multiload procedure for loading ingame levels.

The realtime LZ-unpacking idea is interesting, I think this can work for TAP just as well with some effort (literal byte: write to dest as is, LZ-sequence: save src/dest/length to a FIFO buffer, and process that during the tape delays). Unfortunately, in most cases it would mean using a custom loader, which makes it less practical today. As well, for instance in TR-DOS the minimum amount of data you can load is 256 bytes (one sector). And if you already have 256 bytes to spare, LDDR is likely a better option.

[TomD]

Finished the first version of my utility, please take a look here viewtopic.php?f=3&t=3239

I got the backwards compressor working and added a feature to increase the delta if required, default is 3. It tests the compression to make sure the given delta works and errors if not. I will probably adapt a future version so it works out the delta and adjust automatically.

I kept away from a customer loader for the tapes as I wanted it to work with the ROM routines and SD card loaders.

The utility also creates +3 disks as well as tapes but I use a decompressor which works straight off the disk so in place isn't required.

The compressor I use is a variant of my LZF one (https://tomdalby.com/other/lzf.html). It doesn't shrink the files as well as zx7 but where is the fun is using somebody else's

TomD

[Alcoholics Anonymous]

zx7 is a great compromise hitting a very neat sweet spot in terms of size, speed and compression compared to other compressors. We've been using it for compressing data+bss segments for code stored in roms (variables get decompressed into ram to initialize ram) for things like sbcs and game rom cartridges. The backwards dzx7 is used to make compressed executables doing what is mentioned in this thread (load to ram, lddr to the top of the memory area that will be occupied, decompress, jump to the start address); I never really thought this would be too terribly useful on mostly 64K systems but there are people out there doing unusual things in the present day like sending programs over serial running at slow rates where this works well. Tape loading compression I never tried to make fully automatic -- I thought the variety of requirements might confound a common formula -- but if it works well like that, that's an easy tool to use.