New snap format that convenient for view/modify

[Weiv]

From zx-pk.ru, mine:
@Lethargeek - a simple snap with uncompressed memory at the beginning of the file and the rest of the info at the end (preferably also readable from the hex-viewer) - why not. It is convenient to disassemble, modify and watch registers. It seems, so far there has not been such a format. It's easy to write a converter for other snaps into it. The location of the information about the state of registers and other things floats from the beginning, but it is fixed from the end - you scroll to the end of the file and look, it's convenient. The number of saved pages is determined by the file size.

I have already written:
At the beginning of the file, the memory is 49152 bytes in the current config, then the remaining pages are sequential. It's inconvenient to search for the right page, but it's convenient to disassemble and modify the current memory config.
But it's better, probably, 5th, 2nd, 0th, then 1,3,4,6,7, so that there are pages at fixed offsets.
But for advanced modes of +3 - first 64kb of the current config, then the rest of the pages in sequence.

Completion (tail, postfix) of the file - 3 last lines in a hex viewer:

4 bytes of TStates (for turbo modes), 4 bytes of FramesCounter (people play some games for months, Elite for example), and 8 reserve bytes (MemPtr, additional config info, etc.)

Penultimate line:
AF BC DE HL AF' BC' DE' HL' - 16 more bytes. the most significant registers at the edges and in the center of the line

Last line:
PC, SP, IX, IY, IR,

bytes of information on interrupts and system status flags:
5th bit - temporary prohibition of interrupts at this point (after EI and prefixes. It would seem that it is not needed, but if we got to this point after the transition, this bit gives us information in case of interruption - will it be or not)
4th bit - Halted Z80 (It would seem that it is not needed, but similar to the previous one - during the execution of the HALT command, Z80 points to the command following it, and it is not known whether HALT was executed, or whether we moved here)
In general, these two flags can, it turns out, be replaced by one flag
4th bit - Was Jump (the jump command was executed last), and the previous flags are set based on it and the byte at the address PC-1,
3rd bit IFF2,
2nd - IFF1,
1st-0th - interrupt mode, total 2 or three bits are still free for other flags;

byte number of the active ROM of the peripherals;

last output to FE port;
last output to port 1FFD (+3 and Scorpion), or EFF7 for Pentagon 1024;
last output to port 7FFD;
Spectrum model.

The tail can expand, but between memory and the standard three hex lines, so the offset of the main information from the end of the file does not change.

It remains to come up with an extension) I suggest .snv - it seems to be free, SNap for View.

Yes, since we are making it convenient for viewing in hex viewer - keep the values in the tail in the format with the high bytes in front.

[AndyC]

I'm not sure what advantage storing the +3 RAM configs memory differently is, it would make loading or interpreting the snapshot more difficult since you'd need to divine the paging arrangement first. Just seems needlessly inconvenient to me when just storing the 128 pages in order after the usual 48K ones makes as much sense really (it's not like it is changing the layout as the upper memory is paged, so it's not even a consistent behaviour).

It's also very "old school" by just shoving everything into a big binary blob. If I were designing a snapshot format for the modern world, I'd follow the kind of "packaging" approach more prevalent elsewhere such as Office files. Store everything in a zip file and then use separate files for different components along with some kind of standardized metadata file to describe what the contents are. So you could have a 128K binary blob for memory contents (it 8 16K files maybe), a bit of JSON describing register contents and hardware state etc.

It's more extensible that way so screenshots, tape or disk images required for multi loads, other associated data etc can be added later on without changing the actual format. And since the container is a zip file, you get the benefits of compression while still being easy for someone to modify (since they can just rename it to .zip and extract/modify components within).

[Weiv]

I'm not sure what advantage storing the +3 RAM configs memory differently is, it would make loading or interpreting the snapshot more difficult since you'd need to divine the paging arrangement first

The point is that you can view and disassemble the content of the file exactly as it was in the memory for that moment. Of course you should be aware first that that's +3 extended mode.

[Ralf]

Do you include ROM in these snapshots or not?

Personally I found in several cases useful to have values in my file exactly at the same addresses as they are in Spectrum memory.

Sometimes I export all Spectrum memory from 0 to 65535 to .bin file and do some stuff with it later.

[AndyC]

The point is that you can view and disassemble the content of the file exactly as it was in the memory for that moment. Of course you should be aware first that that's +3 extended mode.

It's maybe better for editing but it's awkward for actually loading the snapshot, since it involves reading various other fields to figure out the model and paging arrangement. And you can't just memory map the first 128Kb, because the order will be different every time.

[Weiv]

It's maybe better for editing but it's awkward for actually loading the snapshot, since it involves reading various other fields to figure out the model and paging arrangement. And you can't just memory map the first 128Kb, because the order will be different every time.

Yes, you need to read the other fields before memory loading but it's not a problem at all. But instead you can easy explore, disassemble and modify such a snapshot, even without an emulator, just with hex viewer + disassembler.

[Weiv]

Do you include ROM in these snapshots or not?

Personally I found in several cases useful to have values in my file exactly at the same addresses as they are in Spectrum memory.

Sometimes I export all Spectrum memory from 0 to 65535 to .bin file and do some stuff with it later.

I think it's not needed mostly. Of course it will be good if we have the same addresses in snapshot as in memory, but it demands to save 16kb of the same ROMs in mostly every snapshot. It's better to have a customizable disassembler. But we can have an option to save the active ROM page first in this snapshot. (Also as the other ROM pages, not actve, after the RAM pages).

[Weiv]

So the third 16-bytes line from the end of the .snv snapshot:
4 bytes TStates, 4 bytes Frames Counter, MemPtr, the number of saved RAM pages, the number of saved ROM pages, 4 bytes of page numbers of the current memory config (bit 7 of the page number - 0-RAM page, 1-ROM page).
And the first 16-bytes line from the end of file:
PC, SP, IX, IY, IR, byte of system state flags and Interrupt mode, byte of system configuration flags (like Issue 2/3), last value to ports FE, 1FFD, 7FFD, and byte of Spectrum model code.
Not active memory page numbers order is saved directly after the ROM pages and it is aligned to 16-byte line(s).

[Ralf]

I think it's not needed mostly. Of course it will be good if we have the same addresses in snapshot as in memory, but it demands to save 16kb of the same ROMs in mostly every snapshot

.

Well, memory and disk space are cheap nowadays. You can fit everything created for Zx Spectrum on one pendrive

But I understand that keeping the ROM is in most cases redundant.
So good luck with your format! Find some niche where it will work better than existing stuff.

[Weiv]

All the existing snapshot formats for ZX Specrtum are not viewable with simple tools like hex viewer almost at all. It's a very simple, viewable, explorable and editable format - it's his niche.

[AndyC]

I don't think it's a good idea to specify things in terms of bytes from the end of the file. Inevitably you will ultimately have to extend the format for some reason and code that works like that will be brittle. Specifying them by how far from the start they are allows for some degree of extensibility at least (though not nearly as much as using a package format). And that probably also means you'll want to store 128K of RAM and 4 ROM images regardless of model.

[Weiv]

Inevitably you will ultimately have to extend the format for some reason and code that works like that will be brittle. Specifying them by how far from the start they are allows for some degree of extensibility at least (though not nearly as much as using a package format).

This format can be extended with new 16-byte lines added _before_ three base info lines, so offsets of the base info from the end of file will be constant.

And that probably also means you'll want to store 128K of RAM and 4 ROM images regardless of model.

No, it does not, you can save only needed pages from RAM and ROM, and specify their numbers in the page number list after all the pages.

[Weiv]

Updated draft specification for .snv format. (The format is based on @Lethargeek's idea of an easy to view and modify ZX Spectrum computer state format)



At the beginning of the file, the contents of the active memory pages (available to the Z80 at the time the snapshot was taken) are stored, without any compression. Further, other necessary memory pages, both RAM and ROM, can be stored.

The order of pages and their size are now not strictly regulated - the order is specified in the list of page numbers, located immediately after the contents of the last page stored in the file, and the page size is calculated from the file size divided by the number of pages stored in it at the end of the file in the descriptor.
Thus, the beginning of the list of page numbers is aligned to the beginning of the file segment with a length equal to the length of the page (it is convenient to find it in the hex viewer).

The first page numbers in the list correspond to the current memory configuration (active in the Z80 address space at the time of the snapshot). However, these pages do not have to be in the file. The bits that determine which pages of the current configuration are stored in the file are stored in the corresponding snapshot descriptor field at the end of the file later.

The page numbers in the list are single-byte by default, but can be double-byte if more than 128 pages of RAM or ROM need to be addressed. The most significant bit of the page number is 0 for RAM, 1 for ROM. Page numbers 0..7 RAM correspond to 128kb Spectrum pagination. The ROM page numbers must be specified additionally, including the ROM pages of the peripherals.

The length of the page number is determined by the number of pages stored in the file field - if it is >128, the numbers are two-byte, otherwise they are one-byte.


At the end of the file there is a postfix block (tail, descriptor) with the rest of the snapshot information. All fields of the postfix block have a fixed offset from the end of the file - thus, by going to the end of the file in the hex viewer, we can easily read the required data at fixed offsets. All fields longer than one byte are written in big-endian format (high bytes at the beginning of the field, low bytes at the end), for ease of viewing in the hex viewer.


The original specification described four 16-byte descriptor strings. The order of the fields is described from left to right, how they will look in the hex viewer, i.e. the last field has the smallest offset from the end of the file.

1st line from the end of the file:
registers Z80 PC,SP,IX,IY,IR,MemPtr(WZ);

last output to additional config port (#1FFD +3 and Scorpio, #EFF7 for Pentagon 1024, #FF for Timex models)
last output to main configuration port (#7FFD for 128x models, #F4 for Timex models)

bytes of information on interrupts and system status flags (penultimate byte of the file):
7th bit - Halted state of the processor (HALT instruction is being executed, and PC points to the next instruction following it, so it is necessary to know exactly whether the HALT instruction was executed or there was a transition to the current address, also the previous byte #76 may be a part of the data of the previous instruction).
6th bit - TempDI flag - temporary interrupt disable: after the EI command and #DD/#FD prefixes, interrupts are disabled if the PC is not set to the given address after the transition command.
5th bit is IFF2,
4th bit is IFF1,
Bits 1 and 0 are interrupt mode. As long as the contents of the 3rd and 2nd bits are not specified yet, they must be zero. These bit numbers are chosen based on readability in the hex file.

ZX Spectrum or it's clone model code (last byte of the file).



2nd line from the end of the file - registers Z80 AF,BC,DE,HL,AF',BC',DE',HL' (the most significant registers at the edges and in the center of the hex viewer line);



3rd line from the end of the file:

total number of memory pages stored in the file (2 bytes). I hope that 65535 pages will be enough forever)

byte of the current memory configuration pages state bits: if bit=1, page is stored in the file.
3rd bit - page in window #C000..#FFFF
2nd bit - page in window #8000..#BFFF
1st bit - page in window #4000..#7FFF
0th bit - page in window #0000..#3FFF
For 8K pages - 7th bit specifies the page in window #E000..#FFFF, 6th bit - in window #C000..$DFFF, etc.

last output to port #FE

System configuration flags field (4 bytes) - various configuration details like Issue 2/3 of 48kb model, connected peripherals, etc. Not yet specified.

FramesCounter (4 bytes) - counter of frames from Z80 reset
TStates (4 bytes) - counter of Z80 cycles from the beginning of the frame. 4 bytes for turbo modes.




4th line from the end - determines the state of the registers of the AY sound chip, starting from register 0. Might be omitted if AY is not present in the system configuration and there are no later descriptor extensions behind it, if later extensions are present it must be present anyway.

In the 4 most significant bits of the 2nd byte from the beginning of the line (1st register) - additional information:
4th bit - sound chip model - 0-AY-3-8910/12, 1 - YM2149F

The 4 high bits of the 14th byte from the beginning of the line (13th register) store the number of the selected AY register (the last output to the port of the register number is #FFFD for 128x and Didactic Melodic, #F5 for Timex models, #3F for Fuller Box)





Format extensions are done by adding additional lines before the main lines, so that the field offsets of the main lines from the end of the file do not change. The prescriptions for format extensions are:

1) each extension occupies an integer number of 16-byte lines
2) each extension has a fixed offset from the end of the file
3) the choice of fields and their order should be based on the principle of readability of fields from the hex viewer, thus, more important fields should be located on the edges and in the centre of the 16-byte line.
4) peripheral memory pages should be stored in the general format of memory pages, behind the RAM/RAM pages of the model, and their numbers should be encoded in the general order of memory page numbers.



Left unspecified:

ROM page number order
system configuration flags in the corresponding field

[AndyC]

This format can be extended with new 16-byte lines added _before_ three base info lines, so offsets of the base info from the end of file will be constant.

Whilst that makes it easier from a hex editor perspective (well unless your hex editor happens to not use 16-bytes per line) it also makes it a pain in the ass to deal with from a programming perspective. You can't use a standard loader, like you could with a package or RIFF format and you can't even just memory map the file onto a C structure, because you don't know where things begin in advance. So you instead have to pretty much read it byte by byte and process it serially, which is decidedly slower and more cumbersome.

[Weiv]

Whilst that makes it easier from a hex editor perspective (well unless your hex editor happens to not use 16-bytes per line) it also makes it a pain in the ass to deal with from a programming perspective. You can't use a standard loader, like you could with a package or RIFF format and you can't even just memory map the file onto a C structure, because you don't know where things begin in advance. So you instead have to pretty much read it byte by byte and process it serially, which is decidedly slower and more cumbersome.

Yes, but the speed of reading will not much slower) maybe it will be a bit more cumbersome code of the snapshot reading/writing . But the snapshots will still be more readable and convenient to view/explore/modify, as it's the main purpose of the format.

[ketmar]

it's quite easy to write converters from existing snapshot formats to anything you want. there is libspectrum, for example, which can parse almost anything. so what is the reason to have a new format instead of writing small utility that will convert things back and forth? it's like we don't have enough snapshot formats already…

[Lethargeek]

Whilst that makes it easier from a hex editor perspective (well unless your hex editor happens to not use 16-bytes per line) it also makes it a pain in the ass to deal with from a programming perspective. You can't use a standard loader, like you could with a package or RIFF format and you can't even just memory map the file onto a C structure, because you don't know where things begin in advance. So you instead have to pretty much read it byte by byte and process it serially, which is decidedly slower and more cumbersome.

you can't just memory map a standard 128k sna onto a single C structure either (and it's a bad practice anyway)

it's quite easy to write converters from existing snapshot formats to anything you want. there is libspectrum, for example, which can parse almost anything. so what is the reason to have a new format instead of writing small utility that will convert things back and forth? it's like we don't have enough snapshot formats already…

so what is the reason to not having the emulator transparently converting things back and forth for you instead of doing it manually?

[ketmar]

so what is the reason to not having the emulator transparently converting things back and forth for you instead of doing it manually?

the low need for such thing for emulator authors. but anybody can take any FOSS emulator and modify it to suit their needs, without trying to introduce Yet Another Snapshot format (of questionable usefulness, i'd say).

[Lethargeek]

the low need for such thing for emulator authors

eh, so this means you think emulator authors have low need for their emulators being used by some of the most active people in the community

but anybody can take any FOSS emulator and modify it to suit their needs, without trying to introduce Yet Another Snapshot format (of questionable usefulness, i'd say).

wut, so you say anybody can support a new format without introducing a new format

[AndyC]

It would be about a billion times easier to write something that opens existing snapshot formats and just present it in the hex editor view, if that's what you want though.

Making a new format doesn't make a lot of sense.

[ketmar]

eh, so this means you think emulator authors have low need for their emulators being used by some of the most active people in the community :?

you mean "by people most actively proposing new snapshot formats"? yeah, you are right.

wut, so you say anybody can support a new format without introducing a new format :? :lol:

wut, you suddenly lost the ability to understand context of the written text? definitely a good way to convince one of the emulator authors to support your new format.

[firelord]

It would be about a billion times easier to write something that opens existing snapshot formats and just present it in the hex editor view, if that's what you want though.

Making a new format doesn't make a lot of sense.

Yes I wrote in PHP very quickly a "kind of" parser that you loaded a SNA file and a list if RAM addresses . Then you can modify that RAM addresses . I used it mainly for changing item names in some games I play. I did it very quickly.

What I'm saying us that I agree that a parser/editor would be much more useful. Also, since I use old emulators (DOS era) in my systems I would probably ignore any new format.

[Bedazzle]

And since the container is a zip file

Where you can have a ton of ram pages separately in files, even by 8K chunks for ZX Next.
Each page can be viewed separately, or easily replaced.

[Bedazzle]

So the third 16-bytes line from the end of the .snv snapshot:

Why you try to put everything in one file?
Pages as separate binary files, registers and other sysinfo in text file (json, xml, ...) at my point of view is more manageable and explorable by hand/naked eyes.

[Bedazzle]

Making a new format doesn't make a lot of sense.

...and what, if new format is more extensible, and supports previously unknown things, or even new hardware (ZX Next for example)?