ZXDB and ZX-Art

This is the place for general discussion and updates about the ZXDB Database. This forum is not specific to Spectrum Computing.
moroz1999
Berk
Posts: 47
Joined: Fri Mar 30, 2018 8:22 pm

ZXDB and ZX-Art

Post by moroz1999 » Fri Mar 30, 2018 8:38 pm

Hello guys!
First of all, thanks for ZXDB efforts. I've once imported a lot of ZXDB information to ZX-Art, and today I've tried to rerun the import again with some updated functionality.
I've found a strange problem: it seems like IDs of downloads have changed. For example, The Spy Who Loved Me TZX download "SpyWhoLovedMeThe.tzx.zip" previously had ID 228, and now it seems to have ID 226.
This brings some problems to my import procedure: I'm storing each original ID of ZXDB software and downloads, so later I would be able to re-run the import and only update what really changed. This is however impossible if all IDs change in different versions of ZXDB, so I'll have to delete all ZXDB downloads from ZX-Art before each import procedure run.
Did I understand the problem correctly? Or should I recheck my import procedure code to find out why the download IDs have changed?
0 x

User avatar
Einar Saukas
Manic Miner
Posts: 526
Joined: Wed Nov 15, 2017 2:48 pm

Re: ZXDB and ZX-Art

Post by Einar Saukas » Sat Mar 31, 2018 5:33 am

moroz1999 wrote:
Fri Mar 30, 2018 8:38 pm
Hello guys!
First of all, thanks for ZXDB efforts.
Thanks for the incentive! :)

moroz1999 wrote:
Fri Mar 30, 2018 8:38 pm
I've once imported a lot of ZXDB information to ZX-Art, and today I've tried to rerun the import again with some updated functionality.
I've found a strange problem: it seems like IDs of downloads have changed.
Sorry, I changed these IDs a while ago because "someone" complained about non-consecutive IDs in ZXDB. But they are meaningless anyway! This column DOWNLOADS.ID only exists for technical reasons. The natural key in this table would be (ENTRY_ID + FILE_LINK), however I couldn't use it because data I imported from original WoS archive contained thousands of empty files and it's still my goal to ensure that ZXDB preserves 100% of the original WoS content.

When importing from table DOWNLOADS, I suggest you skip all rows with FILE_LINK IS NULL, and use (ENTRY_ID + FILE_LINK) as composite key instead.

Please let me know (either here or by email) if you have problems with anything else!

moroz1999 wrote:
Fri Mar 30, 2018 8:38 pm
This brings some problems to my import procedure: I'm storing each original ID of ZXDB software and downloads, so later I would be able to re-run the import and only update what really changed. This is however impossible if all IDs change in different versions of ZXDB, so I'll have to delete all ZXDB downloads from ZX-Art before each import procedure run.
As a general rule, IDs in ZXDB rarely change, but only ENTRIES,ID is really guaranteed to never change. Even so, a few entries may sometimes disappear, when someone finds out that 2 different entries are just re-releases of the same program, perhaps under a different name. This is the reason old WoS archive doesn't have contiguous IDs, in the beginning those merges happened all the time!

If you can afford to delete all your ZXDB content and re-import everything periodically, this is by far the easiest way. Otherwise if you can give me more information about your database model (again either here or by email), we can work together to figure out a better integration to make this process easier for you.
0 x

User avatar
Einar Saukas
Manic Miner
Posts: 526
Joined: Wed Nov 15, 2017 2:48 pm

Re: ZXDB and ZX-Art

Post by Einar Saukas » Wed Apr 04, 2018 9:28 pm

Next ZXDB update will remove meaningless IDs, including the one you mentioned. For further details, please take a look at this post:

viewtopic.php?f=32&t=146&start=70#p8414
0 x

moroz1999
Berk
Posts: 47
Joined: Fri Mar 30, 2018 8:22 pm

Re: ZXDB and ZX-Art

Post by moroz1999 » Fri Apr 06, 2018 10:41 pm

Thank you! Now I understood. I'll remove the releases from ZX-Art and reimport the database, since it's still possible to do it. However, in future I'll only update the entries and releases without deletion, because in ZX-Art the releases can be commented and soon could be voted as well, so deleting release will lose this information.
Also, I'm planning to group the software by "entries" (prods in ZX-Art terminology). One entry can contain releases from numerous sources and databases.

How do you think is it possible that FILE_LINK would ever change for existing downloads? My goal is making a continuous integration, where import from ZXDB can be re-run periodically, and only the changes/additions would be imported without deleting anything, so multiple bases could be united in one.
0 x

moroz1999
Berk
Posts: 47
Joined: Fri Mar 30, 2018 8:22 pm

Re: ZXDB and ZX-Art

Post by moroz1999 » Sun Apr 15, 2018 5:20 pm

I'm experimenting with smarter continuous integration. I've got the case with our Crystal Kingdom Dizzy remake. Since it has been added into both databases separately, there is no ZXDB entry ID in ZXArt database yet.
I was going to resolve it using my current common logic I use for all unknown software entries:
1. Check if there is ZXDB entry ID from previous import.
2. If there is not, then check every releases's file MD5. If there is a file with same MD5 in ZXArt, then it's 99.99% the same software entry.
3. If there are not, then try to search for the software with same name and same year.

My problem is that we have different files for releases:
https://spectrumcomputing.co.uk/zxdb/si ... 17.tap.zip
https://zxart.ee/release/id:192279/file ... 017%29.zip

SpectrumComputing copy don't contain the text file and SFV checksum file. Where have you got this copy from? The problems with throwing out the info files are:
* The new version of release file appears even if there are no changes in the game. That's what really bothers me.
* Some authors can get really angry because of this. I can remember a lot of bad blood between zxaaa.net archive and demoscene authors because of release text files being throwing out from original archives on zxaaa.net.

I know it's easy to update Crystal Kingdom Dizzy files, but are there more? Please check.
0 x

moroz1999
Berk
Posts: 47
Joined: Fri Mar 30, 2018 8:22 pm

Re: ZXDB and ZX-Art

Post by moroz1999 » Tue Apr 17, 2018 10:10 pm

So, having entry_id + release_seq as a GUID for ZX-Art release won't help really. It seems like I have to use something like:
md5 (entry_id + release_seq + file_link).

Why? Because of architecture differences: ZXDB can have more than file download for release, and ZX-Art cannot. Why? For example, let's take Castlevania. It's being stored as 1 Entry, 1 Release and 11 downloads. In ZX-Art I would like to store 1 Entry and 11 Releases, each release having one only download file, since each release would have it's format, it's supported hardware and supported language.
0 x

User avatar
Einar Saukas
Manic Miner
Posts: 526
Joined: Wed Nov 15, 2017 2:48 pm

Re: ZXDB and ZX-Art

Post by Einar Saukas » Wed Apr 18, 2018 12:01 am

moroz1999 wrote:
Sun Apr 15, 2018 5:20 pm
SpectrumComputing copy don't contain the text file and SFV checksum file.
The text file is available from Spectrum Computing:

https://spectrumcomputing.co.uk/index.p ... 6&id=30381

Unless you mean a different text file?

The SFV checksum file is not provided because, frankly, it's not needed. A SFV checksum doesn't work as proof of authencity (anyone modifying your file on purpose could easily make careful changes to produce the same SFV checksum). Therefore SVF checksum only helps detect corrupted files. However all playable files referenced in ZXDB are always zipped, and the zip format already contains corruption detection.

moroz1999 wrote:
Sun Apr 15, 2018 5:20 pm
Where have you got this copy from? The problems with throwing out the info files are:
We didn't throw out files, except for the SVF checksum that would have no purpose here.

Both the TAP and TXT files are available from the same Spectrum Computing. Keeping them accessible individually from the webpage, readable by just clicking on it, actually increases the chance players will actually read it.

moroz1999 wrote:
Sun Apr 15, 2018 5:20 pm
* The new version of release file appears even if there are no changes in the game. That's what really bothers me.
I'm sorry, but I don't know what you meant here...

moroz1999 wrote:
Sun Apr 15, 2018 5:20 pm
* Some authors can get really angry because of this. I can remember a lot of bad blood between zxaaa.net archive and demoscene authors because of release text files being throwing out from original archives on zxaaa.net.
Again, we are not throwing out any text files. I doubt any authors would get angry just because we are providing individual links to all their files.

Notice that even ZX-Art allows downloading these files separately:

https://zxart.ee/mobile/eng/software/ga ... 7-v104tap/

moroz1999 wrote:
Sun Apr 15, 2018 5:20 pm
I know it's easy to update Crystal Kingdom Dizzy files, but are there more? Please check.
Yes, there are about 100,000 more files, with playable files available for download separately from instructions (although they are always displayed together in the same page). Most of these files were inherited from Martijn's original WoS archive, where they always got distributed this way. AFAIK authors never complained about it in 2 decades of WoS existence!

Perhaps a different organization would work better, but frankly it's too late to change it now.
0 x

User avatar
Einar Saukas
Manic Miner
Posts: 526
Joined: Wed Nov 15, 2017 2:48 pm

Re: ZXDB and ZX-Art

Post by Einar Saukas » Wed Apr 18, 2018 12:14 am

moroz1999 wrote:
Tue Apr 17, 2018 10:10 pm
So, having entry_id + release_seq as a GUID for ZX-Art release won't help really. It seems like I have to use something like:
md5 (entry_id + release_seq + file_link).

Why? Because of architecture differences: ZXDB can have more than file download for release, and ZX-Art cannot. Why? For example, let's take Castlevania. It's being stored as 1 Entry, 1 Release and 11 downloads. In ZX-Art I would like to store 1 Entry and 11 Releases, each release having one only download file, since each release would have it's format, it's supported hardware and supported language.
1. Whatever you have just called "Release", can you rename it to "Download"?

Now your phrases will read like this:

"ZX-Art cannot have more than FILE for each DOWNLOAD. In ZX-Art I would like to store Castlevania as 1 ENTRY and 11 DOWNLOADS, each DOWNLOAD having one only FILE, since each DOWNLOAD would have it's format, it's supported hardware and supported language."


2. Now create a new class called "Release".

It should be like this:

Each ENTRY can have many RELEASES. Each RELEASE can have many DOWNLOADS. Each DOWNLOAD can have just one FILE, since each DOWNLOAD would have it's format, it's supported hardware and supported language.
0 x

moroz1999
Berk
Posts: 47
Joined: Fri Mar 30, 2018 8:22 pm

Re: ZXDB and ZX-Art

Post by moroz1999 » Wed Apr 18, 2018 8:51 am

We just have different views on "releases". For me each release is a different version of the same software. My arguments are:
1. This architecture decision fits better to idea of importing different versions from different sources.
2. One release can only have one file archive. It can surely have some additional materials as screenshots, cover art and so on, but there is only one file which can be started up in emulator.
3. Releases have languages, not the downloads have. For example, what is a supported languages set for cover art? For *.ay file? Or what supported hardware can cover art have? That's just a signal that may be supported hardware/language are a part of release, not download, since all releases can have supported hardware naturally, and supported hardware is not applicable to some types of downloads.
4. Release can only have a single version number. Each time authors make a new release (as with Pro Tracker or Burial Graphics Editor), a new release should be done (or a current one updated with new info), but not combining different versions in single release (as with Castlevania).

May be I should have named it "versions" not "releases". That would fit my concept idea much better.

I'm not telling that's the only "right" way to build software archive architecture, just a bit alternative way to look on things. And I've written it just to demonstrate that integer PK in most cases is not redundant, it's really practical as a GUID for integration. Much more practical than md5 (entry_id + release_seq + file_link) or simply (entry_id + release_seq + file_link).
0 x

User avatar
kolbeck
Dizzy
Posts: 64
Joined: Mon Nov 13, 2017 9:04 pm

Re: ZXDB and ZX-Art

Post by kolbeck » Wed Apr 18, 2018 10:47 am

moroz1999 wrote:
Wed Apr 18, 2018 8:51 am
2. One release can only have one file archive. It can surely have some additional materials as screenshots, cover art and so on, but there is only one file
I don't agree on this one. One release can have multiple files (or downloads as Einar mentions) - for example the same release can be available in the following, but not limited to, formats:
.tzx file
.tap file
.z80 snapshot
.sna snapshot

...and all the other emulator formats.

/T
0 x
https://zxinfo.dk - another ZXDB frontend
https://api.zxinfo.dk/doc/ - ZXDB API for developers

Post Reply