New Database Model ZXDB

This is the place for general discussion and updates about the ZXDB Database. This forum is not specific to Spectrum Computing.

Moderator: druellan

Post Reply
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

Einar Saukas wrote: Fri Feb 15, 2019 7:36 amAlso technically, I'm now working on some internal improvements to the ZXDB schema (I will detail them later).
I recently mentioned I'm working on ZXDB improvements. Let's talk about it!

According to the old WoS model, each game in the archive could be either a mod of one other game, or inspired by one other game. However there are quite a few exceptions to this rule. For instance, Ku-Ku is a mod of 2 different games: Project Future (game code/engine) and Sabre Wulf (graphics). Miner Willy's Nightmare is a mod of Atic Atac, and at the same time inspired by Manic Miner. A few other titles were inspired by two different games, combining ideas from both. And so on... Since ZXDB schema was originally based on the old WoS site, it also had the same restrictions. Not anymore. ZXDB now contains a new table "relations" to store all kinds of NxN relationships between titles, so each game can now have multiple inspirations, and it can be a mod from multiple titles.

The old WoS site didn't have a proper place to store information about game editors. Instead, this information was simply stored in comments. For instance, Chaos has a comment "There also was a Chaos Editor. There is a patch (Additional material) to create a 256-colour edition for the ULAPlus", and Chaos Editor has a comment "An editor for Chaos". This information was mixed with other comments, and harder to search. Now this information will be also moved to the new table "relations".

This table "relations" will also store graphics engines (like Churrera and NIRVANA), game engines (like PAW and AGD), libraries (like FZX and ZX7), and other development tools currently displayed under the description "authored with". This information was previously stored in "frameworks", but we don't need a separate table anymore since "relations" is much more flexible.

Technically, this new table "relations" will replace table "framework", and columns "entries.is_mod" and "entries.original_id". As usual, I didn't remove the old table and fields yet, so websites based on ZXDB can have time to make the proper changes. But they will be removed in the next ZXDB update.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

A new ZXDB update is available!
User avatar
druellan
Dynamite Dan
Posts: 1466
Joined: Tue Apr 03, 2018 7:19 pm

Re: New Database Model ZXDB

Post by druellan »

Einar Saukas wrote: Tue Feb 26, 2019 3:11 am The old WoS site didn't have a proper place to store information about game editors. Instead, this information was simply stored in comments. For instance, Chaos has a comment "There also was a Chaos Editor. There is a patch (Additional material) to create a 256-colour edition for the ULAPlus", and Chaos Editor has a comment "An editor for Chaos". This information was mixed with other comments, and harder to search. Now this information will be also moved to the new table "relations".
Cool. I was checking that recently and found that it was very confusing, since there are games that have external non-official tools, but others with the editor included on the code or on the side B of the cassette. Now we can have a better classification for those cases too.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

Column "entries.spanish_price" will be removed in the next ZXDB update. I just finished moving this information to the proper releases, using SPA2 as reference. Many thanks to Juan for providing access to SPA2!

Thus if you run a website based on ZXDB, please make the required changes in your queries before the next update.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

A new ZXDB release is available!

This update contains a few new titles, lots of new inlays, and the database structure improvements described in my latest posts :)
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

Yet another ZXDB update is available!

As usual, [mention]R-Tape[/mention] had added recent releases to ZXDB. And [mention]druellan[/mention] is now working on the task of processing corrections reported by users in this forum. Many thanks to both!

My apologies to [mention]pavero[/mention] that I couldn't add the latest instructions and inlays he provided yet. I promise they will be in the next update!
User avatar
PeterJ
Site Admin
Posts: 6854
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: New Database Model ZXDB

Post by PeterJ »

Thanks everyone!

SC has now been updated. Notes will follow shortly.

Peter
User avatar
jpablo
Drutt
Posts: 46
Joined: Sun Jan 13, 2019 12:05 am
Location: Zaragoza, Spain
Contact:

Re: New Database Model ZXDB

Post by jpablo »

PeterJ wrote: Fri Apr 05, 2019 11:07 am SC has now been updated. Notes will follow shortly.
Excellent. Thank you!

I noticed that there is no "main file downloads" block in here:

https://spectrumcomputing.co.uk/index.p ... 96&id=1217

I may be missing something - I was wondering if that's normal
User avatar
PeterJ
Site Admin
Posts: 6854
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: New Database Model ZXDB

Post by PeterJ »

Hi [mention]jpablo[/mention]

We don't currently offer Ocean Software titles for download, although this will hopefully change in the near future.

Peter
User avatar
jpablo
Drutt
Posts: 46
Joined: Sun Jan 13, 2019 12:05 am
Location: Zaragoza, Spain
Contact:

Re: New Database Model ZXDB

Post by jpablo »

I see - thanks for clarifying!
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

A new ZXDB update is available!

It contains lots of new titles (thanks [mention]R-Tape[/mention]!), lots of corrections (thanks [mention]druellan[/mention]!), lots of improved inlays and instructions (thanks [mention]pavero[/mention]!), and probably lots of other stuff I can't remember right now :)
User avatar
PeterJ
Site Admin
Posts: 6854
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: New Database Model ZXDB

Post by PeterJ »

Einar Saukas wrote: Thu Apr 18, 2019 9:26 pm A new ZXDB update is available!

It contains lots of new titles (thanks @R-Tape!), lots of corrections (thanks @druellan!), lots of improved inlays and instructions (thanks @pavero!), and probably lots of other stuff I can't remember right now :)
Thanks everyone. I will update SC over the weekend.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

Here's some technical information that's only really relevant to websites based on ZXDB:

As part of our ongoing effort to improve ZXDB structure, we will soon remove table "formattypes", and the corresponding column "formattype_id" from tables "downloads", "extras", "labelfiles", and "magfiles".

This column currently stores the data format of each file. For instance, if it's a +3 disk image, then it contains the ID for "+3 disk (DSK)". If it's a PDF, it contains the ID for "Document (PDF)". And so on...

The problem is, this information is completely redundant. Just looking at the filename extension will give you exactly the same information. All +3 disk files have extension ".dsk.zip", all PDF documents have extension ".pdf", all TXT documents have extension ".txt", etc. Sometimes the people contributing with ZXDB updates have made mistakes filling this information, so it doesn't match the actual file type. Therefore this is a field that's unnecessary at best, plain wrong at worst.

There's now a new table called "extensions", that store an explanation for each file extension supported by ZXDB. Websites based on ZXDB can simply check the data formats themselves. If anyone needs help with SQL changes here (or anything else really), please let me know!

The main file extensions supported by ZXDB (as detailed in table "extensions") are listed below:

Code: Select all

.avi.zip            Video (AVI)
.ay.zip             Music (AY)
.d80.zip            Didaktik disk (D80)
.dck.zip            Timex expansion (DCK)
.dsk.zip            +3 disk (DSK)
.fdi.zip            TR-DOS disk (FDI)
.gif                Picture (GIF)
.html               Webpage (HTML)
.ifl                Screen dump 8x2 (IFL)
.img.zip            Disciple/+D disk (IMG)
.jpg                Picture (JPG)
.mdr.zip            Microdrive cartridge (MDR)
.mgt.zip            Disciple/+D disk (MGT)
.mid.zip            Music (MIDI)
.mlt                Screen dump 8x1 (MLT)
.mp3.zip            Music (MP3)
.mpg.zip            Video (MPG)
.opd.zip            Opus Discovery disk (OPD)
.p.zip              ZX81 tape (P)
.pdf                Document (PDF)
.png                Picture (PNG)
.pok                Pokes (POK)
.rom.zip            ROM image dump (ROM)
.rzx.zip            Game recording (RZX)
.scl.zip            TR-DOS disk (SCL)
.scr                Screen dump (SCR)
.slt.zip            Snapshot (SLT)
.sna.zip            Snapshot (SNA)
.szx.zip            Snapshot (SZX)
.tap.zip            Tape (TAP)
.trd.zip            TR-DOS disk (TRD)
.txt                Document (TXT)
.tzx.zip            Perfect tape (TZX)
.wdr.zip            Wafadrive wafer disk (WDR)
.wmv.zip            Video (WMV)
.z80.zip            Snapshot (Z80)
.z81.zip            ZX81 tape (81)
.zxk                ZX Vega keymap (ZXK)
_Fonts.zip          Fonts package
_SourceCode.zip     Source code package
This modification will happen gradually. For now, both formattypes 54 (document ".pdf") and 55 (document ".txt") will be removed in the next ZXDB update. OK?
User avatar
PeterJ
Site Admin
Posts: 6854
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: New Database Model ZXDB

Post by PeterJ »

Thanks [mention]Einar Saukas[/mention]

Sounds good to me.
User avatar
pavero
Dynamite Dan
Posts: 1569
Joined: Sat Dec 09, 2017 11:49 pm
Location: The Czech Republic
Contact:

Re: New Database Model ZXDB

Post by pavero »

I agree, this information was really redundant.

But big number of PHP scripts as SC site will be probably complete nonfunctional after removing formattype_id attribute from ZXDB.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

pavero wrote: Fri May 17, 2019 7:13 pm I agree, this information was really redundant.

But big number of PHP scripts as SC site will be probably complete nonfunctional after removing formattype_id attribute from ZXDB.
That's the reason I always announce these changes plenty of time in advance, implement changes as gradually as possible, and wait for feedback so site owners can let me know if there are any other issues I need to consider.

This way, even if sites like SC forget to make the required changes, they will probably just stop showing a text file icon or something similar. I doubt this would break anything important.
User avatar
PeterJ
Site Admin
Posts: 6854
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: New Database Model ZXDB

Post by PeterJ »

This is especially important as ZXDB gets used in more and more projects.
User avatar
pavero
Dynamite Dan
Posts: 1569
Joined: Sat Dec 09, 2017 11:49 pm
Location: The Czech Republic
Contact:

Re: New Database Model ZXDB

Post by pavero »

Einar Saukas wrote: Fri May 17, 2019 7:51 pm This way, even if sites like SC forget to make the required changes, they will probably just stop showing a text file icon or something similar. I doubt this would break anything important.
I can't agree. Many SQL queries will be invalid and the page won't be loaded at all. Especially formattype_id in downloads table is critical.

Maybe some backward compatibility should be considered for the future ...
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

pavero wrote: Fri May 17, 2019 10:13 pmI can't agree. Many SQL queries will be invalid and the page won't be loaded at all. Especially formattype_id in downloads table is critical.
That could happen if I was simply dropping column "formattype_id" right away. But that's not the plan that I presented. Instead, this change will happen gradually.

In the next ZXDB update, this column will still exist, but it won't be populated anymore for formattypes 54 (document ".pdf") and 55 (document ".txt"). That's all. This change won't invalidate current SQL queries or break any pages.

Afterwards, I will let everybody know in advance about the next couple formattype values that won't be populated in the following ZXDB release. And so on.

Each website can choose to remove all references to formattypes at once. Or make the changes gradually as I announce them. Or simply leave it for now and deal with everything later when they have more free time, since the impact will be minimal for now.

pavero wrote: Fri May 17, 2019 10:13 pmMaybe some backward compatibility should be considered for the future ...
That's exactly what's being already considered!

We cannot stop improving ZXDB, just because many projects are using it. However every change we make is always planned carefully, announced in advance, explained in detail, and implemented gradually, in order to make it as easy as possible for others to adapt accordingly. That's the best trade-off possible between evolution and backward compatibility IMHO.

In this particular case, simply removing a redundant column is going to take several ZXDB releases, in order to make it easier for everybody else. I'm not in a hurry :)
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

A new ZXDB update is available!

As usual, it contains several additional titles and hi-res inlays. Special thanks to [mention]R-Tape[/mention], [mention]druellan[/mention], and [mention]pavero[/mention] for their contributions :)
User avatar
StooB
Dynamite Dan
Posts: 1073
Joined: Mon Oct 22, 2018 11:03 am
Contact:

Re: New Database Model ZXDB

Post by StooB »

Einar Saukas wrote: Thu May 23, 2019 5:01 am A new ZXDB update is available!

As usual, it contains several additional titles and hi-res inlays. Special thanks to @R-Tape, @druellan, and @pavero for their contributions :)
Always good to have an update, but I've noticed that the corrections in the Little Bugs thread are being applied very sporadically, or not at all. These reports are now six months old. Is ZXDB suffering from the problems seen in a lot of open-source projects, where only shiny new things are of any interest, and there's no impetus to fix the old issues?

As someone stated in the bugs thread:
richl wrote: Tue Dec 18, 2018 2:30 pm It's a shame that there's probably a ton of other fixes that people have submitted over on the WOS forums during the post-handover period that'll eventually get lost as and when it finally sinks into the mire but whenever I see anything that I vaguely recall posting something about I try and post it here now instead. We should have some sort of Christmas appeal for people to resubmit info here and try and educate others to not waste their time posting on WOS. It's a shame as people genuinely want to help but since no-one other than a blurred circle knows what the Big Secret Plan™ is you'd be better off sending corrections to Santa.... or Einar as I think he now likes to be called ;)
User avatar
djnzx48
Manic Miner
Posts: 729
Joined: Wed Dec 06, 2017 2:13 am
Location: New Zealand

Re: New Database Model ZXDB

Post by djnzx48 »

I was sort of wondering the same thing about these bugfixes in general. I know ZXDB doesn't deal with loading screens (or does it?), but there was a report made about a wrong loading screen over a year ago and nothing happened since then ;)

I guess the problem is related to what was brought up in this thread about marking ZXDB posts, in that you don't actually know what happens with these fixes, whether they get noticed or just left to be forgotten in the depths of some old thread.
Ralf
Rick Dangerous
Posts: 2279
Joined: Mon Nov 13, 2017 11:59 am
Location: Poland

Re: New Database Model ZXDB

Post by Ralf »

I suppose admins have right to edit every post. It would be wrong to overuse this option but in this case it could be justified to make a small edit. Add "Fixed" or something like that in the bottom of the original post. It could work without any changes to the forum engine, I guess.
User avatar
djnzx48
Manic Miner
Posts: 729
Joined: Wed Dec 06, 2017 2:13 am
Location: New Zealand

Re: New Database Model ZXDB

Post by djnzx48 »

How about just changing the subject line of the post? It might be safer than modifying the post's contents.
User avatar
druellan
Dynamite Dan
Posts: 1466
Joined: Tue Apr 03, 2018 7:19 pm

Re: New Database Model ZXDB

Post by druellan »

StooB wrote: Thu May 23, 2019 9:36 am Always good to have an update, but I've noticed that the corrections in the Little Bugs thread are being applied very sporadically, or not at all. These reports are now six months old. Is ZXDB suffering from the problems seen in a lot of open-source projects, where only shiny new things are of any interest, and there's no impetus to fix the old issues?
I'm into that, trying to implement as much as I can of those fixes, in fact, I usually "like" the posts I'm working on, so, you can follow my progress on the "little bugs" thread (both 1 and 2). BUT! I must admit I have some problems with a couple of yours, so, expect a reply about them, soon :)

For now, I'm walking back the old thread, validating the fixes and applying everything I can. After that I think could be a good idea to repost fixes that are still not resolved on the new thread, but first give me some time to go back and check everything I can.
Post Reply