New Database Model ZXDB

This is the place for general discussion and updates about the ZXDB Database. This forum is not specific to Spectrum Computing.

Moderator: druellan

Post Reply
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

A new ZXDB update is available!

It contains lots of new titles (thanks [mention]R-Tape[/mention]!), lots of corrections (thanks [mention]druellan[/mention]!), lots of improved inlays and instructions (thanks [mention]pavero[/mention]!), and probably lots of other stuff I can't remember right now :)
User avatar
PeterJ
Site Admin
Posts: 6854
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: New Database Model ZXDB

Post by PeterJ »

Einar Saukas wrote: Thu Apr 18, 2019 9:26 pm A new ZXDB update is available!

It contains lots of new titles (thanks @R-Tape!), lots of corrections (thanks @druellan!), lots of improved inlays and instructions (thanks @pavero!), and probably lots of other stuff I can't remember right now :)
Thanks everyone. I will update SC over the weekend.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

Here's some technical information that's only really relevant to websites based on ZXDB:

As part of our ongoing effort to improve ZXDB structure, we will soon remove table "formattypes", and the corresponding column "formattype_id" from tables "downloads", "extras", "labelfiles", and "magfiles".

This column currently stores the data format of each file. For instance, if it's a +3 disk image, then it contains the ID for "+3 disk (DSK)". If it's a PDF, it contains the ID for "Document (PDF)". And so on...

The problem is, this information is completely redundant. Just looking at the filename extension will give you exactly the same information. All +3 disk files have extension ".dsk.zip", all PDF documents have extension ".pdf", all TXT documents have extension ".txt", etc. Sometimes the people contributing with ZXDB updates have made mistakes filling this information, so it doesn't match the actual file type. Therefore this is a field that's unnecessary at best, plain wrong at worst.

There's now a new table called "extensions", that store an explanation for each file extension supported by ZXDB. Websites based on ZXDB can simply check the data formats themselves. If anyone needs help with SQL changes here (or anything else really), please let me know!

The main file extensions supported by ZXDB (as detailed in table "extensions") are listed below:

Code: Select all

.avi.zip            Video (AVI)
.ay.zip             Music (AY)
.d80.zip            Didaktik disk (D80)
.dck.zip            Timex expansion (DCK)
.dsk.zip            +3 disk (DSK)
.fdi.zip            TR-DOS disk (FDI)
.gif                Picture (GIF)
.html               Webpage (HTML)
.ifl                Screen dump 8x2 (IFL)
.img.zip            Disciple/+D disk (IMG)
.jpg                Picture (JPG)
.mdr.zip            Microdrive cartridge (MDR)
.mgt.zip            Disciple/+D disk (MGT)
.mid.zip            Music (MIDI)
.mlt                Screen dump 8x1 (MLT)
.mp3.zip            Music (MP3)
.mpg.zip            Video (MPG)
.opd.zip            Opus Discovery disk (OPD)
.p.zip              ZX81 tape (P)
.pdf                Document (PDF)
.png                Picture (PNG)
.pok                Pokes (POK)
.rom.zip            ROM image dump (ROM)
.rzx.zip            Game recording (RZX)
.scl.zip            TR-DOS disk (SCL)
.scr                Screen dump (SCR)
.slt.zip            Snapshot (SLT)
.sna.zip            Snapshot (SNA)
.szx.zip            Snapshot (SZX)
.tap.zip            Tape (TAP)
.trd.zip            TR-DOS disk (TRD)
.txt                Document (TXT)
.tzx.zip            Perfect tape (TZX)
.wdr.zip            Wafadrive wafer disk (WDR)
.wmv.zip            Video (WMV)
.z80.zip            Snapshot (Z80)
.z81.zip            ZX81 tape (81)
.zxk                ZX Vega keymap (ZXK)
_Fonts.zip          Fonts package
_SourceCode.zip     Source code package
This modification will happen gradually. For now, both formattypes 54 (document ".pdf") and 55 (document ".txt") will be removed in the next ZXDB update. OK?
User avatar
PeterJ
Site Admin
Posts: 6854
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: New Database Model ZXDB

Post by PeterJ »

Thanks [mention]Einar Saukas[/mention]

Sounds good to me.
User avatar
pavero
Dynamite Dan
Posts: 1569
Joined: Sat Dec 09, 2017 11:49 pm
Location: The Czech Republic
Contact:

Re: New Database Model ZXDB

Post by pavero »

I agree, this information was really redundant.

But big number of PHP scripts as SC site will be probably complete nonfunctional after removing formattype_id attribute from ZXDB.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

pavero wrote: Fri May 17, 2019 7:13 pm I agree, this information was really redundant.

But big number of PHP scripts as SC site will be probably complete nonfunctional after removing formattype_id attribute from ZXDB.
That's the reason I always announce these changes plenty of time in advance, implement changes as gradually as possible, and wait for feedback so site owners can let me know if there are any other issues I need to consider.

This way, even if sites like SC forget to make the required changes, they will probably just stop showing a text file icon or something similar. I doubt this would break anything important.
User avatar
PeterJ
Site Admin
Posts: 6854
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: New Database Model ZXDB

Post by PeterJ »

This is especially important as ZXDB gets used in more and more projects.
User avatar
pavero
Dynamite Dan
Posts: 1569
Joined: Sat Dec 09, 2017 11:49 pm
Location: The Czech Republic
Contact:

Re: New Database Model ZXDB

Post by pavero »

Einar Saukas wrote: Fri May 17, 2019 7:51 pm This way, even if sites like SC forget to make the required changes, they will probably just stop showing a text file icon or something similar. I doubt this would break anything important.
I can't agree. Many SQL queries will be invalid and the page won't be loaded at all. Especially formattype_id in downloads table is critical.

Maybe some backward compatibility should be considered for the future ...
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

pavero wrote: Fri May 17, 2019 10:13 pmI can't agree. Many SQL queries will be invalid and the page won't be loaded at all. Especially formattype_id in downloads table is critical.
That could happen if I was simply dropping column "formattype_id" right away. But that's not the plan that I presented. Instead, this change will happen gradually.

In the next ZXDB update, this column will still exist, but it won't be populated anymore for formattypes 54 (document ".pdf") and 55 (document ".txt"). That's all. This change won't invalidate current SQL queries or break any pages.

Afterwards, I will let everybody know in advance about the next couple formattype values that won't be populated in the following ZXDB release. And so on.

Each website can choose to remove all references to formattypes at once. Or make the changes gradually as I announce them. Or simply leave it for now and deal with everything later when they have more free time, since the impact will be minimal for now.

pavero wrote: Fri May 17, 2019 10:13 pmMaybe some backward compatibility should be considered for the future ...
That's exactly what's being already considered!

We cannot stop improving ZXDB, just because many projects are using it. However every change we make is always planned carefully, announced in advance, explained in detail, and implemented gradually, in order to make it as easy as possible for others to adapt accordingly. That's the best trade-off possible between evolution and backward compatibility IMHO.

In this particular case, simply removing a redundant column is going to take several ZXDB releases, in order to make it easier for everybody else. I'm not in a hurry :)
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

A new ZXDB update is available!

As usual, it contains several additional titles and hi-res inlays. Special thanks to [mention]R-Tape[/mention], [mention]druellan[/mention], and [mention]pavero[/mention] for their contributions :)
User avatar
StooB
Dynamite Dan
Posts: 1073
Joined: Mon Oct 22, 2018 11:03 am
Contact:

Re: New Database Model ZXDB

Post by StooB »

Einar Saukas wrote: Thu May 23, 2019 5:01 am A new ZXDB update is available!

As usual, it contains several additional titles and hi-res inlays. Special thanks to @R-Tape, @druellan, and @pavero for their contributions :)
Always good to have an update, but I've noticed that the corrections in the Little Bugs thread are being applied very sporadically, or not at all. These reports are now six months old. Is ZXDB suffering from the problems seen in a lot of open-source projects, where only shiny new things are of any interest, and there's no impetus to fix the old issues?

As someone stated in the bugs thread:
richl wrote: Tue Dec 18, 2018 2:30 pm It's a shame that there's probably a ton of other fixes that people have submitted over on the WOS forums during the post-handover period that'll eventually get lost as and when it finally sinks into the mire but whenever I see anything that I vaguely recall posting something about I try and post it here now instead. We should have some sort of Christmas appeal for people to resubmit info here and try and educate others to not waste their time posting on WOS. It's a shame as people genuinely want to help but since no-one other than a blurred circle knows what the Big Secret Plan™ is you'd be better off sending corrections to Santa.... or Einar as I think he now likes to be called ;)
User avatar
djnzx48
Manic Miner
Posts: 729
Joined: Wed Dec 06, 2017 2:13 am
Location: New Zealand

Re: New Database Model ZXDB

Post by djnzx48 »

I was sort of wondering the same thing about these bugfixes in general. I know ZXDB doesn't deal with loading screens (or does it?), but there was a report made about a wrong loading screen over a year ago and nothing happened since then ;)

I guess the problem is related to what was brought up in this thread about marking ZXDB posts, in that you don't actually know what happens with these fixes, whether they get noticed or just left to be forgotten in the depths of some old thread.
Ralf
Rick Dangerous
Posts: 2279
Joined: Mon Nov 13, 2017 11:59 am
Location: Poland

Re: New Database Model ZXDB

Post by Ralf »

I suppose admins have right to edit every post. It would be wrong to overuse this option but in this case it could be justified to make a small edit. Add "Fixed" or something like that in the bottom of the original post. It could work without any changes to the forum engine, I guess.
User avatar
djnzx48
Manic Miner
Posts: 729
Joined: Wed Dec 06, 2017 2:13 am
Location: New Zealand

Re: New Database Model ZXDB

Post by djnzx48 »

How about just changing the subject line of the post? It might be safer than modifying the post's contents.
User avatar
druellan
Dynamite Dan
Posts: 1466
Joined: Tue Apr 03, 2018 7:19 pm

Re: New Database Model ZXDB

Post by druellan »

StooB wrote: Thu May 23, 2019 9:36 am Always good to have an update, but I've noticed that the corrections in the Little Bugs thread are being applied very sporadically, or not at all. These reports are now six months old. Is ZXDB suffering from the problems seen in a lot of open-source projects, where only shiny new things are of any interest, and there's no impetus to fix the old issues?
I'm into that, trying to implement as much as I can of those fixes, in fact, I usually "like" the posts I'm working on, so, you can follow my progress on the "little bugs" thread (both 1 and 2). BUT! I must admit I have some problems with a couple of yours, so, expect a reply about them, soon :)

For now, I'm walking back the old thread, validating the fixes and applying everything I can. After that I think could be a good idea to repost fixes that are still not resolved on the new thread, but first give me some time to go back and check everything I can.
User avatar
PeterJ
Site Admin
Posts: 6854
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: New Database Model ZXDB

Post by PeterJ »

Thank you again to [mention]druellan[/mention], [mention]Einar Saukas[/mention], [mention]R-Tape[/mention], [mention]pavero[/mention] and all the others who spend their free time on ZXDB. Also thanks to all of you submitting little bugs

I'm not sure what others think, but I always imagined the WoS database was pretty accurate, and I never expected the number of issues and corrections to surface.

Please be patient, and rest assured the team are doing their very best.

Regarding the marking of changes as 'done' let me speak to the others, and I'm sure we can find a solution.
User avatar
Stefan
Manic Miner
Posts: 792
Joined: Mon Nov 13, 2017 9:51 pm
Location: Belgium
Contact:

Re: New Database Model ZXDB

Post by Stefan »

Just thinking about the (lack of) relationship between issues found and fixes.

No point reinventing the wheel of an issue tracker.

Could zxdb be "hosted" on github with each record being a json file or so?
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

Stefan wrote: Thu May 23, 2019 9:09 pm Just thinking about the (lack of) relationship between issues found and fixes.

No point reinventing the wheel of an issue tracker.

Could zxdb be "hosted" on github with each record being a json file or so?
ZXDB is already hosted on Github:

https://github.com/zxdb/ZXDB

I mentioned it at the beginning of this thread :)

I agree it would be more appropriate to manage reported bugs using an issue tracker. However I very much doubt users here would bother to visit it regularly. The advantage of posting bug reports directly in this forum is that more people can see and help validate (or dispute) each report.

The main reason we have so many bugs still pending is that, for a long time, I was the only person working on reported issues. For instance, I have added most missing cover artist credits reported in this thread until recently. However I had to split my free time with other activities (implementing ZX-Ed, improving ZXDB structure, organizing releases, etc) so I was lagging behind. Fortunately [mention]druellan[/mention] recently joined the ZXDB team specifically to work on these issues. His main priority is processing reported bugs and he's fixing them faster than people are reporting new ones. There's a huge backlog to process so it's going to take a while, but at current rate he's going to catch up eventually. His contributions are not as visible as new titles added by [mention]R-Tape[/mention] but he's doing a great job, we just need to give him more time!
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

djnzx48 wrote: Thu May 23, 2019 10:01 amI was sort of wondering the same thing about these bugfixes in general. I know ZXDB doesn't deal with loading screens (or does it?), but there was a report made about a wrong loading screen over a year ago and nothing happened since then ;)
This particular case is tricky, because the loading screen is wrong but fixing it would make it look worse... Frankly we have been prioritizing other "easier" fixes for now while we give this case more thought.

djnzx48 wrote: Thu May 23, 2019 10:01 amI guess the problem is related to what was brought up in this thread about marking ZXDB posts, in that you don't actually know what happens with these fixes, whether they get noticed or just left to be forgotten in the depths of some old thread.
Yes, that's part of the problem. [mention]druellan[/mention] is currently marking processed fixes with "likes" but they are hardly visible. Ideally we should have a more visible method to identify fixed issues, so everyone can notice our progress and validate if the fixes were applied correctly. Also it will make our work more productive. However I don't really like my own proposal to edit other people's posts... So I'm not sure what to do about that.
User avatar
Stefan
Manic Miner
Posts: 792
Joined: Mon Nov 13, 2017 9:51 pm
Location: Belgium
Contact:

Re: New Database Model ZXDB

Post by Stefan »

Einar Saukas wrote: Thu May 23, 2019 10:18 pm ZXDB is already hosted on Github:

https://github.com/zxdb/ZXDB

I mentioned it at the beginning of this thread :)
One SQL file containing all definitions and data is indeed "hosted", but also unmanageable.

I was thinking more something along the lines of a json / sql file per entry allowing direct linking between issues and "sources". On the other hand abusing github as a database and putting 100,000 (?) files into a repository may be a bit over the top (and may violate some github fair use policy)

But, whatever is easiest for the contributors is best.
User avatar
StooB
Dynamite Dan
Posts: 1073
Joined: Mon Oct 22, 2018 11:03 am
Contact:

Re: New Database Model ZXDB

Post by StooB »

Einar Saukas wrote: Thu May 23, 2019 11:12 pm Yes, that's part of the problem. @druellan is currently marking processed fixes with "likes" but they are hardly visible. Ideally we should have a more visible method to identify fixed issues, so everyone can notice our progress and validate if the fixes were applied correctly. Also it will make our work more productive. However I don't really like my own proposal to edit other people's posts... So I'm not sure what to do about that.
Isn't the easiest solution to have a Little Bugs sub-forum with a separate post for each bug or group of related bugs? You could track fixes by either adding prefixes like "FIXED" to post titles, or alternatively have further sub-forums for open/fixed/rejected issues and move posts between them.
User avatar
R-Tape
Site Admin
Posts: 6353
Joined: Thu Nov 09, 2017 11:46 am

Re: New Database Model ZXDB

Post by R-Tape »

StooB wrote: Fri May 24, 2019 7:21 am Isn't the easiest solution to have a Little Bugs sub-forum with a separate post for each bug or group of related bugs? You could track fixes by either adding prefixes like "FIXED" to post titles, or alternatively have further sub-forums for open/fixed/rejected issues and move posts between them.
That's going to be a lot of threads bumping up and down. Someone would still have to try and keep on top of things—it's a lot of work.

Personally I think bugs1, bugs2 etc can work well, as things are put into manageable chunks that can be blitzed when someone has time. The whole thread can be locked when it gets big enough.

As the main person currently doing corrections at the moment, what do you prefer [mention]druellan[/mention] ? Would it be useful to have a volunteer (perhaps [mention]StooB[/mention] ?) to keep on top of a finalised corrections spreadsheet, that is basically a list of ZXDB ID and a concise explanation of the correction needed? (Or would you rather be left alone to get on with it?!)
hikoki
Manic Miner
Posts: 576
Joined: Thu Nov 16, 2017 10:54 am

Re: New Database Model ZXDB

Post by hikoki »

Did you know google spreadsheets are nicely displayed in this forum? try to make one and post the shared link in here
User avatar
StooB
Dynamite Dan
Posts: 1073
Joined: Mon Oct 22, 2018 11:03 am
Contact:

Re: New Database Model ZXDB

Post by StooB »

R-Tape wrote: Fri May 24, 2019 9:57 am
That's going to be a lot of threads bumping up and down. Someone would still have to try and keep on top of things—it's a lot of work.
It's the same number of bugs whether they're in separate threads or in a couple of big threads. So it's a lot of work either way!

R-Tape wrote: Fri May 24, 2019 9:57 am Personally I think bugs1, bugs2 etc can work well, as things are put into manageable chunks that can be blitzed when someone has time. The whole thread can be locked when it gets big enough.
If you lock the threads then how does [mention]durellan[/mention], or anyone else for that matter, respond to any of the bugs? You'd end up with a situation where you could have a reply in the 5th bugs thread which is about a bug posted in the 1st bug thread. There's still the problem of marking bugs as fixed more visibly, without liking or editing.
User avatar
StooB
Dynamite Dan
Posts: 1073
Joined: Mon Oct 22, 2018 11:03 am
Contact:

Re: New Database Model ZXDB

Post by StooB »

hikoki wrote: Fri May 24, 2019 11:13 am Did you know google spreadsheets are nicely displayed in this forum? try to make one and post the shared link in here
Here's one. It doesn't seem to embed - perhaps there's a setting needs to be changed?

https://docs.google.com/spreadsheets/d/ ... ED/pubhtml
Post Reply