Database model ZXDB

This is the place for general discussion and updates about the ZXDB Database. This forum is not specific to Spectrum Computing.

Moderator: druellan

User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: Database model ZXDB

Post by Einar Saukas »

... and more details about this last update were provided here.
User avatar
Alessandro
Dynamite Dan
Posts: 1908
Joined: Wed Nov 15, 2017 11:10 am
Location: Messina, Italy
Contact:

Re: Database model ZXDB

Post by Alessandro »

R-Tape wrote: Tue Feb 13, 2018 11:04 am Apologies: Missed this update but in the next.
[...]
About a hundred file changes for Alessandro :-p
Missed again :evil:
User avatar
R-Tape
Site Admin
Posts: 6353
Joined: Thu Nov 09, 2017 11:46 am

Re: Database model ZXDB

Post by R-Tape »

Alessandro wrote: Sat Feb 24, 2018 1:12 pm Missed again :evil:
Not entirely, these have happened:
Put LIMS v1 into a .zip as historical dev material
Put Apulija13 v1 into a .zip as historical dev material
Sort out 'The Trunk'
These may have happened (I've lost track of all the new versions):
Update SCL,TRD, PDF for Sophia.
ADB: remove the TRD and PDF, add the new TRD,MGT,SCL, PDF
These definitely have not happened:
ADB: remove the Tape images
Remove LIMS TRD files.
Sorry Ale, we'll get it sorted but it may be a few days now :(
User avatar
R-Tape
Site Admin
Posts: 6353
Joined: Thu Nov 09, 2017 11:46 am

Re: Database model ZXDB

Post by R-Tape »

ADB: remove the Tape images
Remove LIMS TRD files.
This is now sorted (ta Peter), so just the disk updates to check.
User avatar
Alessandro
Dynamite Dan
Posts: 1908
Joined: Wed Nov 15, 2017 11:10 am
Location: Messina, Italy
Contact:

Re: Database model ZXDB

Post by Alessandro »

Thanks for replying, I understand synchronizing the database is still a huge task. Will wait some more time then :)
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: Database model ZXDB

Post by Einar Saukas »

A new ZXDB update is now available here.

Hopefully this time we got right the dozens of files that Alessandro wants us to replace every couple weeks! :)
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: Database model ZXDB

Post by Einar Saukas »

A new ZXDB update is available here!

It contains new games, playable files and screenshots (thanks R-Tape and all contributors!), lots of new ZX81 information (thanks Thomas and Neil Parsons!), new RZX Archive content (thanks Daren!), and probably lots more content I forgot to mention! :)
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: Database model ZXDB

Post by Einar Saukas »

Another ZXDB update is available here!

It contains lots of stuff, that R-Tape will describe later today :)
User avatar
PeterJ
Site Admin
Posts: 6854
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: Database model ZXDB

Post by PeterJ »

Thanks [mention]Einar Saukas[/mention] I will get the site updated this evening.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: Database model ZXDB

Post by Einar Saukas »

A huge ZXDB update is now available here!

This time, it contains over a thousand new titles and over a thousand new files. I will let R-Tape try to figure out and describe everything that has changed :)
Ralf
Rick Dangerous
Posts: 2279
Joined: Mon Nov 13, 2017 11:59 am
Location: Poland

Re: Database model ZXDB

Post by Ralf »

This time, it contains over a thousand new titles and over a thousand new files.
What? Where did you get 1000 missing titles? :o :)
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: Database model ZXDB

Post by Einar Saukas »

Ralf wrote: Wed Mar 28, 2018 9:42 amWhat? Where did you get 1000 missing titles? :o :)
Peter just posted a detailed list of changes here. We have lots of people collaborating with ZXDB content now, so we are gathering new content faster than ever!

A few numbers:

* Exactly 5,108 new titles are now catalogued in ZXDB (including all available information about authors, publishers, etc). The original WoS archive catalogued 32,018 titles, therefore ZXDB titles increased 19%.

* Exactly 8,457 new files are now indexed in ZXDB. The original WoS archive indexed 96,126 files, therefore ZXDB files increased 8.8%.

* Exactly 1,650 author roles (programmer, artist, musician, etc) are now identified in ZXDB. The original WoS archive didn't distinguish any author roles.

* Exactly 19,484 new external references were added to ZXDB. The original WoS archive had 7,279 external references (1,009 Wikipedia references, 4,893 Tipshop references, 222 Lost in Translation references, and 1,155 Freebase references). ZXDB added 10,394 ZXSR references, 3,255 MobyGames references, 2,233 CASA references, 3,027 RZX Archive references, 651 ZX81Stuff references, 1,975 RZX YouTube Channel references, 107 ZX81 Youtube references, and a few others. Therefore integration with other websites is about 3.7 times larger than the original WoS archive. In practice, it means whenever you look at your favorite games at Spectrum Computing or ZX Info, you get many links to check additional information about the game almost everywhere else (including links to the old WoS archive).

Moreover, about 10 different Spectrum sites now support the same IDs from ZXDB (such as ZX-Art and Speccy Screenshot Maps), thus making it much easier to have automatic integration between all of them. There's also the ZX Pokemaster project (by Helga) that associates the entire TOSEC archive to their corresponding ZXDB entries, and the ZXDB API (by Thomas) that allows direct access to ZXDB from everywhere else.

I never thought this project would get so big! :)
User avatar
kolbeck
Manic Miner
Posts: 309
Joined: Mon Nov 13, 2017 9:04 pm

Re: Database model ZXDB

Post by kolbeck »

Einar Saukas wrote: Thu Mar 29, 2018 5:58 am I never thought this project would get so big! :)
It's big and beautiful :lol:

Thank you all for making ZXDB probably the best source for Sinclair related information.

I know you mentioned elsewhere, but it is also worth to mention that ZXDB has become a fab source for ZX81 as well - collection far more information about ZX81 releases, than seen before - including new releases, proper ZXPand and Chroma interface tagging, lots of screenshots, Youtube links etc.

/T
https://api.zxinfo.dk/v3/ - ZXDB API for developers
zxinfo-file-browser - Cross platform app to manage your files
https://zxinfo.dk - another ZXDB frontend
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: Database model ZXDB

Post by Einar Saukas »

Good point!

Next ZXDB update will have information about more than 1,700 programs for the ZX81. Over half of this data was catalogued by [mention]kolbeck[/mention] alone :)
User avatar
PeterJ
Site Admin
Posts: 6854
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: Database model ZXDB

Post by PeterJ »

Crikey [mention]R-Tape[/mention] the next update page will need to be serialised across 10 volumes!
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: Database model ZXDB

Post by Einar Saukas »

PeterJ wrote: Fri Mar 30, 2018 5:03 pm Crikey @R-Tape the next update page will need to be serialised across 10 volumes!
No! I just mean next ZXDB update will reach over 1,700 ZX81 titles in total. It already has over 1,600 ZX81 titles right now.
User avatar
PeterJ
Site Admin
Posts: 6854
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: Database model ZXDB

Post by PeterJ »

Phew.... Stand done [mention]R-Tape[/mention]
Nomad
Manic Miner
Posts: 600
Joined: Thu Dec 28, 2017 12:38 pm

Re: Database model ZXDB

Post by Nomad »

I think its a minority view, but I look forward to the potential analytics from all these new programs. If they could be archived in a way that the code could be analyzed it could answer so many interesting questions about how people created software.

Especially stuff like the utilities/tools so little was written about them in the magazines compared to the games.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: Database model ZXDB

Post by Einar Saukas »

This is a fairly technical post, targeted at people using ZXDB database in their websites or importing data directly from it. Everybody else should ignore this post! :)

I'm planning a few changes in database schema for the next ZXDB update. The main reasons are:

* Improve ZXDB model, by removing redundant or obsolete fields. Now that all relevant inconsistencies from original WoS archive were sorted out in ZXDB, we can finally afford to adopt a more strict model, that should help prevent most inconsistencies from ever happening again.

* Help integration with other databases and systems, by ensuring ZXDB tables use meaningful natural keys as PK whenever possible. This should induce any external references to ZXDB data to be more meaninful too, thus helping to better identify what exactly has changed in ZXDB in case of incremental imports.

The main changes in ZXDB will be:

* Table DOWNLOADS: Remove meaningless column ID. It will use natural key (ENTRY_ID, FILE_LINK) as PK.

* Table LABELFILES: Remove meaningless column ID. It will use natural key (LABEL_ID, FILE_LINK) as PK.

* Table MAGFILES: Remove meaningless column ID. It will use natural key (ISSUE_ID, FILE_LINK) as PK.

* Table LABELS: Remove columns IS_COMPANY and COMPANYTYPE_ID. These are deprecated fields, everybody should be already using LABELTYPE_ID instead.

There will be also changes in a few historical tables that atore deleted and "lost" files from old WoS archive, such as table EXTRAS. I'm sure nobody is using those other tables anyway, therefore nobody should be affected.
User avatar
hitm4n
Manic Miner
Posts: 604
Joined: Fri Nov 17, 2017 12:56 pm

Re: Database model ZXDB

Post by hitm4n »

Absolutely superb work Einar, as always, and a big thank you to you and all those helping you.

I'm paticularly interested in the ZX81 information which is currently growing as this was my first computer (10 years old, 1981). I have some very fond, but hazy (it was so long ago), memories of these early years. I keep checking in on the DB updates, keep seeing whats new and with people contributing to the ZX81 info, links have been appearing to sites which have been great to have a look through.

Only the other day, i managed to find "Hitch Hiker" and was playing it in an emulator. I loved that game.

Out of interest, what is a good zx81 emulator these days? One that i can just throw .p files at, good features and accurate emulation? I;ve got eightyone0.43c at the moment... Anyone?

Cheers.
I don't have anything cool to put here, so i'll just be off now to see a priest with yeast stuck between his teeth and his friend called Keith who's a hairpiece thief...
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: Database model ZXDB

Post by Einar Saukas »

Thanks for the kind words!

Regarding your question about emulators, I created a separate thread to avoid going off-topic here:

viewtopic.php?f=23&t=607
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: Database model ZXDB

Post by Einar Saukas »

Yet another large ZXDB update is out! Download it here.

This time we added entries for almost a thousand electronic magazines. However the corresponding files are not available yet, it will be easier to add them in a separate update.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: Database model ZXDB

Post by Einar Saukas »

Next ZXDB update will have one more change:

* Table ROLES: Remove redundant column AUTHOR_SEQ.

You will notice that table AUTHORS already have natural key PK = (ENTRY_ID, LABEL_ID) instead of previous PK = (ENTRY_ID, AUTHOR_SEQ). Although column AUTHOR_SEQ won't be removed from table AUTHORS, anyone integrating or importing from ZXDB should use the new PK.

Likewise, table ROLES now have PK = (ENTRY_ID, LABEL_ID, ROLETYPE_ID) and FK = (ENTRY_ID, LABEL_ID). They were both based on AUTHOR_SEQ, that will be removed now.

I could have made this change in last ZXDB update, but I'm doing it in 2 steps on purpose (added ROLES.LABEL_ID in previous update and will remove ROLES.AUTHOR_SEQ in next update), to make it easier for other systems to adapt.
User avatar
moroz1999
Manic Miner
Posts: 329
Joined: Fri Mar 30, 2018 9:22 pm

Re: Database model ZXDB

Post by moroz1999 »

I really hope that LABEL_ID would stay the same.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: Database model ZXDB

Post by Einar Saukas »

moroz1999 wrote: Sun Apr 08, 2018 5:26 pm I really hope that LABEL_ID would stay the same.
Yes, it will stay the same.
Locked