New Database Model ZXDB

This is the place for general discussion and updates about the ZXDB Database. This forum is not specific to Spectrum Computing.

Moderator: druellan

Post Reply
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

hikoki wrote: Thu Jun 20, 2019 1:29 pm @Einar Saukas you are right in this case as footers don't seem to be OCRed
well, you can automatically detect the lack of footers by crawling mags with different URLs. If your search term doesn't return one single result then provide just the estimated /page url
I wouldn't mind storing in ZXDB a different "search term" for each magazine issue, if this was a reliable solution. But apparently it's not. I don't like the idea of adopting a solution that only works sometimes.

Therefore I think the best option we have is storing the "cover page number" for each magazine issue. It seems this approach will always work, except when there's an error in the PDF itself (but then we can work to fix the PDF instead).
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

djnzx48 wrote: Thu Jun 20, 2019 4:00 am
Einar Saukas wrote: Wed Jun 19, 2019 7:25 pm Yes, that's a problem. And it's something that needs to be fixed in ZXDB, not SC.
OK, I didn't know whether the links were stored in the database or autogenerated on the website. But are you sure the problem isn't just a simple off-by-one error? Having a brief look, all the magazines I've seen so far have the correct page numbers on archive.org, except for:

PCN: the numbering starts at the contents page rather than the cover. So the archive.org numbers are two pages ahead.

The Spectrum Show Magazine (issues #0 and #1 only): the numbering starts on the next page after the cover. So the archive.org numbers are one page ahead.

In the archive.org URLs, #page/n1 refers to the second page of the magazine, #page/n2 refers to the third, and so on. For the title page, this parameter is simply omitted.
Thanks a lot!!!

I will include this information in the next ZXDB update, afterwards I can help Peter incorporate this logic into SC.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

BTW table "formattypes" and columns "formattype_id" will be removed in the next ZXDB update.

This table is now empty and these columns only contain NULL, so this change shouldn't affect anybody.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

PeterJ wrote: Wed Jun 19, 2019 9:27 pm Thanks for the great explanation @Einar Saukas
You are welcome!

PeterJ wrote: Wed Jun 19, 2019 9:27 pm When I go to our MicroMart Page (I didn't know that was included in ZXDB).

https://spectrumcomputing.co.uk/index.p ... mag_id=280

I click on any PDF and it says Page not Found on archive.org. Is that something with the path? Every PDF link seems to be trying to go to:

https://archive.org/download/Micro-Mart ... pecial.pdf
It seems to be working now :)
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

There's a new ZXDB update already!

Special thanks to [mention]druellan[/mention] for bug fixes, [mention]pavero[/mention] for additional files, and [mention]djnz48[/mention] for magazine numbering information!
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

And another ZXDB update is available!

This update includes mostly new titles (thanks [mention]R-Tape[/mention]!) and hires inlays (thanks [mention]pavero[/mention]!)
User avatar
Juan F. Ramirez
Bugaboo
Posts: 5102
Joined: Tue Nov 14, 2017 6:55 am
Location: Málaga, Spain

Re: New Database Model ZXDB

Post by Juan F. Ramirez »

Einar Saukas wrote: Wed Jul 03, 2019 5:05 am This update includes mostly new titles (thanks @R-Tape!) and hires inlays (thanks @pavero!)
As to the inlays... are they the ones mentioned here?
viewtopic.php?f=29&t=1661#p23769
User avatar
pavero
Dynamite Dan
Posts: 1569
Joined: Sat Dec 09, 2017 11:49 pm
Location: The Czech Republic
Contact:

Re: New Database Model ZXDB

Post by pavero »

Juan F. Ramirez wrote: Wed Jul 03, 2019 3:33 pm
Einar Saukas wrote: Wed Jul 03, 2019 5:05 am This update includes mostly new titles (thanks @R-Tape!) and hires inlays (thanks @pavero!)
As to the inlays... are they the ones mentioned here?
viewtopic.php?f=29&t=1661#p23769
No, [mention]PeterJ[/mention] must update SC site firstly.
User avatar
PeterJ
Site Admin
Posts: 6855
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: New Database Model ZXDB

Post by PeterJ »

All done [mention]pavero[/mention] & [mention]Juan F. Ramirez[/mention]
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

Another ZXDB version is available!

This update features integration with JSW Central (thanks [mention]jetsetdanny[/mention]!), hires inlays (thanks [mention]pavero[/mention]!) and more bugfixes (thanks [mention]druellan[/mention]!)
jetsetdanny
Dizzy
Posts: 85
Joined: Thu May 02, 2019 10:22 pm
Contact:

Re: New Database Model ZXDB

Post by jetsetdanny »

Thanks a lot, Einar! ZXDB's integration with JSW Central is greatly appreciated! :)
Website: JSW Central
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

jetsetdanny wrote: Sun Jul 14, 2019 6:18 am Thanks a lot, Einar! ZXDB's integration with JSW Central is greatly appreciated! :)
You are welcome!
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

Yet another ZXDB version is now available!

This update includes new titles (thanks to [mention]R-Tape[/mention]!), new Danish magazines "SOFT" and "SOFT Today" (thanks to [mention]kolbeck[/mention]!), and fixed magazine references to New Computer Express (thanks to myself! :) )
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

And another ZXDB version is available!

This update includes more titles (thanks [mention]R-Tape[/mention]!), more hires inlays (thanks [mention]pavero[/mention]!), and more magazine updates (I will describe these later)


IMPORTANT: From now on, the auxiliary tables to help database searches (prefixed with "search_by_") are distributed separately. If you have a website that uses these tables, don't forget to always execute ZXDB_help_search.sql after running the main ZXDB script.
User avatar
R-Tape
Site Admin
Posts: 6353
Joined: Thu Nov 09, 2017 11:46 am

Re: New Database Model ZXDB

Post by R-Tape »

Einar Saukas wrote: Thu Jul 25, 2019 12:56 pm And another ZXDB version is available!
Excellent! We should start counting these, there are 35 updates so far in this thread (I'll look at the other thread later).

EDIT - The other thread has 19 updates, so plus the 35 so far takes the total (from this forum) to erm...54!

There were plenty announced at WoS to include if anyone wanted to count them too.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

I know ZXDB had 42 content updates in the last 15 months, because I have been assigning a version number to each release since it moved to Github.
User avatar
druellan
Dynamite Dan
Posts: 1466
Joined: Tue Apr 03, 2018 7:19 pm

Re: New Database Model ZXDB

Post by druellan »

Einar Saukas wrote: Thu Jul 25, 2019 7:43 pm I know ZXDB had 42 content updates in the last 15 months, because I have been assigning a version number to each release since it moved to Github.
BTW, any reason you are not using the "release" feature on Github, Einar? Since Github has an API to access those, can open the door to automatic or one-click updates on sites using the ZXDB.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

druellan wrote: Fri Jul 26, 2019 1:45 pm
Einar Saukas wrote: Thu Jul 25, 2019 7:43 pm I know ZXDB had 42 content updates in the last 15 months, because I have been assigning a version number to each release since it moved to Github.
BTW, any reason you are not using the "release" feature on Github, Einar? Since Github has an API to access those, can open the door to automatic or one-click updates on sites using the ZXDB.
The reason is, this "release" feature on Github is intended for software releases. The idea is that Github repository stores source code files, and the release section stores distribution packages. I'm using it in ZX-Ed, for instance.

However it's not much adequate for ZXDB, where a distinction between source code and release doesn't make sense. Every update to ZXDB is already a new release on itself. Using the "release" feature for every update, containing exactly the same files, would be simply duplicating information without any practical benefit.

However, if anybody is planning to use Github API for automatic ZXDB updates, please contact me so we can work out the best solution for you.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

There are now 31 magazines in ZXDB indexed as PDF from Archive.org:

48K
ACE (Advanced Computer Entertainment)
Big K
Computer Input
Crash
El Ordenador Personal
Input
Jogos 80
Magazine ZX
Micro Mart
MicroHobby
MicroHobby-BR
Mundo Spectrum
New Computer Express
Personal Computer News
Planeta Sinclair Almanaque
Retro Maniac
RetroMagazine
Sinclair User
SOFT
SOFT Today
SoftSpectrum
Spectrum Today-EN
The Spectrum Show
Timex Sinclair User
Todospectrum
Your Sinclair
ZX
ZX Spectrum Gamer
ZXShed
ZXShed Reloaded

It means you can now view these magazine issues online from Archive.org, or download it as PDF.

Now it should also open the right page, when you click on a magazine reference to some specific page number (as discussed here). If anything doesn't work as expected, please let me know!
User avatar
Juan F. Ramirez
Bugaboo
Posts: 5102
Joined: Tue Nov 14, 2017 6:55 am
Location: Málaga, Spain

Re: New Database Model ZXDB

Post by Juan F. Ramirez »

Hi [mention]Einar Saukas[/mention] , it seems there's a problem in the Microhobby folder.

Whenever I click on [PDF] works ok (the PDF file is shown) but if I click on [VIEW] a new archive.org screen appears with the next error message:

'The item you have requested has a problem with one or more of the metadata files that describe it, which prevents us from displaying this page'

There's no similar problem with Sinclair User or Crash, everything is OK.

Besides, in the list of Microhobby issues, before #1 issue some random issues appear (with no corresponding picture):

1985/10
1987/3
1987/7
1991/4
1991/5
1991/12

I'm using Windows 10 & Chrome.

Hope it helps.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

Juan F. Ramirez wrote: Fri Jul 26, 2019 6:42 pm Hi @Einar Saukas , it seems there's a problem in the Microhobby folder.

Whenever I click on [PDF] works ok (the PDF file is shown) but if I click on [VIEW] a new archive.org screen appears with the next error message:

'The item you have requested has a problem with one or more of the metadata files that describe it, which prevents us from displaying this page'
That's a problem at Archive.org. I will try to find a different source. Thanks for reporting it!

Juan F. Ramirez wrote: Fri Jul 26, 2019 6:42 pm Besides, in the list of Microhobby issues, before #1 issue some random issues appear (with no corresponding picture):

1985/10
1987/3
1987/7
1991/4
1991/5
1991/12
These are "lost issues".

For instance, according to Martijn's WoS there was a reference to game "Combinatoria" in MicroHobby year 1985 month 10 page 9. The problem is, MicroHobby was a weekly magazine, therefore several issues of this magazine were published in October 1985. Which one of them contained this reference?

I have fixed literally thousands of these reference problems in ZXDB, but I didn't notice MicroHobby still had a few problems left. I will fix them all, thanks for reporting it!
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

A new ZXDB version is available!

This update includes new titles (thanks [mention]R-Tape[/mention]!), new hi-res inlays (thanks [mention]pavero[/mention]!), new magazines, updated integration with RZX Archive (thanks Daren!) and ZXSR (thanks Chris!).
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

I'm planning to remove column "is_electronic" from table "magazines" in the next ZXDB update.

This column is supposed to distinguish between paper and electronic magazines. However it's more informative to simply obtain this information directly from the magazine format: PDF for paper magazines, TAP or TZX for tape magazines, MGT or DISK for disk magazines, etc. In particular, I know SpectrumComputing is already doing it.

Moreover, I had identified a few cases where column "is_electronic" conflicted with the magazine format. In all such cases, "is_electronic" was wrong. I could have fixed these values, but there's no reason to keep a column that's redundant anyway.

I'm announcing this change in advance, to give site owners the chance to let me know if this change would affect them.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

I'm also planning to remove columns "link" and "link2" from table "magrefs" in the next ZXDB update.

This information will be moved to a new table, because it's technically a one-to-many relationship.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

A new ZXDB version is already available!

This update contains lots of updated information about magazines. I will talk about them in further detail after SC is updated with this new version, because it will be easier to describe these changes when users are able to see them in practice. :)
Post Reply