Download area cannot be listed anymore

[duck-rh]

Quack,

I understand you're doing that on purpose but that's causing problems to update the Debian package. It's not possible anymore to monitor newly available versions and I am not going to do that manually. I considered using the Github repo instead but the releases are not signed. Could you consider publishing the signature file alongside the tarball on Github?

Regards.
\_o<

[mboelen]

Hi Marc,

How does the Debian check work to see if there is an update? Or in other words, what does it need to detect it?

[duck-rh]

Quack Michael,

The tool checks a specific page (currently https://downloads.cisofy.com/lynis/), detect the links in the page and filter them to get the various versions available, get the associated asc files too. The tool which is called uscan is used to grab the tarball securely and check the signature and also leveraged in an automated way in a Debian service to detect new versions to package.

It would be possible to use the Github release page instead but the asc files are missing and the signature check would not be possible anymore.

\_o<

[mboelen]

Thanks, I did not know how Debian do this check. FreeBSD has a similar tool (portswatch) and they simply try to find a new version, instead of reading the directory listing.

I have reverted a few things, so that the directory index is available for the newer packages. I might make a small change soon to look things a bit prettier, but then the directory structure for the main path should remain.

Is the type of listing that uscan scans something you manage as a package maintainer in the watch file?
Can you trigger a new check to see that things are working again?

[duck-rh]

It can read from any page, not just directory listings I think. I wonder how FreeBSD do it.

I can parametrize the URL and pattern matching but there is not listing type. It seems any kind of http or ftp URL is fine.

I cannot trigger a check but I can run the tool locally and it works again now, thanks:

$ uscan --report --verbose
uscan info: uscan (version 2.23.7) See uscan(1) for help
uscan info: Scan watch files in .
uscan info: Check debian/watch and debian/changelog in .
uscan info: package="lynis" version="3.0.9-1" (as seen in debian/changelog)
uscan info: package="lynis" version="3.0.9" (no epoch/revision)
uscan info: ./debian/changelog sets package="lynis" version="3.0.9"
uscan info: Found upstream signing keyring: debian/upstream/signing-key.asc
uscan info: Process watch file at: debian/watch
    package = lynis
    version = 3.0.9
    pkg_dir = .
uscan info: opts: pgpsigurlmangle=s/$/.asc/
uscan info: line: https://downloads.cisofy.com/lynis/ lynis(?:[-_]?[Vv]?(\d[\-+\.:\~\da-zA-Z]*))(?i)(?:\.(?:tar\.xz|tar\.bz2|tar\.gz|tar\.zstd?|zip|tgz|tbz|txz)) debian uupdate
uscan info: Parsing pgpsigurlmangle=s/$/.asc/
uscan info: line: https://downloads.cisofy.com/lynis/ lynis(?:[-_]?[Vv]?(\d[\-+\.:\~\da-zA-Z]*))(?i)(?:\.(?:tar\.xz|tar\.bz2|tar\.gz|tar\.zstd?|zip|tgz|tbz|txz)) debian uupdate
uscan info: Last orig.tar.* tarball version (from debian/changelog): 3.0.9
uscan info: Last orig.tar.* tarball version (dversionmangled): 3.0.9
uscan info: Requesting URL:
   https://downloads.cisofy.com/lynis/
uscan info: Matching pattern:
   (?:(?:https://downloads.cisofy.com)?\/lynis\/)?lynis(?:[-_]?[Vv]?(\d[\-+\.:\~\da-zA-Z]*))(?i)(?:\.(?:tar\.xz|tar\.bz2|tar\.gz|tar\.zstd?|zip|tgz|tbz|txz))
uscan info: Found the following matching hrefs on the web page (newest first):
   https://downloads.cisofy.com/lynis/lynis-3.1.1.tar.gz (3.1.1) index=3.1.1-1 
   https://downloads.cisofy.com/lynis/lynis-3.1.1.tar.gz (3.1.1) index=3.1.1-1 
   https://downloads.cisofy.com/lynis/lynis-3.1.0.tar.gz (3.1.0) index=3.1.0-1 
   https://downloads.cisofy.com/lynis/lynis-3.1.0.tar.gz (3.1.0) index=3.1.0-1 
uscan info: Looking at $base = https://downloads.cisofy.com/lynis/ with
    $filepattern = lynis(?:[-_]?[Vv]?(\d[\-+\.:\~\da-zA-Z]*))(?i)(?:\.(?:tar\.xz|tar\.bz2|tar\.gz|tar\.zstd?|zip|tgz|tbz|txz)) found
    $newfile     = https://downloads.cisofy.com/lynis/lynis-3.1.1.tar.gz
    $newversion  = 3.1.1
    $lastversion = 3.0.9
uscan info: Matching target for downloadurlmangle: https://downloads.cisofy.com/lynis/lynis-3.1.1.tar.gz
uscan info: Upstream URL(+tag) to download is identified as    https://downloads.cisofy.com/lynis/lynis-3.1.1.tar.gz
uscan info: Filename (filenamemangled) for downloaded file: lynis-3.1.1.tar.gz
Newest version of lynis on remote site is 3.1.1, local version is 3.0.9
 => Newer package available from:
        => https://downloads.cisofy.com/lynis/lynis-3.1.1.tar.gz
uscan info: Scan finished

[mboelen]

Perfect. That is very useful for testing if we alter the page. I was thinking, an option could be the replace the page and only list the link to the latest version (and the signature).

Can you share the definition that does the check? The one that does the check at downloads.cisofy.com. Or if possible, the full configuration file. Then I can use that later to test and confirm that I don't break things ;-)

FreeBSD does it by checking the "next" version to show up. So if 3.1.1 is the current one, they check for 3.1.2, 3.2.0, and 4.0.0.

[mboelen]

As things work now, closing this issue. Feel free to add details if needed.

[duck-rh]

@mboelen uscan reads various package configuration files, not just its own configuration, so you need to clone the whole https://salsa.debian.org/debian/lynis.git repository and then you can run uscan --report --verbose (available in the devscripts package) to test if that works fine.

The check is interesting but in my experience I think that would mean to test de shitload of versions. Sometimes upstream decide to go with 3.1.1.1, or skip a version because of some error during the release steps or because of a last minute fix, or would change the version scheme entirely… I don't think uscan folks would like to implement such approach. Thanks for sharing anyway.

Thanks again for the fix.