lp:~ilkeremrekoc/launchpad

Author: İlker Emre Koç
Author Date: 2025-09-22 15:31:38 UTC

WIP

Author: İlker Emre Koç
Author Date: 2025-09-22 04:16:07 UTC

Upgrade celery

Celery 5 breaks the previous CLI option distribution, we updated them to
fit with the new standard.

Celery 5 also gets rid of celery.task module, which was used in our
tests, we replace that.

We also add both the dependencies and transitive dependencies of celery
5 and upgrade lazr.jobrunner since the celery version range of the
previous version was blocking us.

Author: İlker Emre Koç
Author Date: 2025-09-16 09:02:32 UTC

Add import/export job list endpoints

- Added endpoints for import/export job list endpoints.
- Modified the soss export to have a permission checker.
- Modified export vulnerability job to get the distribution map from
  general vulnerability model file instead.

Author: İlker Emre Koç
Author Date: 2025-09-10 12:44:43 UTC

Remove hardcoded distribution from Export job

- Removed the hardcoded soss distribution.
- Changed the distribution and exporter acquisition from if statements
  to dict.get() access.
- Removed too verbose logs.
- Reduced the scope of the accessed CVEs from the whole table to only
  the ones within any given distribution.

Author: İlker Emre Koç
Author Date: 2025-09-08 11:02:36 UTC

Add an additional memory preserver to missed test

Another test, this time inside the BranchScanJob, was silently
modifying the memory limit, as such, we wrapped it with
"preserve_limit()".

To avoid these memory changes, the preserve_limit context manager first
saves the current RLIMIT_AS resource, which denotes the virtual memory
limit, and then resets it to the saved value after the run has
completed.

Also added more descriptive explanations and reduced the scope of
the error catch within the new context manager.

Author: İlker Emre Koç
Author Date: 2025-09-08 11:02:36 UTC

Add an additional memory preserver to missed test

Another test, this time inside the BranchScanJob, was silently
modifying the memory limit, as such, we wrapped it with
"preserve_limit()".

To avoid these memory changes, the preserve_limit context manager first
saves the current RLIMIT_AS resource, which denotes the virtual memory
limit, and then resets it to the saved value after the run has
completed.

Also added more descriptive explanations and reduced the scope of
the error catch within the new context manager.

Author: İlker Emre Koç
Author Date: 2025-09-01 08:51:42 UTC

Change libraian-gc file deletion to autocommit

In pyscopg2, even "SELECT" queries are counted as transactions by
default. And currently, librarian-gc uses autocommit with the value
False. This results in a need to close each transaction manually, which
were logical while we were using a single large query, but since we
seperated the queries into chunks, this results in each query to be
kept open. Thus we change the autocommit value to True so that after
each query we start working on the next query.

Author: İlker Emre Koç
Author Date: 2025-08-26 08:04:37 UTC

Add dry-run to librarian-gc file deletions

To test a change to the librarian-gc in a safe manner, we need a
dry-run to ensure nothing is deleted when we run the file deletion
script.

Author: İlker Emre Koç
Author Date: 2025-08-25 10:06:43 UTC

delete cursor name for multiple query executions

Author: İlker Emre Koç
Author Date: 2025-08-25 06:25:29 UTC

Apply optimization to another part of librarian-gc and clean

Author: İlker Emre Koç
Author Date: 2025-07-02 06:35:59 UTC

Add session to metadata publishing for artifactory

AQL (Artifactory Query Language needs a bearer token for
authentication. To add that and other session parameters to our
connection we used the previous artifactory publishing code.

Then we restored the commented out calls to property publishing
function and tests of it.

Author: İlker Emre Koç
Author Date: 2025-06-16 07:32:59 UTC

export mailing list to the API

Author: İlker Emre Koç
Author Date: 2025-06-10 10:10:10 UTC

comment out the _publish_properties method calls

Currently, calls to artifactory using _publish_properties return 403
errors. Thus, to deploy the rest of the code safely, we will comment
the calls to it until we investigate and fix it.

Author: İlker Emre Koç
Author Date: 2025-06-04 08:03:27 UTC

use publish url to extract root path

To search within artifactory we need a root path to start from.
Originally, I tried getting it from a configuration field, but
decided extracting it from the already provided publishing url
would be the better option.

Author: İlker Emre Koç
Author Date: 2025-06-03 14:03:52 UTC

Add metadata to the published craft packages

There isn't a reliable way to put the necessary metadata to Artifactory
during publishing of Cargo and Maven builds. As a result, we will
add these metadata right after publishing instead.

Author: İlker Emre Koç
Author Date: 2025-04-14 10:28:13 UTC

Close files after they are used

The temporary files created for re-creation of HTTPInputStreams could
cause a backlog of unclosed files on the system, possibly resulting in
dangerous slow-downs. Thus we close the files that won't be used
anymore explicitly.

Also seperated the test case for the processInputs into two since
test cases with a large number of tests would result in a resource
warning during testing.

This is revert of a revert. I am reapplying the commit:
"620b8da6acdd164a53d4094488018a9e9a0f83a5"

Author: İlker Emre Koç
Author Date: 2025-04-30 14:59:08 UTC

Fix the link enablement tests

The link enablement change had affected tests in various places across
Launchpad. Since that commit changed the logic behind the link
enablement by makin all the recipe links available, I had to change
these tests to also expect enabled links all the time.

Author: İlker Emre Koç
Author Date: 2025-04-28 06:24:20 UTC

Revert "upgrade multipart with line-break parsing"

I am reverting since. The upgrade failed some buildbot tests.

This reverts commit a3fbbd49e00800ee6d2501ecec93ceaf3b3c5535.

Author: İlker Emre Koç
Author Date: 2025-04-14 10:28:13 UTC

Close files after they are used

The temporary files created for re-creation of HTTPInputStreams could
cause a backlog of unclosed files on the system, possibly resulting in
dangerous slow-downs. Thus we close the files that won't be used
anymore explicitly.

Also seperated the test case for the processInputs into two since
test cases with a large number of tests would result in a resource
warning during testing.

Author: İlker Emre Koç
Author Date: 2025-04-11 07:39:41 UTC

Enable all recipe links

The dynamic enablement of recipe links use a auto generated query, that
is slow and complex, resulting in unnecessary computation and timeouts.
Thus, enabling all the links from the get go would eliminate timeouts
on profiles, projects and teams.

Author: Enrique Sanchez Cardoso
Author Date: 2025-04-04 16:16:55 UTC

Fix glob expression to match all CVEs

Merged from https://code.launchpad.net/~enriqueesanchz/launchpad/+git/launchpad/+merge/483817

Author: İlker Emre Koç
Author Date: 2025-03-28 11:18:38 UTC

change ignore pfoe log level to info

We cannot see the logs of IgnorePoolFileOverwriteError right now since
the diskpool configs (which artifactory inherits) sets the log level
to INFO while we currently try to log it into DEBUG. Thus, we change
the log level of the log from DEBUG to INFO.

Author: İlker Emre Koç
Author Date: 2025-02-27 13:12:18 UTC

Add CRLF acceptance into FileBugDataParser

FileBugDataParser is the only parser that works on ApportJobs i.e. jobs
from Apport package. Originally, only the LF line-break was expected
from Apport, and a hard-coded logic was put inside FileBugDataParser.
The problem is, this was in violation of the RFC standards for HTTP
request formats, which only accepts CRLF as its linebreaks. Because of
this, an upgrade to multipart package which parsed multipart/form-data
requests broke the apport reporting system by becoming more strict in
its parsing by refusing to parse LF line-breaks.

Thus, to upgrade multipart we need to patch apport package to only
send CRLF line-breaks, but to patch apport package we need to
make FileBugDataParser accept CRLF line-breaks in the first place.

This commit is intended to make FileBugDataParser accept CRLF while
retaining LF parsing for backwards compatibility.

To read the input blob message without changing the cursor placement in
the file-alias object I implemented a "reset" using a "close()" call
followed by an "open()" call. This was because there were no "seek()"
or "peek()" functionality in LibraryFileAlias class used in the parser

In the testing module for the FileBugDataParser, I implemented a
MockBytesIO class that inherits the built-in BytesIO class. This was
because the tests of the parser all used BytesIO as their input
stream/file, which doesn't have a open() method while having a close()
method that deletes the message from the memory. Thus, I implemented
this class so that the tests can be made with an object that acts the
same as LibraryFileAlias in our use-case.

I also added a new test text-file into the codebase to ensure we have a
message file that uses CRLF for its line-breaks so that we can test the
ApportJobs better.

Lastly, I added some unit tests for both the parser and the
ProcessApportJob class which uses the parser.

Author: İlker Emre Koç
Author Date: 2025-01-29 13:17:52 UTC

revert wadllib upgrade

As the multipart package needed a revert to its upgrade because of an apport bug, the upgrade to wadllib package which depends on multipart must also be reverted.

Author: İlker Emre Koç
Author Date: 2025-01-29 11:51:36 UTC

revert multipart package for apport bug

The newest versions of multipart package have strict parsing needs, most prominently their strictness over HTTP request line-breaks where they only accept CRLF line-breaks. Meanwhile the bug-report command on Ubuntu in particular and apport package in general uses LF line-breaks on its HTTP requests to Launchpad which breaks the parsing the multipart package does. Ultimately, resulting in a failure to report bugs. As a result, we are temporarily reverting the multipart package upgrade until better fixes on Launchpad and, later, apport package can be published.

Author: İlker Emre Koç
Author Date: 2025-01-28 08:42:17 UTC

upgraded wadllib from 1.3.6 tp 2.0.0

- Upgraded wadllib from 1.3.6 to 2.0.0
- Added importlib-resources in version 6.4.5

Author: İlker Emre Koç
Author Date: 2024-12-05 14:05:14 UTC

refactor different uses of release_id variable name

In the Puplisher class' C_updateArtifactoryProperties method "release_id" was used for storing release ids of packages from both launchpad database as well as artifactory, which sometimes didn't align with eachother. Thus, I renamed these different use cases to improve readability.

Author: Alvaro Crespo Serrano
Author Date: 2025-01-09 18:43:23 UTC

Upgrade psutil to version 6.1.0

Merged from https://code.launchpad.net/~alvarocs/launchpad/+git/launchpad/+merge/478661

Author: İlker Emre Koç
Author Date: 2024-12-19 13:20:54 UTC

upgrade lpjsmin from 0.6 to 0.7

Note that lpjsmin is only used in the make file. Thus testing involved using phony aliases within the make file in various ways especially when lpjsmin is called directly or when phony aliases using lpjsmin is called. At the end of it, the builds and tests still ran as expected thus the commit went forward.

Author: İlker Emre Koç
Author Date: 2024-12-18 14:39:05 UTC

fix requirements for pyparsing

- Reverted the pyparsing version change to ztk-versions.cfg
- Added upgraded pyparsing version into launchpad.txt

Author: İlker Emre Koç
Author Date: 2024-12-03 13:17:07 UTC

upgrade python-dateutil 2.8.1 to 2.9.0.post0

- Upgraded python-dateutil from 2.8.1 to 2.9.0.post0
- Upgraded types-python-dateutil from 2.8.1 to 2.9.0.20241003. Note that this change is retroactive and seem to only affect mypy and tox functionalities. No additional packages are installed in lp-dependencies.

Author: İlker Emre Koç
Author Date: 2024-12-03 11:53:24 UTC

upgrade pyyaml from 6.0.1 to 6.0.2

Author: İlker Emre Koç
Author Date: 2024-11-26 08:25:58 UTC

remove typing-extensions package

Removed the typing-extensions package imports from code-base and requirements files. It was only used in two modules for importing TypedDict class which has been built into python since version=3.8

Author: İlker Emre Koç
Author Date: 2024-11-28 14:09:21 UTC

upgrade boto3 from 1.16.63 to 1.35.71

- Upgraded boto3 from 1.16.63 to 1.35.71
- Upgraded botocore from 1.19.63 to 1.35.71
- Upgraded s3transfer from 0.3.6 to 0.10.4
Other dependencies of boto3 seem to be sufficiently supported in the current environment, so no additional changes were made.

Author: İlker Emre Koç
Author Date: 2024-11-27 14:16:08 UTC

upgrade markdown from 3.2.2 to 3.7

In requirements files, upgraded Markdown and its dependencies.
Markdown from 3.2.2 to 3.7
importlib-metadata from 2.1.1 to 8.5.0
zipp from 1.2.0 to 3.20.2

Author: İlker Emre Koç
Author Date: 2024-11-27 09:26:44 UTC

Update pyparsing version in ztk-versions.cfg

Author: İlker Emre Koç
Author Date: 2024-11-22 12:50:59 UTC

Remove the mention of twitter from error pages

Author: Ines Almeida
Author Date: 2024-11-19 11:48:00 UTC

Print correct base name in error when arch not supported

Currently when using `build-base` instead of `base` as defined in https://snapcraft.io/docs/release-notes-snapcraft-3-7 we print out the wrong base name ('unknown') because we print the value of `base`. This prints the actual base used to get the supported architectures

Merged from https://code.launchpad.net/~ines-almeida/launchpad/+git/launchpad/+merge/474468

Author: 🤖 stg-launchpad-buildbot
Author Date: 2024-11-13 14:30:07 UTC

Automatic merge from stable up to d2bc21dc3e93

Author: 🤖 stg-launchpad-buildbot
Author Date: 2024-11-13 09:15:09 UTC

Automatic merge from stable up to a17846e6f4d8