Plugins » Katello

I'm working on the Dynflowization of repo set enablement: the parallel execution of repo creations might probably help + we can try to identify the slow bits and address that ad-hoc.

Dynflowized repo enable for reposet enable with 12 repos took 12 seconds against mirrored cdn, 25s against real one.

There are two slow things here:

1. loading the listing files from cdn (don't know yet how long it takes, will have more data tomorrow)
2. creating the repositories: one single repo is created in about 2 seconds. However, when we run the
creating of 12 repos in parallel, the last one takes about 8 seconds to create becuase waiting in a
queue for a while for repo publish.

With the second step, we could:

a. skip the repo publishing and postpone it for syncing, as without syncing, the repo is not that much useful.
  b. increase the pulp concurrency options to be able to run more distribution publish in parallel

The a. option could also help with http://projects.theforeman.org/issues/4724

I've extracted the reposet enable code into more dynflow actions to have more numbers in hand:

1. load initial data to be able to start scanning: 2s
2. the scanning of cnd itself: 3.5s
3. create 12 repos locally in db: 16 s: majority of the time, it was the repository.save itself
4. create 12 repos in pulp in parallel (7 s total, in scale of 3-7s: most of the time it was the distributor publish stuff)

This gives us around 30s for repo set enable.

This is already boost comparing the old style, where the repo creation (both local db and pulp orchestration) was run
in sequence, which gives us more than 60s for the task.

Possible optimization are:

1. run even the local repo creation in parallel (would give us around - 10s)
2. disable the metadata genration after repo is created (as we don't need that until some content appears there anyway) (would give us around -8s)
3. postpone the repo creation to the repo-enable phase (would give us - -25s) it would be just about loading the listing files from cdn, which takes around 5 seconds. Later optimization could be

My favorite is the option 3. and now is good time to do this change, we need to add the support for repo-enable in new cli so we will touch the code anyway.

Having just the cdn scan for reposet enable checkbox takes us to something around 8 seconds, which is not that bad.