0
0
mirror of https://github.com/go-gitea/gitea.git synced 2024-12-22 16:36:04 +01:00
Commit Graph

103 Commits

Author SHA1 Message Date
Chongyi Zheng
e80466f734
Resolve lint for unused parameter and unnecessary type arguments (#30750)
Resolve all cases for `unused parameter` and `unnecessary type
arguments`

Related: #30729

---------

Co-authored-by: Giteabot <teabot@gitea.io>
2024-04-29 08:47:56 +00:00
Chongyi Zheng
4ae6b1a553
Remove unused parameter for some functions in services/mirror (#30724)
Suggested by gopls `unusedparams`
2024-04-27 10:44:49 +00:00
silverwind
74f0c84fa4
Enable more revive linter rules (#30608)
Noteable additions:

- `redefines-builtin-id` forbid variable names that shadow go builtins
- `empty-lines` remove unnecessary empty lines that `gofumpt` does not
remove for some reason
- `superfluous-else` eliminate more superfluous `else` branches

Rules are also sorted alphabetically and I cleaned up various parts of
`.golangci.yml`.
2024-04-22 11:48:42 +00:00
Jason Song
fd59cd9450
Avoid losing token when updating mirror settings (#30429)
Fix #30416.

Before (it shows as "Unset" while there's a token):

<img width="980" alt="image"
src="https://github.com/go-gitea/gitea/assets/9418365/d7148e3e-62c9-4d2e-942d-3d795b79515a">

After:

<img width="977" alt="image"
src="https://github.com/go-gitea/gitea/assets/9418365/24aaa1db-5baa-4204-9081-470b15ea72b5">

The username shows as "oauth2" because of
f9fdac9809/services/migrations/dump.go (L99)

I have checked that all usage of `MirrorRemoteAddress` has been updated.

<img width="1806" alt="image"
src="https://github.com/go-gitea/gitea/assets/9418365/2f042501-2824-4511-9203-c84a6731a02d">

However, it needs to be checked again when backporting.

---------

Co-authored-by: Giteabot <teabot@gitea.io>
2024-04-13 17:41:57 +02:00
yp05327
b4d86912ef
Fix mirror error when mirror repo is empty (#30432)
Fix #30424

Co-authored-by: Giteabot <teabot@gitea.io>
2024-04-13 03:01:02 +00:00
Lunny Xiao
3c6fc25a77
Use repo object format name instead of detecting from git repository (#29702)
It's unnecessary to detect the repository object format from git
repository. Just use the repository's object format name.
2024-03-10 22:30:36 +01:00
Lunny Xiao
25b842df26
Move get/set default branch from git package to gitrepo package to hide repopath (#29126) 2024-03-08 15:30:10 +08:00
Lunny Xiao
5f82ead13c
Simplify how git repositories are opened (#28937)
## Purpose
This is a refactor toward building an abstraction over managing git
repositories.
Afterwards, it does not matter anymore if they are stored on the local
disk or somewhere remote.

## What this PR changes
We used `git.OpenRepository` everywhere previously.
Now, we should split them into two distinct functions:

Firstly, there are temporary repositories which do not change:

```go
git.OpenRepository(ctx, diskPath)
```

Gitea managed repositories having a record in the database in the
`repository` table are moved into the new package `gitrepo`:

```go
gitrepo.OpenRepository(ctx, repo_model.Repo)
```

Why is `repo_model.Repository` the second parameter instead of file
path?
Because then we can easily adapt our repository storage strategy.
The repositories can be stored locally, however, they could just as well
be stored on a remote server.

## Further changes in other PRs
- A Git Command wrapper on package `gitrepo` could be created. i.e.
`NewCommand(ctx, repo_model.Repository, commands...)`. `git.RunOpts{Dir:
repo.RepoPath()}`, the directory should be empty before invoking this
method and it can be filled in the function only. #28940
- Remove the `RepoPath()`/`WikiPath()` functions to reduce the
possibility of mistakes.

---------

Co-authored-by: delvh <dev.lh@web.de>
2024-01-27 21:09:51 +01:00
delvh
778ad795fd
Refactor deletion (#28610)
Introduce the new generic deletion methods
- `func DeleteByID[T any](ctx context.Context, id int64) (int64, error)`
- `func DeleteByIDs[T any](ctx context.Context, ids ...int64) error`
- `func Delete[T any](ctx context.Context, opts FindOptions) (int64,
error)`

So, we no longer need any specific deletion method and can just use
the generic ones instead.

Replacement of #28450

Closes #28450

---------

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-12-25 21:25:29 +01:00
Lunny Xiao
408a484224
Adjust object format interface (#28469)
- Remove `ObjectFormatID`
- Remove function `ObjectFormatFromID`.
- Use `Sha1ObjectFormat` directly but not a pointer because it's an
empty struct.
- Store `ObjectFormatName` in `repository` struct
2023-12-17 11:56:08 +00:00
Adam Majer
cbf923e87b
Abstract hash function usage (#28138)
Refactor Hash interfaces and centralize hash function. This will allow
easier introduction of different hash function later on.

This forms the "no-op" part of the SHA256 enablement patch.
2023-12-13 21:02:00 +00:00
JakobDev
cc5df26680
Even more db.DefaultContext refactor (#27352)
Part of #27065

---------

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: delvh <dev.lh@web.de>
2023-10-03 10:30:41 +00:00
delvh
e709bc199f
Differentiate between push and pull mirror sync in progress (#27390)
Previously, if you had both a push and a pull mirror, the message did
not clarify if you've accidentally synchronized the wrong one.
Additionally fixed two typos that were encountered while debugging.

## Screenshots

![grafik](https://github.com/go-gitea/gitea/assets/51889757/164d5d20-728d-4365-9cb5-c37e95857cdf)

![grafik](https://github.com/go-gitea/gitea/assets/51889757/90bfffd2-abd4-4847-b8e2-db4231700a6d)
2023-10-02 14:52:18 +00:00
JakobDev
cf0df023be
More db.DefaultContext refactor (#27265)
Part of #27065

This PR touches functions used in templates. As templates are not static
typed, errors are harder to find, but I hope I catch it all. I think
some tests from other persons do not hurt.
2023-09-29 12:12:54 +00:00
JakobDev
7047df36d4
Another round of db.DefaultContext refactor (#27103)
Part of #27065

---------

Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
2023-09-25 13:17:37 +00:00
JakobDev
f91dbbba98
Next round of db.DefaultContext refactor (#27089)
Part of #27065
2023-09-16 14:39:12 +00:00
Lunny Xiao
540bf9fa6d
Move notification interface to services layer (#26915)
Extract from #22266
2023-09-05 18:37:47 +00:00
Chongyi Zheng
43652746f2
Move modules/mirror to services (#26737)
To solve the cyclic imports in a better way

Closes #20261
2023-08-27 10:24:45 +08:00
Lunny Xiao
45ac90eb54
Sync branches when mirroring (#25722)
Caused by #22743

---------

Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
2023-07-06 13:18:37 +02:00
silverwind
88f835192d
Replace interface{} with any (#25686)
Result of running `perl -p -i -e 's#interface\{\}#any#g' **/*` and `make fmt`.

Basically the same [as golang did](2580d0e08d).
2023-07-04 18:36:08 +00:00
Lunny Xiao
f9cfd6ce5b
Use the type RefName for all the needed places and fix pull mirror sync bugs (#24634)
This PR replaces all string refName as a type `git.RefName` to make the
code more maintainable.

Fix #15367
Replaces #23070 
It also fixed a bug that tags are not sync because `git remote --prune
origin` will not remove local tags if remote removed.

We in fact should use `git fetch --prune --tags origin` but not `git
remote update origin` to do the sync.

Some answer from ChatGPT as ref.

> If the git fetch --prune --tags command is not working as expected,
there could be a few reasons why. Here are a few things to check:
> 
>Make sure that you have the latest version of Git installed on your
system. You can check the version by running git --version in your
terminal. If you have an outdated version, try updating Git and see if
that resolves the issue.
> 
>Check that your Git repository is properly configured to track the
remote repository's tags. You can check this by running git config
--get-all remote.origin.fetch and verifying that it includes
+refs/tags/*:refs/tags/*. If it does not, you can add it by running git
config --add remote.origin.fetch "+refs/tags/*:refs/tags/*".
> 
>Verify that the tags you are trying to prune actually exist on the
remote repository. You can do this by running git ls-remote --tags
origin to list all the tags on the remote repository.
> 
>Check if any local tags have been created that match the names of tags
on the remote repository. If so, these local tags may be preventing the
git fetch --prune --tags command from working properly. You can delete
local tags using the git tag -d command.

---------

Co-authored-by: delvh <dev.lh@web.de>
2023-05-26 01:04:48 +00:00
wxiaoguang
6f9c278559
Rewrite queue (#24505)
# ⚠️ Breaking

Many deprecated queue config options are removed (actually, they should
have been removed in 1.18/1.19).

If you see the fatal message when starting Gitea: "Please update your
app.ini to remove deprecated config options", please follow the error
messages to remove these options from your app.ini.

Example:

```
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options
```

Many options in `[queue]` are are dropped, including:
`WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`,
`BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed
from app.ini.

# The problem

The old queue package has some legacy problems:

* complexity: I doubt few people could tell how it works.
* maintainability: Too many channels and mutex/cond are mixed together,
too many different structs/interfaces depends each other.
* stability: due to the complexity & maintainability, sometimes there
are strange bugs and difficult to debug, and some code doesn't have test
(indeed some code is difficult to test because a lot of things are mixed
together).
* general applicability: although it is called "queue", its behavior is
not a well-known queue.
* scalability: it doesn't seem easy to make it work with a cluster
without breaking its behaviors.

It came from some very old code to "avoid breaking", however, its
technical debt is too heavy now. It's a good time to introduce a better
"queue" package.

# The new queue package

It keeps using old config and concept as much as possible.

* It only contains two major kinds of concepts:
    * The "base queue": channel, levelqueue, redis
* They have the same abstraction, the same interface, and they are
tested by the same testing code.
* The "WokerPoolQueue", it uses the "base queue" to provide "worker
pool" function, calls the "handler" to process the data in the base
queue.
* The new code doesn't do "PushBack"
* Think about a queue with many workers, the "PushBack" can't guarantee
the order for re-queued unhandled items, so in new code it just does
"normal push"
* The new code doesn't do "pause/resume"
* The "pause/resume" was designed to handle some handler's failure: eg:
document indexer (elasticsearch) is down
* If a queue is paused for long time, either the producers blocks or the
new items are dropped.
* The new code doesn't do such "pause/resume" trick, it's not a common
queue's behavior and it doesn't help much.
* If there are unhandled items, the "push" function just blocks for a
few seconds and then re-queue them and retry.
* The new code doesn't do "worker booster"
* Gitea's queue's handlers are light functions, the cost is only the
go-routine, so it doesn't make sense to "boost" them.
* The new code only use "max worker number" to limit the concurrent
workers.
* The new "Push" never blocks forever
* Instead of creating more and more blocking goroutines, return an error
is more friendly to the server and to the end user.

There are more details in code comments: eg: the "Flush" problem, the
strange "code.index" hanging problem, the "immediate" queue problem.

Almost ready for review.

TODO:

* [x] add some necessary comments during review
* [x] add some more tests if necessary
* [x] update documents and config options
* [x] test max worker / active worker
* [x] re-run the CI tasks to see whether any test is flaky
* [x] improve the `handleOldLengthConfiguration` to provide more
friendly messages
* [x] fine tune default config values (eg: length?)

## Code coverage:

![image](https://user-images.githubusercontent.com/2114189/236620635-55576955-f95d-4810-b12f-879026a3afdf.png)
2023-05-08 19:49:59 +08:00
Jason Song
7b9e23cc1d
Fill head commit to in payload when notifying push commits for mirroring (#23215)
Just like what has been done when pushing manually:

7a5af25592/services/repository/push.go (L225-L226)

Before:

<img width="448" alt="image"
src="https://user-images.githubusercontent.com/9418365/222100123-cd4839d1-2d4d-45f7-b7a0-0cbc73162b44.png">

After:

<img width="448" alt="image"
src="https://user-images.githubusercontent.com/9418365/222100225-f3c5bb65-7ab9-41e2-8e39-9d84c23c352d.png">

---------

Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-03-03 23:45:53 -05:00
Gusted
1cb8d14bf7
Use proxy for pull mirror (#22771)
- Use the proxy (if one is specified) for pull mirrors syncs.
- Pulled the code from
c2774d9e80/modules/git/repo.go (L164-L170)

Downstream issue: https://codeberg.org/forgejo/forgejo/issues/302

---------

Co-authored-by: Lauris BH <lauris@nix.lv>
2023-02-11 08:39:50 +08:00
wxiaoguang
6bc3079c00
Refactor git command package to improve security and maintainability (#22678)
This PR follows #21535 (and replace #22592)

## Review without space diff

https://github.com/go-gitea/gitea/pull/22678/files?diff=split&w=1

## Purpose of this PR

1. Make git module command completely safe (risky user inputs won't be
passed as argument option anymore)
2. Avoid low-level mistakes like
https://github.com/go-gitea/gitea/pull/22098#discussion_r1045234918
3. Remove deprecated and dirty `CmdArgCheck` function, hide the `CmdArg`
type
4. Simplify code when using git command

## The main idea of this PR

* Move the `git.CmdArg` to the `internal` package, then no other package
except `git` could use it. Then developers could never do
`AddArguments(git.CmdArg(userInput))` any more.
* Introduce `git.ToTrustedCmdArgs`, it's for user-provided and already
trusted arguments. It's only used in a few cases, for example: use git
arguments from config file, help unit test with some arguments.
* Introduce `AddOptionValues` and `AddOptionFormat`, they make code more
clear and simple:
    * Before: `AddArguments("-m").AddDynamicArguments(message)`
    * After: `AddOptionValues("-m", message)`
    * -
* Before: `AddArguments(git.CmdArg(fmt.Sprintf("--author='%s <%s>'",
sig.Name, sig.Email)))`
* After: `AddOptionFormat("--author='%s <%s>'", sig.Name, sig.Email)`

## FAQ

### Why these changes were not done in #21535 ?

#21535 is mainly a search&replace, it did its best to not change too
much logic.

Making the framework better needs a lot of changes, so this separate PR
is needed as the second step.


### The naming of `AddOptionXxx`

According to git's manual, the `--xxx` part is called `option`.

### How can it guarantee that `internal.CmdArg` won't be not misused?

Go's specification guarantees that. Trying to access other package's
internal package causes compilation error.

And, `golangci-lint` also denies the git/internal package. Only the
`git/command.go` can use it carefully.

### There is still a `ToTrustedCmdArgs`, will it still allow developers
to make mistakes and pass untrusted arguments?

Generally speaking, no. Because when using `ToTrustedCmdArgs`, the code
will be very complex (see the changes for examples). Then developers and
reviewers can know that something might be unreasonable.

### Why there was a `CmdArgCheck` and why it's removed?

At the moment of #21535, to reduce unnecessary changes, `CmdArgCheck`
was introduced as a hacky patch. Now, almost all code could be written
as `cmd := NewCommand(); cmd.AddXxx(...)`, then there is no need for
`CmdArgCheck` anymore.


### Why many codes for `signArg == ""` is deleted?

Because in the old code, `signArg` could never be empty string, it's
either `-S[key-id]` or `--no-gpg-sign`. So the `signArg == ""` is just
dead code.

---------

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-04 10:30:43 +08:00
flynnnnnnnnnn
e81ccc406b
Implement FSFE REUSE for golang files (#21840)
Change all license headers to comply with REUSE specification.

Fix #16132

Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
2022-11-27 18:20:29 +00:00
KN4CK3R
044c754ea5
Add context.Context to more methods (#21546)
This PR adds a context parameter to a bunch of methods. Some helper
`xxxCtx()` methods got replaced with the normal name now.

Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-11-19 16:12:33 +08:00
wxiaoguang
dcd9fc7ee8
Refactor git command arguments and make all arguments to be safe to be used (#21535)
Follow #21464

Make all git command arguments strictly safe. Most changes are one-to-one replacing, keep all existing logic.
2022-10-23 22:44:45 +08:00
Lunny Xiao
f860a6d2e4
Add system setting table with cache and also add cache supports for user setting (#18058) 2022-10-17 07:29:26 +08:00
Mohamed Sekour
0e61a74e5a
Add new API endpoints for push mirrors management (#19841)
- Add a new push mirror to specific repository
- Sync now ( send all the changes to the configured push mirrors )
- Get list of all push mirrors of a repository
- Get a push mirror by ID
- Delete push mirror by ID

Signed-off-by: Mohamed Sekour <mohamed.sekour@exfo.com>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: zeripath <art27@cantab.net>
2022-07-30 18:45:59 +02:00
Chongyi Zheng
49f9d43afe
Implement sync push mirror on commit (#19411)
Support synchronizing with the push mirrors whenever new commits are pushed or synced from pull mirror.

Related Issues: #18220

Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-07-08 20:45:12 +01:00
Gusted
f0ce5470e5
Always try to fetch repo for mirrors (#19975)
- Always give a best-effort to fetching the repositories, if even that
fails indeed give a disconnected mirror found error.
- *Partially* resolves #19928

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-06-15 23:58:44 +08:00
Lunny Xiao
ce3dd04c63
Fix some mirror bugs (#18649)
* Fix some mirror bugs

* Remove unnecessary code

* Fix lint

* rename stdard url

* Allow more charactors in git ssh protocol url

* improve the detection

* support ipv6 for git url parse

* Fix bug

* Fix template

* Fix bug

* fix template

* Fix tmpl

* Fix tmpl

* Fix parse ssh with interface

* Rename functions name

Co-authored-by: zeripath <art27@cantab.net>
2022-06-11 21:50:14 +08:00
Lunny Xiao
26095115f4
Move some repository related code into sub package (#19711)
* Move some repository related code into sub package

* Move more repository functions out of models

* Fix lint

* Some performance optimization for webhooks and others

* some refactors

* Fix lint

* Fix

* Update modules/repository/delete.go

Co-authored-by: delvh <dev.lh@web.de>

* Fix test

* Merge

* Fix test

* Fix test

* Fix test

* Fix test

Co-authored-by: delvh <dev.lh@web.de>
2022-06-06 16:01:49 +08:00
zeripath
e32ab429ff
Prevent NPE on update mirror settings (#19864)
A `repo_model.Mirror` repository field (`.Repo`) will not automatically
be set, but is used without checking in mirror_pull.go:UpdateAddress.
This will cause an NPE.

This PR changes UpdateAddress to use the helper function GetRepository()
helping prevent future NPEs but also changes modules/context/repo.go to
ensure that the Mirror.Repo is set.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2022-06-02 12:56:32 -04:00
Lunny Xiao
fd7d83ace6
Move almost all functions' parameter db.Engine to context.Context (#19748)
* Move almost all functions' parameter db.Engine to context.Context
* remove some unnecessary wrap functions
2022-05-20 22:08:52 +08:00
wxiaoguang
124b072f0b
Remove git.Command.Run and git.Command.RunInDir* (#19280)
Follows #19266, #8553, Close #18553, now there are only three `Run..(&RunOpts{})` functions.
 * before: `stdout, err := RunInDir(path)`
 * now: `stdout, _, err := RunStdString(&git.RunOpts{Dir:path})`
2022-04-01 10:55:30 +08:00
wxiaoguang
c83168104b
Use a more general (and faster) method to sanitize URLs with credentials (#19239)
Use a more general method to sanitize URLs with credentials: Simple and intuitive / Faster /  Remove all credentials in all URLs
2022-03-31 10:25:40 +08:00
6543
3e88af898a
Make git.OpenRepository accept Context (#19260)
* OpenRepositoryCtx -> OpenRepository
* OpenRepository -> openRepositoryWithDefaultContext, only for internal usage
2022-03-30 03:13:41 +08:00
zeripath
889a8c268c
Use full output of git show-ref --tags to get tags for PushUpdateAddTag (#19235)
Strangely #19038 appears to relate to an issue whereby a tag appears to
be listed in `git show-ref --tags` but then does not appear when `git
show-ref --tags -- short_name` is called.

As a solution though I propose to stop the second call as it is
unnecessary and only likely to cause problems.

I've also noticed that the tags calls are wildly inefficient and aren't using the common cat-files - so these have been added.

I've also noticed that the git commit-graph is not being written on mirroring - so I've also added writing this to the migration which should improve mirror rendering somewhat. 

Fix #19038

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
2022-03-29 19:12:33 +02:00
zeripath
d2ca021df0
Touch mirrors on even on fail to update (#19217)
* Touch mirrors on even on fail to update

If a mirror fails to be synchronised it should be pushed to the bottom of the queue
of the awaiting mirrors to be synchronised. At present if there LIMIT number of
broken mirrors they can effectively prevent all other mirrors from being synchronized
as their last_updated time will remain earlier than other mirrors.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-03-27 15:40:17 +01:00
Lunny Xiao
c29fbc6d23
Hide sensitive content on admin panel progress monitor (#19218)
Sanitize urls within git process descriptions.

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Andrew Thornton <art27@cantab.net>
2022-03-27 12:54:09 +01:00
wxiaoguang
7a550b3af2
Use ctx instead of db.DefaultContext in some packages(routers/services/modules) (#19163)
* Remove `db.DefaultContext` usage in routers, use `ctx` directly

* Use `ctx` directly if there is one, remove some `db.DefaultContext` in `services`

* Use ctx instead of db.DefaultContext for `cmd` and some `modules` packages

* fix incorrect context usage
2022-03-22 16:22:54 +01:00
zeripath
75eb6cc9c1
Improve SyncMirrors logging (#19045)
Yet another issue has come up where the logging from SyncMirrors does not provide
enough context. This PR adds more context to these logging events.

Related #19038

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-03-10 10:09:48 +00:00
Gusted
a14b6f3aae
Refactor mirror code & fix StartToMirror (#18904)
* Use MirrorID instead of RepoID

- Use the MirrorID as index(SQL uses `id` column not the `repo_id`).
Passes the Mirror ID's into the Sync functions.

* Check for MirrorID == 0

* Fix `StartToMirror` + refactor

* Update services/mirror/mirror.go

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-03-02 15:43:11 +08:00
Lunny Xiao
b75ad7b87f
Improve mirror iterator (#18928)
* Improve mirror iterator

* fix test
2022-02-28 20:41:06 +01:00
Martin Scholz
26718a785a
Change git.cmd to RunWithContext (#18693)
Change all `cmd...Pipeline` commands to `cmd.RunWithContext`.

#18553

Co-authored-by: Martin Scholz <martin.scholz@versasec.com>
2022-02-11 13:47:22 +01:00
zeripath
df44017328
Restart zero worker if there is still work to do (#18658)
* Restart zero worker if there is still work to do

It is possible for the zero worker to timeout before all the work is finished.
This may mean that work may take a long time to complete because a worker will only
be induced on repushing.

Also ensure that requested count is reset after pulls and push mirror sync requests and add some more trace logging to the queue push.

Fix #18607

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-02-08 14:02:32 +00:00
6543
3043eb36bf
Delete old git.NewCommand() and use it as git.NewCommandContext() (#18552) 2022-02-06 20:01:47 +01:00
zeripath
a82fd98d53
Pause queues (#15928)
* Start adding mechanism to return unhandled data

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Create pushback interface

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Add Pausable interface to WorkerPool and Manager

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Implement Pausable and PushBack for the bytefifos

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Implement Pausable and Pushback for ChannelQueues and ChannelUniqueQueues

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Wire in UI for pausing

Signed-off-by: Andrew Thornton <art27@cantab.net>

* add testcases and fix a few issues

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix build

Signed-off-by: Andrew Thornton <art27@cantab.net>

* prevent "race" in the test

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix jsoniter mismerge

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix conflicts

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix format

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Add warnings for no worker configurations and prevent data-loss with redis/levelqueue

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Use StopTimer

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-01-22 21:22:14 +00:00