0
0
mirror of https://github.com/go-gitea/gitea.git synced 2024-12-14 18:23:58 +01:00
gitea/tests
Bruno Sofiato f64fbd9b74
Updated tokenizer to better matching when search for code snippets (#32261)
This PR improves the accuracy of Gitea's code search. 

Currently, Gitea does not consider statements such as
`onsole.log("hello")` as hits when the user searches for `log`. The
culprit is how both ES and Bleve are tokenizing the file contents (in
both cases, `console.log` is a whole token).

In ES' case, we changed the tokenizer to
[simple_pattern_split](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-simplepatternsplit-tokenizer.html#:~:text=The%20simple_pattern_split%20tokenizer%20uses%20a,the%20tokenization%20is%20generally%20faster.).
In such a case, tokens are words formed by digits and letters. In
Bleve's case, it employs a
[letter](https://blevesearch.com/docs/Tokenizers/) tokenizer.

Resolves #32220

---------

Signed-off-by: Bruno Sofiato <bruno.sofiato@gmail.com>
2024-11-06 20:51:20 +00:00
..
e2e Add typescript guideline and typescript-specific eslint plugins and fix issues (#31521) 2024-07-03 17:48:14 +02:00
fuzz Rework markup link rendering (#26745) 2024-01-15 08:49:24 +00:00
gitea-lfs-meta Test views of LFS files (#22196) 2022-12-23 07:41:56 +08:00
gitea-repositories-meta Updated tokenizer to better matching when search for code snippets (#32261) 2024-11-06 20:51:20 +00:00
integration Fix milestone deadline and date related problems (#32339) 2024-11-05 07:46:40 +00:00
testdata/data Add artifacts test fixture (#30300) 2024-11-01 10:29:54 +08:00
mssql.ini.tmpl Azure blob storage support (#30995) 2024-05-30 07:33:50 +00:00
mysql.ini.tmpl Disable query token param in integration tests (#28592) 2023-12-23 11:29:51 +08:00
pgsql.ini.tmpl add {{TEST_MINIO_ENDPOINT}} for local testing "with/without" docker + fix pgsql testing doc (#32105) 2024-10-03 01:00:56 +00:00
sqlite.ini.tmpl Disable query token param in integration tests (#28592) 2023-12-23 11:29:51 +08:00
test_utils.go Refactor tests to prevent from unnecessary preparations (#32398) 2024-11-01 23:18:29 +08:00