Add a config option to block "expensive" pages for anonymous users (#34024) (#34071)

Backport #34024 since there are too many AI crawlers. The new code is
covered by tests and it does nothing if users don't set it.

custom/conf/app.example.ini

@@ -774,6 +774,9 @@ LEVEL = Info
 ;ALLOW_ONLY_EXTERNAL_REGISTRATION = false
 ;;
 ;; User must sign in to view anything.
+;; After 1.23.7, it could be set to "expensive" to block anonymous users accessing some pages which consume a lot of resources,
+;; for example: block anonymous AI crawlers from accessing repo code pages.
+;; The "expensive" mode is experimental and subject to change.
 ;REQUIRE_SIGNIN_VIEW = false
 ;;
 ;; Mail notification