From c8026e11136e2afac8f7d91cf2fd3dd570708185 Mon Sep 17 00:00:00 2001 From: Afham Fardeen Date: Mon, 19 Jan 2026 10:39:00 +0530 Subject: [PATCH 1/2] doc: adding usage doc: update cli doc: adding windows cli doc: fix description --- docs/USAGE.md | 428 +++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 333 insertions(+), 95 deletions(-) diff --git a/docs/USAGE.md b/docs/USAGE.md index 6f0bc51..d4d5321 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -6,115 +6,353 @@ Assuming the setup script was run, you can kick off the application by running ' If Go is installed, the project can be downloaded and run with `go run go-earlybird.go` ### Streamed / Piped input -Using the `-stream` flag, users can stream or pipe file contents to 'go-earlybird'. +Using the `-stream` flag, users can stream or pipe file contents to 'go-earlybird'. -``` +
+macOS/Linux + +```shell ᐅ go-earlybird -stream < /path/to/file -``` -... or: -``` ᐅ cat /path/to/file | go-earlybird -stream ``` -### HTTP API +
+ +
+Windows + +```shell +> go-earlybird -stream > \path\to\file +> type \path\to\file | go-earlybird -stream ``` -ᐅ go-earlybird --http 0.0.0.0:3000 + +
+ +### HTTP API + +```shell +go-earlybird --http 0.0.0.0:3000 ``` + + `/scan` will accept a multi-part upload and scan the contents, returning json output. -The normal HTTP listener will operate on HTTP/1.1. Go-EarlyBird can be run as HTTPS/2 with the `-https [ip:port]` flag. Note that this also requires the `-https-cert [/path/to/cert]` and `-https-key [/path/to/key]` parameters. +The normal HTTP listener will operate on HTTP/1.1. Go-EarlyBird can be run as HTTPS/2 with the `-https [ip:port]` flag. Note that this also requires the `-https-cert [/path/to/cert]` and `-https-key [/path/to/key]` parameters. -The simple webserver configuration file can be found in the local config directory (`~/.go-earlybird/webserver.json` or `C:\Users\[me]\AppData\go-earlybird\webserver.json`). A separate config file can be specified using the `-http-config [/path/to/configfile]` flag. +The simple webserver configuration file can be found in the local config directory: +- macOS/Linux: `~/.go-earlybird/webserver.json` +- Windows: `C:\Users\[me]\AppData\go-earlybird\webserver.json` +A separate config file can be specified using the `-http-config [/path/to/configfile]` flag. ### Local Git Scanning With the flag `-git-staged` or `-git-tracked`, Go-EarlyBird can limit its scan to only look at files that are staged or tracked (respectively) by Git. -## Usage +## Usage(CLI) The executable can be called from the command line with the following syntax: + +```shell +go-earlybird --help +``` + +Please note that you can download the earlybird repository using the command +``` +git clone https://github.com/americanexpress/earlybird.git +cd earlybird +``` + +# General Options + +- **`--path string`**
+`Type: String`, `Default: '/home/earlybird'`
+Specify the **directory** to scan (absolute path only). + +
+macOS/Linux + +```shell +ᐅ go run go-earlybird.go --path=/scanning_dir --config=./config +``` + +
+ +
+Windows + +```shell +> go run go-earlybird.go --path=.\scanning_dir --config=.\config +``` + +
+ +# Selective scanning + +- **`--enable value`**
+`Type: String`, `Default:''`
+Enable individual scanning modules. The supported module names are: **`inclusivity-rules`**, **`password-secret`** + +- **`--ignore-fp-rules`**
+`Type: Boolean`, `Default: false`
+If enabled it **ignore false positive** post-process rules. + +# Scan Result Display Options + +- **`--display-confidence`**
+`Type: String`, `Default:low`
+The secret scan results are displayed based on a **minimum confidence threshold**. Only findings with a confidence level equal to or higher than the configured threshold are shown. +The supported confidence levels are: **low**, **medium**, **high** and **critical**. + +The **confidence levels** are prioritized in the following order: +``` +critical: 1 +high: 2 +medium: 3 +low: 4 +``` + +- **`--display-severity string`**
+`Type: String`, `Default:low`
+Secret scan results are displayed based on the **minimum severity threshold**. Only findings with a severity equal to or higher than the configured level are shown. +The supported severity levels are: **info**, **low**, **medium**, **high** and **critical**. + +The **severity levels** are prioritized in the following order: +``` +critical: 1 +high: 2 +medium: 3 +low: 4 +info: 5 +``` + + +- **`--show-full-line`**
+`Type: Boolean`, `Default: false`
+If enabled it display the full line where the pattern match was found. + +- **`--show-rules-only`**
+`Type: Boolean`, `Default: false`
+Display rules that would be run, but do not execute a scan. + + +```shell +go run go-earlybird.go --config=./config --show-rules-only=true +``` + +- **`--show-solutions`**
+`Type: Boolean`, `Default: false`
+If enabled true, the scan displays recommended solutions for each finding. + +## Scanning considered as failure + +- **`--fail-confidence`**
+`Type: String`, `Default: low`
+Specifies the **minimum confidence level** at which the scan will be considered a failure. If the scanner detects one or more valid findings that meet or exceed this threshold, the scan fails and the process exits with status code 1. +- **`--fail-severity`**
+`Type: String`, `Default: low`
+Specifies the **minimum severity level** at which the scan will be considered a failure. If the scanner detects one or more valid findings that meet or exceed this threshold, the scan fails and the process exits with status code 1. + +# File and Output Options + +- **`--file`**
+Specify **output file** to store the scan results. + +
+macOS/Linux + +```shell +ᐅ go run go-earlybird.go --path=/scanning_dir --config=./config --file=/myfile.csv +``` + +
+ +
+Windows + +```shell +> go run go-earlybird.go --path=.\scanning_dir --config=.\config --file=.\myfile.csv +``` + +
+ +- **`--format`**
+`Type: String`, `Default: console`
+Specifies the **output format** for the scan results. +The supported severity levels are: **`console`**, **`json`**, **`csv`** + +# Git Repository Scanning + +We can do the earlybird scan for the **git repository** also instead of scanning the directory. This could be done by passing the git repo url and respective branch name instead of passing the scanning directory path. + +- **`--git`**
+`Type: String`
+Specifies the **full URL** of the Git repository to scan. + +- **`--git-branch string`**
+`Type: String`
+Specifies name of the **branch** to be scanned. + +
+macOS/Linux + +```shell +ᐅ export gituser= +ᐅ export gitpassword= +ᐅ go run go-earlybird.go --config=./config --git=https://github.com/user/repo --git-branch=branch_name +``` + +
+ +
+Windows + +```shell +> set gituser= +> set gitpassword= +> go run go-earlybird.go --config=.\config --git=https://github.com/user/repo --git-branch=branch_name +``` + +
+ +- **`--git-commit-stream`**
+`Type: Boolean`, `Default: false`
+When enabled use **stream IO of Git commit log** as input instead of files. It allows the Earlybird scan to process data directly from a stream (e.g., standard input) instead of scanning files in a directory. +Specifies name of the branch to be scanned. + + +```shell +git log | go run go-earlybird.go --config=./config --git-commit-stream=true +``` + + +- **`--git-project string`**
+`Type: String`
+Specify the full URL to a **GitHub organization** or **Bitbucket project** to scan instead of scanning just a repo. Please note that you need to provide the git user-name and the password also during the scanning. + +```shell +go run go-earlybird.go --config=./config --git-project=https://github.com/org --git-user=user_name ``` -~/go/src/gearlybird (master ✘)✭ ᐅ go-earlybird --help -Usage of go-earlybird: - -config string - Directory where configuration files are stored (default "/Users/janedoe/.go-earlybird/") - -display-confidence string - Lowest confidence level to display [ critical | high | medium | low ] (default "high") - -display-severity string - Lowest severity level to display [ critical | high | medium | low ] (default "medium") - -enable value - Enable individual scanning modules [ ccnumber | content | filename | password-secret ] - -fail-confidence string - Lowest confidence level at which to fail [ critical | high | medium | low ] (default "high") - -fail-severity string - Lowest severity level at which to fail [ critical | high | medium | low ] (default "high") - -file string - Output file -- e.g., 'go-earlybird --file=/home/jdoe/myfile.csv' - -format string - Output format [ console | json | csv ] (default "console"). - -git string - Full URL to a git repo to scan e.g. github.com/user/repo - -git-branch string - Name of branch to be scanned - -git-commit-stream - Use stream IO of Git commit log as input instead of file(s) -- e.g., 'cat secrets.text > go-earlybird' - -git-project string - Full URL to a github organization or bitbucket project to scan e.g. github.com/org - -git-staged - Scan only git staged files - -git-tracked - Scan only git tracked files - -git-user string - If the git repository is private, enter an authorized username - -http string - Listen IP and Port for HTTP API e.g. 127.0.0.1:8080 - -http-config string - Path to webserver config JSON file - -https string - Listen IP and Port for HTTPS/2 API e.g. 127.0.0.1:8080 (Don't forget the https-cert and https-key flags) - -https-cert string - Certificate file for TLS - -https-key string - Private key file for TLS - -ignore-failure - Avoid the exit code 1 in case of scanner finds valid findings and meets fail threshold - -ignore-fp-rules - Ignore the false positive post-process rules - -ignorefile string - Patterns File (including wildcards) for files to ignore. (e.g. *.jpg) (default "/Users/jhans12/.ge_ignore") - -max-file-size int - Maximum file size to scan (in bytes) (default 10240000) - -path string - Directory to scan (defaults to CWD) -- ABSOLUTE PATH ONLY (default "/Users/jhans12/go/src/gearlybird") - -show-full-line - Display the full line where the pattern match was found (warning: this can be dangerous with minified script files) - -show-rules-only - Display rules that would be run, but do not execute a scan - -skip-comments - Skip scanning comments in files -- applies only to the 'content' module - -stream - Use stream IO as input instead of file(s) - -strict-jks - Checks for private keys in the JKS file and only return finding if found. If not passed, it will flag jks file. Default is false. - -suppress - Suppress reporting of the secret found (important if output is going to Slack or other logs) - -update - Update module configurations - -verbose - Reports details about file reads - -version - Display version information and exit - -with-console - Prints findings in console with JSON format report - -workers int - Set number of workers. (default 100) - -worksize int - Set Line Wrap Length. (default 2500) - -module-config-file string - Absolute path to a json or yaml file for per module level config -- {"modules": { "aModule": { "display_severity": "medium" } } } - ``` - -### Performing a scan with only certain modules enabled: - -```bash -go-earlybird -path /dir/to/scan -enable password-secret -enable content -enable inclusivity-rules -``` \ No newline at end of file + + +- **`--git-staged`**
+`Type: Boolean` `Default: false`
+When enabled, it Scans only **Git staged files** instead of scanning the whole file. + +- **`--git-tracked`**
+`Type: Boolean` `Default: false`
+When enabled, it scan only **Git tracked files** instead of scanning the whole file. + +- **`--git-user`**
+`Type: String`
+Specify the authorized **username** for private Git repositories scanning. + +# Earlybird as an HTTP/HTTPS Server +Instead of directly scanning the repository or directory we can **create an HTTP/HTTPS server** also and send multiple requests to scan the repositories. + +- **`--disable-keep-alives`**:
+`Type: Boolean`, `Default: false`
+It is a configuration utility for the earlybird http-Server that controls whether HTTP keep-alives are enabled. By default, keep-alives is enabled, but they can be disabled to close idle connections and conserve resources. This is particularly useful in environments with limited resources. + +- **`--http`**
+`Type: String`
+Specify an **IP and Port** while creating an HTTP server to listen to the API (e.g., `127.0.0.1:8080`). +- **`--http-config`**
+`Type: String`
+Specify the **path to webserver config** JSON file. (e.g., `my-file.json`). +```shell +go run go-earlybird.go --http=127.0.0.1:8080 --http-config=my-file.json +``` +The default json configuration looks as listed below: +```shell +{ + WriteTimeout: 60, + ReadTimeout: 60, + IdleTimeout: 120, +} +``` + +- **`--https`**
+`Type: String`
+Specify an **IP and Port** while create an HTTPS/2 server to listen to the API (e.g., `127.0.0.1:8080`). Note, please also add the https-cert and https-key using the cli flags listed below. + +- **`--https-cert`**
+`Type: String`
+Specify the Certificate file for TLS connect for the https server. +- **`--https-key`**
+`Type: String`
+Specify the private key file for TLS connect for the https server. + +
+macOS/Linux + +```shell +ᐅ go run go-earlybird.go --https=127.0.0.1:8080 --https-cert=/path/to/cert --https-key=/path/to/key +``` + +
+ +
+Windows + +```shell +> go run go-earlybird.go --https=127.0.0.1:8080 --https-cert=\path\to\cert --https-key=\path\to\key +``` + +
+ +# Additional Options + +- **`--ignorefile`**
+`Type: String`, `Default: './.ge_ignore`
+Specify the patterns file for files to ignore. You can list files inside the ge_ignore as listed below: +```shell +/folder-name1/** +/folder-name2/subfolder/** +**/file-name.extension +``` + + +- **`--ignore-failure`**
+`Type: Boolean`, `Default: false`
+If enabled it avoid **exit code 1** if valid findings meet fail threshold. + +- **`--max-file-size`**
+`Type: Int`, `Default: 10240000`
+Specify the maximum file size in bytes to scan. +- **`--module-config-file`**
+`Type: String`
+Specify the path to file with per-module config settings. + +- **`--skip-comments`**
+`Type: Boolean`, `Default: false`
+If enabled it Skips scanning comments in files (applies only to the `content` module). +- **`--stream`**
+`Type: Boolean`, `Default: false`
+When enabled true the Earlybird scan uses stream IO as input instead of files. So instead of passing the scanning directory we passes a stream data source. + + +```shell +echo 'my file has access_key = secret_key' | go run go-earlybird.go --config=./config --stream=true +``` + + +- **`--suppress`**
+`Type: Boolean`, `Default: false`
+When enabled true it mask the secret line value(*****) while reporting of the secret found. +- **`--update`**
+`Type: Boolean`, `Default: false`
+When set to true it updates module configurations. +- **`--verbose`**
+`Type: Boolean`, `Default: false`
+When set to true the scan report comes with details about file reads. +- **`--version`**
+`Type: Boolean`, `Default: false`
+Display version information and then exit. +- **`--with-console`**
+`Type: Boolean`, `Default: false`
+If enabled it allows using along with the `--format` flag, to print findings report in console with JSON format. +- **`--workers int`**
+`Type: Int`, `Default: 100`
+Set number of workers needed to create the worker pool while scanning the code. +- **`--worksize`**
+`Type: Int`, `Default: 2500`
+Set line wrap length. When a lengthy line is fed as an input from the code for the scan work-size define the max character size of line for scanning, remaining character get fed as a new input with the same process, Please note that there is also an overlap happens to not miss any secrets that got split. \ No newline at end of file From 8d926d7f385fd7ca4d2f11c31d07650355683069 Mon Sep 17 00:00:00 2001 From: Afham Fardeen Date: Wed, 21 Jan 2026 20:01:48 +0530 Subject: [PATCH 2/2] doc: adding cli usage --- docs/USAGE.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/USAGE.md b/docs/USAGE.md index d4d5321..488af7b 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -87,9 +87,14 @@ Specify the **directory** to scan (absolute path only). # Selective scanning -- **`--enable value`**
-`Type: String`, `Default:''`
-Enable individual scanning modules. The supported module names are: **`inclusivity-rules`**, **`password-secret`** +- **`--ignorefile`**
+`Type: String`, `Default: './.ge_ignore`
+Specify the patterns file for files to ignore. You can list files inside the ge_ignore as listed below: +```shell +/folder-name1/** +/folder-name2/subfolder/** +**/file-name.extension +``` - **`--ignore-fp-rules`**
`Type: Boolean`, `Default: false`
@@ -301,14 +306,9 @@ Specify the private key file for TLS connect for the https server. # Additional Options -- **`--ignorefile`**
-`Type: String`, `Default: './.ge_ignore`
-Specify the patterns file for files to ignore. You can list files inside the ge_ignore as listed below: -```shell -/folder-name1/** -/folder-name2/subfolder/** -**/file-name.extension -``` +- **`--enable value`**
+`Type: String`, `Default:''`
+Enable individual scanning modules. By default all module are enabled and we recommend to use this feature only when specific module scanning is required. The supported module names are: **`inclusivity-rules`**, **`password-secret`**, **`ccnumber`**, **`content`** and **`filename`** - **`--ignore-failure`**