Merge branch '3.1dev' into starttls_injection

Author: drwetter

.travis.yml

@@ -7,6 +7,12 @@ addons:
     packages:
     - dnsutils
     - jsonlint
+before_install:
+  - if ! git diff --name-only $TRAVIS_COMMIT_RANGE | grep -qvE '(.md)|(.pem)|(.pdf)|(.html)|^(LICENSE)|^(docs)|^(utils)|^(bin)|(Dockerfile)'
+    then
+      echo "no code was updated, not running the CI."
+      exit
+  fi
 install:
   - cpanm --notest Test::More
   - cpanm --notest Data::Dumper

CHANGELOG.md

@@ -27,7 +27,10 @@
 * Added environment variable for amount of attempts for ssl renegotiation check
 * Added --user-agent argument to support using a custom User Agent
 * Added --overwrite argument to support overwriting output files without warning
-* Headerflag X-XSS-Protection is labeled as INFO
+* Headerflag X-XSS-Protection is now labeled as INFO
+* Client simulation runs in wide mode which is even better readable
+* Added --reqheader to support custom headers in HTTP requests
+
 
 ### Features implemented / improvements in 3.0
 

CREDITS.md

@@ -42,6 +42,7 @@ Full contribution, see git log.
 * Jim Blankendaal
   - maximum certificate lifespan of 398 days
   - ssl renegotiation amount variable
+  - custom http request headers
 
 * Frank Breedijk
   - Detection of insecure redirects
@@ -181,4 +182,3 @@ Probably more I forgot to mention which did give me feedback, bug reports and he
 * Ivan Ristic/Qualys for the liberal license which made it possible to make partly use of the client data
 
 * My family for supporting me doing this work
-

Dockerfile.md

@@ -1,6 +1,21 @@
-## Usage:
+## Usage
+
+### From git directory
+
+```
+docker build .
+```
+
+Catch is when you run without image tags you need to catch the ID when building
+
+```
+[..]
+---> 889fa2f99933
+Successfully built 889fa2f99933
+```
+
+More comfortable is
 
-(in git directory):
 ```
 docker build -t mytestssl .
 docker run --rm -t mytestssl example.com
@@ -13,22 +28,22 @@ docker run -t mytestssl --help
 docker run --rm -t mytestssl -p --header example.com
 ```
 
-or pull the image from dockerhub and run:
+### From dockerhub
+
+You can pull the image from dockerhub and run:
 
 ```
-docker run --rm -t drwetter/testssl.sh --pfs example.com
+docker run --rm -t drwetter/testssl.sh --fs example.com
 ```
 
-Tags supported are: ``latest``, ``stable`` which _for now_ are all the same and point to ``3.0``.
+Supported tages are: ``3.1dev`` and ``latest`, which are the same, i.e. the rolling release. ``3.0`` is the latest stable version from git which might have a few improvements (see git log) over the released version 3.0.X.
 
 ``docker run --rm -t drwetter/testssl.sh:stable example.com``.
 
-And for the indomitable users who prefer to run old stuff you can use the tag ``2.9.5``. Please note ``2.9dev`` should not be used anymore.
-
-Keep in mind that any output file (--log, --html, --json etc.) will be created in the container. If you wish to have this created in a local directory you can mount a volume into the container and change the output prefix where the container user has write access to, e.g.:
+Keep in mind that any output file (--log, --html, --json etc.) will be created within the container. If you wish to have this created in a local directory on your host you can mount a volume into the container and change the output prefix where the container user has write access to, e.g.:
 
 ```
 docker run --rm -t -v /tmp:/data drwetter/testssl.sh --htmlfile /data/ example.com
 ```
 
-which writes the output to ``/tmp/example.com_p443-<date>-<time>.html.`` The uid/gid is the one from the docker user but normally the file is 644. testssl.sh's docker container uses a non-root user (usually with user/groupid 1000:1000).
+which writes the HTML output to ``/tmp/example.com_p443-<date>-<time>.html.`` The uid/gid is the one from the docker user. Normally the file is 644. testssl.sh's docker container uses a non-root user (usually with user/groupid 1000:1000).

Readme.md

@@ -40,7 +40,7 @@ to get bugfixes, other feedback and more contributions.
 Testssl.sh is working on every Linux/BSD distribution out of the box. Latest by 2.9dev
 most of the limitations of disabled features from the openssl client are gone
 due to bash-socket-based checks. As a result you can also use e.g. LibreSSL or OpenSSL >=
-1.1.1 . testssl.sh also works on other unixoid system out of the box, supposed they have
+1.1.1 . testssl.sh also works on other unixoid systems out of the box, supposed they have
 `/bin/bash` >= version 3.2 and standard tools like sed and awk installed. An implicit
 (silent) check for binaries is done when you start testssl.sh . System V needs probably
 to have GNU grep installed. MacOS X and Windows (using MSYS2, Cygwin or WSL) work too.
@@ -53,33 +53,31 @@ You can download testssl.sh branch 3.1dev just by cloning this git repository:
 
     git clone --depth 1 https://github.com/drwetter/testssl.sh.git
 
-Think of 3.1dev like a rolling release, see below. For the stable version help yourself by downloading the [ZIP](https://github.com/drwetter/testssl.sh/archive/3.0.2.zip) or [tar.gz](https://github.com/drwetter/testssl.sh/archive/3.0.2.zip) archive. Just ``cd`` to the directory created (=INSTALLDIR) and run it off there.
+Think of 3.1dev like a rolling release, see below. For the stable version help yourself by downloading the [ZIP](https://codeload.github.com/drwetter/testssl.sh/zip/3.0.4) or [tar.gz](https://codeload.github.com/drwetter/testssl.sh/tar.gz/3.0.4) archive. Just ``cd`` to the directory created (=INSTALLDIR) and run it off there.
 
 #### Docker
 
-Testssl.sh has minimal requirements. As stated you don't have to install or build anything.  You can just run it from the pulled/cloned directory. Still if you don't want to pull the github repo to your directory of choice you can pull a container from dockerhub and run it:
+Testssl.sh has minimal requirements. As stated you don't have to install or build anything. You can just run it from the pulled/cloned directory. Still if you don't want to pull the github repo to your directory of choice you can pull a container from dockerhub and run it:
 
 ```
-docker run --rm -ti  drwetter/testssl.sh:3.1dev <your_cmd_line>
+docker run --rm -ti  drwetter/testssl.sh <your_cmd_line>
 ```
 
 Or if you have cloned this repo you also can just ``cd`` to the INSTALLDIR and run
 ```
-docker build .
+docker build . -t imagefoo && docker run --rm -t imagefoo example.com
 ```
 
-followed by ``docker run -ti <ID> <your_cmd_line>`` where ``ID`` is the identifier in the last line from the build command like
+For more please consult [Dockerfile.md](https://github.com/drwetter/testssl.sh/blob/3.1dev/Dockerfile.md).
 
-```
- ---> 889fa2f99933
-Successfully built 889fa2f99933
-```
 
 ### Status
 
-We're currently in the development phase, version 3.1dev. 3.1dev will eventually become 3.2. Bigger features are developed in a separate branch before merged into 3.1dev to avoid hiccups or inconsistencies. Albeit we try to keep 3.1dev as solid as possible things will certainly change in 3.1dev. So if you need stability the 3.0 branch is better for you. Think of the 3.1dev branch like a rolling release.
+We're currently in the development phase, version 3.1dev. 3.1dev will eventually become 3.2. Bigger features are developed in a separate branch before merged into 3.1dev to avoid hiccups or inconsistencies. Albeit we try to keep 3.1dev as solid as possible things will certainly change in 3.1dev. Think of the 3.1dev branch like a rolling release. So if you need stability the 3.0 branch is better for you. 
+
+Version 3.0.X receives bugfixes, labeled as 3.0.1, 3.0.2 and so on. This will happen until 3.2 is released. 
 
-Support for 2.9.5 has been dropped. Supported is 3.0.x only.
+Support for 2.9.5 has been dropped. Supported is >= 3.0.x only.
 
 ### Documentation
 
@@ -90,7 +88,7 @@ Support for 2.9.5 has been dropped. Supported is 3.0.x only.
 
 ### Contributing
 
-Contributions are welcome! See [CONTRIBUTING.md](https://github.com/drwetter/testssl.sh/blob/3.0/CONTRIBUTING.md) for details.
+Contributions are welcome! See [CONTRIBUTING.md](https://github.com/drwetter/testssl.sh/blob/3.1dev/CONTRIBUTING.md) for details. Please also have a look at the [Coding Convention](https://github.com/drwetter/testssl.sh/blob/3.1dev/Coding_Convention.md}.
 
 ### Bug reports
 

doc/testssl.1

@@ -136,6 +136,9 @@ Please note that \fBfname\fR has to be in Unix format\. DOS carriage returns won
 .P
 \fB\-\-basicauth <user:pass>\fR This can be set to provide HTTP basic auth credentials which are used during checks for security headers\. BASICAUTH is the ENV variable you can use instead\.
 .
+.P
+\fB\-\-reqheader <header>\fR This can be used to add additional HTTP request headers in the correct format \fBHeadername: headercontent\fR\. This parameter can be called multiple times if required\. For example: \fB\-\-reqheader \'Proxy\-Authorization: Basic dGVzdHNzbDpydWxlcw==\' \-\-reqheader \'ClientID: 0xDEADBEAF\'\fR\. REQHEADER is the corresponding environment variable\.
+.
 .SS "SPECIAL INVOCATIONS"
 \fB\-t <protocol>, \-\-starttls <protocol>\fR does a default run against a STARTTLS enabled \fBprotocol\fR\. \fBprotocol\fR must be one of \fBftp\fR, \fBsmtp\fR, \fBpop3\fR, \fBimap\fR, \fBxmpp\fR, \fBxmpp-server\fR, \fBtelnet\fR, \fBldap\fR, \fBirc\fR, \fBlmtp\fR, \fBnntp\fR, \fBpostgres\fR, \fBmysql\fR\. For the latter four you need e\.g\. the supplied OpenSSL or OpenSSL version 1\.1\.1\. Please note: MongoDB doesn\'t offer a STARTTLS connection, LDAP currently only works with \fB\-\-ssl\-native\fR\. \fBtelnet\fR and \fBirc\fR is WIP\.
 .

doc/testssl.1.html

@@ -110,6 +110,8 @@ The same can be achieved by setting the environment variable `WARNINGS`.
 
 `--basicauth <user:pass>` This can be set to provide HTTP basic auth credentials which are used during checks for security headers. BASICAUTH is the ENV variable you can use instead.
 
+`--reqheader <header>` This can be used to add additional HTTP request headers in the correct format `Headername: headercontent`. This parameter can be called multiple times if required. For example: `--reqheader 'Proxy-Authorization: Basic dGVzdHNzbDpydWxlcw==' --reqheader 'ClientID: 0xDEADBEAF'`. REQHEADER is the corresponding environment variable.
+
 
 ### SPECIAL INVOCATIONS
 

doc/testssl.1.md

@@ -28,6 +28,8 @@ If you want to check trust against e.g. a company internal CA you need to use ``
 
 * ``cipher-mapping.txt`` contains information about all of the cipher suites defined for SSL/TLS
 
+* ``curves-mapping.txt`` contains information about all of the eliptic curves defined by IANA
+
 * ``ca_hashes.txt`` is used for HPKP test in order to have a fast comparison with known CAs. Use
    ``~/utils/create_ca_hashes.sh`` for an update
 

etc/README.md

@@ -14,7 +14,7 @@ The whole process is done manually.
 * Retrieve "handshakebytes" by marking the Record Layer --> Copy --> As a hex stream.
 * Figure out "protos" and "tlsvers" by looking at the supported_versions TLS extension (43=0x002b). May work only on modern clients. Be careful as some do not list all TLS versions here (OpenSSL 1.1.1 lists only TLS 1.2/1.3 here)
 * Adjust "lowest_protocol" and "highest_protocol" accordingly.
-* Get "curves" from at the supported groups TLS extension 10 = 0x00a. Omit any GREASE.
+* For "curves" mark the supported groups TLS extension --> Copy --> As a hex stream, remove any leading GREASE ciphers (?a?a) and supply it to `~/utils/hexstream2curves.sh`
 * Retrieve "alpn" by looking at the alpn TLS extension 16 (=0x0010).
 * Review TLS extension 13 (=0x000d) whether any SHA1 signature algorithm is listed. If not "requiresSha2" is true
 * Leave "maxDhBits"/"minDhBits" and "minRsaBits"/"maxRsaBits" at -1, unless you know for sure what the client can handle