Version 0.57.0

.github/SECURITY.md

@@ -20,4 +20,6 @@ Please include a description of the vulnerability, steps to reproduce,
 the affected version, and impact assessment if possible.
 
 I will acknowledge receipt within 72 hours and aim to release a fix
-within 90 days. Public disclosure will be coordinated with the reporter.
+within 90 days. Public disclosure will be coordinated with the reporter. If
+the vulnerability cannot be confirmed and the opener does not respond within
+90 days, your advisory may be automatically closed.

.github/workflows/python-package.yml

@@ -27,6 +27,9 @@ jobs:
         python -m pip install --upgrade pip
         python -m pip install flake8 pytest
         if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+        if [ -f test_requirements.txt ]; then pip install -r test_requirements.txt; fi
+    - name: Install wkhtmltopdf
+      run: sudo apt install -y wkhtmltopdf
     - name: Lint with flake8
       run: |
         # stop the build if there are Python syntax errors or undefined names

.gitignore

@@ -32,6 +32,7 @@ __pycache__/
 !/example-msg-files/expected-outputs/2016-02-23_0657 MSG Test File/*
 !/example-msg-files/strangeDate.msg
 !/example-msg-files/unicode.msg
+!/example-msg-files/multi-stream.msg
 !/example-msg-files/multi-to.msg
 !/example-msg-files/multi-to-to.msg
 !/example-msg-files/unicode-header.msg

CHANGELOG.md

@@ -1,3 +1,20 @@
+**v0.57.0**
+* [[Advisory GHSA-cxpg-prmv-g4pp](https://github.com/TeamMsgExtractor/msg-extractor/security/advisories/GHSA-cxpg-prmv-g4pp)] Fixed a vulnerability related to unchecked input passed out to wkhtmltopdf when converting the MSG file to PDF. Javascript is now strongly blocked from being enabled and will be stripped from the HTML where appropriate before reaching wkhtmltopdf. Additionally, the option `--wk-allow-external-files` on the command line (or `wkAllowExternalFiles` to `MessageBase.save()`) will be required for external files to be loaded into the HTML to convert. Without these, they will be stripped before reaching wkhtmltopdf for security.
+* [[Advisory GHSA-mqxr-j8rh-p4q4](https://github.com/TeamMsgExtractor/msg-extractor/security/advisories/GHSA-mqxr-j8rh-p4q4)] Fixed an issue where an attacker controlled MSG file could take advantage of a poorly written section of code in the RTF deencapsulation function on the `extract-msg` side to waste massive amounts of time and memory resources.
+* [[Advisory GHSA-rgjf-vmqv-8pv9](https://github.com/TeamMsgExtractor/msg-extractor/security/advisories/GHSA-rgjf-vmqv-8pv9)] Change the way multiple type properties are accessed for binary and string. This includes modifications to `MSGFile.getMultipleBinary()`, `MSGFile.getMultipleString()`, and `MSGFile.exists()` to significantly reduce wasted times for malicious files. Malicious files previously could cause extract-msg to do unnecessarily large numbers of calls to `MSGFile.exists()` which previously did not use caching to speed up its search.
+* Changed `MSGFile.listDir()` to return a tuple of tuples to prevent it's cache results from being tampered with.
+* Fixed an issue with `MSGFile.listDir()` that caused it to not cache in certain circumstances when it should always cache.
+* Fixed a check in `MSGFile.getMultipleBinary()` that executed wrong and did not properly check for bad data.
+* Fixed `MSGFile._getTypedSteam()` to correctly parse binary and string streams when an entry is missing.
+* Fixed `StandardViolationError` being raised without checking error behavior in `MSGFile.getMultipleBinary()` and `MSGFile.getMultipleString()`.
+* Added check for standard violation in `MSGFile.getMultipleString()` that is raised (or logged, depending on error behavior) if a string stream is missing a null byte. This also doesn't strip the last character in the stream if it is not null.
+* Added length check log to `MSGFile.getMultipleBinary()` and `MSGFile.getMultipleString()` that checks to see if the expected length for each stream matches what the length stream says. If not, it logs a warning about the mismatch, but does not consider this a gross violation of the standard.
+* Changed how `MessageBase.deencapsulatedRtf` handles RTF data without a closing brace.
+* Fixed wrong return being used in `extract_msg.encoding.utils.singleByteEncode()` and `extract_msg.encoding.utils.variableByteEncode()`.
+* Fixed missing return in `extract_msg.encoding.utils.singleByteEncode()`.
+* Fixed wrong parameter names for codec types made in `extract_msg.encoding.utils.createSBEncoding()` and `extract_msg.encoding.utils.createVBEncoding()`.
+* Fix some values being incorrectly returned from encode/decode functions in `extract_msg.encoding.utils`.
+
 **v0.56.1**
 * [[TeamMsgExtractor #484](https://github.com/TeamMsgExtractor/msg-extractor/issues/484)] Corrected the way the HTML body is decoded in `MessageBase.asEmailMessage()` to prevent encoding errors.
 

README.rst

@@ -1,4 +1,4 @@
-|License: GPL v3| |PyPI3| |PyPI2| |Read the Docs|
+|License: GPL v3| |PyPI3| |PyPI2| |Downloads| |Read the Docs|
 
 extract-msg
 =============
@@ -14,7 +14,7 @@ Documentation can be found in the code, on the `wiki`_, and on the
 NOTICE
 ======
 0.29.* is the branch that supports both Python 2 and Python 3. It is now only
-receiving bug fixes and will not be receiving feature updates.
+receiving security fixes and will not be receiving feature updates.
 
 0.39.* is the last versions that supported Python 3.6 and 3.7. Support for those
 was dropped to allow the use of new features from 3.8 and because the life spans
@@ -59,55 +59,59 @@ Currently, the README is in the process of being redone. For now, please
 refer to the usage information provided from the program's help dialog:
 ::
 
-     usage: extract_msg [-h] [--use-content-id] [--json] [--file-logging] [-v] [--log LOG] [--config CONFIGPATH] [--out OUTPATH] [--use-filename] [--dump-stdout] [--html] [--pdf] [--wk-path WKPATH] [--wk-options [WKOPTIONS ...]]
-                        [--prepared-html] [--charset CHARSET] [--raw] [--rtf] [--allow-fallback] [--skip-body-not-found] [--zip ZIP] [--save-header] [--attachments-only] [--skip-hidden] [--no-folders] [--skip-embedded] [--extract-embedded]
-                        [--overwrite-existing] [--skip-not-implemented] [--out-name OUTNAME | --glob] [--ignore-rtfde] [--progress] [-s]
-                        msg [msg ...]
+     usage: extract_msg [-h] [--use-content-id] [--json] [--file-logging] [-v] [--log LOG] [--config CONFIGPATH] [--out OUTPATH] [--use-filename] [--dump-stdout] [--html]
+                        [--pdf] [--wk-path WKPATH] [--wk-options [WKOPTIONS ...]] [--wk-allow-external-files] [--prepared-html] [--charset CHARSET] [--raw] [--rtf]
+                        [--allow-fallback] [--skip-body-not-found] [--zip ZIP] [--save-header] [--attachments-only] [--skip-hidden] [--no-folders] [--skip-embedded]
+                        [--extract-embedded] [--overwrite-existing] [--skip-not-implemented] [--out-name OUTNAME | --glob] [--ignore-rtfde] [--progress] [-s]
+                        [msg ...]
 
      extract_msg: Extracts emails and attachments saved in Microsoft Outlook's .msg files. https://github.com/TeamMsgExtractor/msg-extractor
 
      positional arguments:
-       msg                   An MSG file to be parsed.
+     msg                   An MSG file to be parsed.
 
      options:
-       -h, --help            show this help message and exit
-       --use-content-id, --cid
-                             Save attachments by their Content ID, if they have one. Useful when working with the HTML body.
-       --json                Changes to write output files as json.
-       --file-logging        Enables file logging. Implies --verbose level 1.
-       -v, --verbose         Turns on console logging. Specify more than once for higher verbosity.
-       --log LOG             Set the path to write the file log to.
-       --config CONFIGPATH   Set the path to load the logging config from.
-       --out OUTPATH         Set the folder to use for the program output. (Default: Current directory)
-       --use-filename        Sets whether the name of each output is based on the msg filename.
-       --dump-stdout         Tells the program to dump the message body (plain text) to stdout. Overrides saving arguments.
-       --html                Sets whether the output should be HTML. If this is not possible, will error.
-       --pdf                 Saves the body as a PDF. If this is not possible, will error.
-       --wk-path WKPATH      Overrides the path for finding wkhtmltopdf.
-       --wk-options [WKOPTIONS ...]
-                             Sets additional options to be used in wkhtmltopdf. Should be a series of options and values, replacing the - or -- in the beginning with + or ++, respectively. For example: --wk-options "+O Landscape"
-       --prepared-html       When used in conjunction with --html, sets whether the HTML output should be prepared for embedded attachments.
-       --charset CHARSET     Character set to use for the prepared HTML in the added tag. (Default: utf-8)
-       --raw                 Sets whether the output should be raw. If this is not possible, will error.
-       --rtf                 Sets whether the output should be RTF. If this is not possible, will error.
-       --allow-fallback      Tells the program to fallback to a different save type if the selected one is not possible.
-       --skip-body-not-found
-                             Skips saving the body if the body cannot be found, rather than throwing an error.
-       --zip ZIP             Path to use for saving to a zip file.
-       --save-header         Store the header in a separate file.
-       --attachments-only    Specify to only save attachments from an msg file.
-       --skip-hidden         Skips any attachment marked as hidden (usually ones embedded in the body).
-       --no-folders          Stores everything in the location specified by --out. Requires --attachments-only and is incompatible with --out-name.
-       --skip-embedded       Skips all embedded MSG files when saving attachments.
-       --extract-embedded    Extracts the embedded MSG files as MSG files instead of running their save functions.
-       --overwrite-existing  Disables filename conflict resolution code for attachments when saving a file, causing files to be overwriten if two attachments with the same filename are on an MSG file.
-       --skip-not-implemented, --skip-ni
-                             Skips any attachments that are not implemented, allowing saving of the rest of the message.
-       --out-name OUTNAME    Name to be used with saving the file output. Cannot be used if you are saving more than one file.
-       --glob, --wildcard    Interpret all paths as having wildcards. Incompatible with --out-name.
-       --ignore-rtfde        Ignores all errors thrown from RTFDE when trying to save. Useful for allowing fallback to continue when an exception happens.
-       --progress            Shows what file the program is currently working on during it's progress.
-       -s, --stdin           Read file from stdin (only works with one file at a time).
+     -h, --help            show this help message and exit
+     --use-content-id, --cid
+                           Save attachments by their Content ID, if they have one. Useful when working with the HTML body.
+     --json                Changes to write output files as json.
+     --file-logging        Enables file logging. Implies --verbose level 1.
+     -v, --verbose         Turns on console logging. Specify more than once for higher verbosity.
+     --log LOG             Set the path to write the file log to.
+     --config CONFIGPATH   Set the path to load the logging config from.
+     --out OUTPATH         Set the folder to use for the program output. (Default: Current directory)
+     --use-filename        Sets whether the name of each output is based on the MSG filename.
+     --dump-stdout         Tells the program to dump the message body (plain text) to stdout. Overrides saving arguments.
+     --html                Sets whether the output should be HTML. If this is not possible, will error.
+     --pdf                 Saves the body as a PDF. If this is not possible, will error.
+     --wk-path WKPATH      Overrides the path for finding wkhtmltopdf.
+     --wk-options [WKOPTIONS ...]
+                           Sets additional options to be used in wkhtmltopdf. Should be a series of options and values, replacing the - or -- in the beginning with + or ++ respectively. For example: --wk-options "+O Landscape"
+     --wk-allow-external-files
+                           Enables external files to appear in the PDF output. WARNING: This allows wkpdftohtml to make url requests that are unchecked, and as such may be a security vulnerability in untrusted emails. Only use this option if you trust the source of the MSG file.
+     --prepared-html       When used in conjunction with --html, sets whether the HTML output should be prepared for embedded attachments.
+     --charset CHARSET     Character set to use for the prepared HTML in the added tag. (Default: utf-8)
+     --raw                 Sets whether the output should be raw. If this is not possible, will error.
+     --rtf                 Sets whether the output should be RTF. If this is not possible, will error.
+     --allow-fallback      Tells the program to fallback to a different save type if the selected one is not possible.
+     --skip-body-not-found
+                           Skips saving the body if the body cannot be found, rather than throwing an error.
+     --zip ZIP             Path to use for saving to a zip file.
+     --save-header         Store the header in a separate file.
+     --attachments-only    Specify to only save attachments from an MSG file.
+     --skip-hidden         Skips any attachment marked as hidden (usually ones embedded in the body).
+     --no-folders          Stores everything in the location specified by --out. Requires --attachments-only and is incompatible with --out-name.
+     --skip-embedded       Skips all embedded MSG files when saving attachments.
+     --extract-embedded    Extracts the embedded MSG files as MSG files instead of running their save functions.
+     --overwrite-existing  Disables filename conflict resolution code for attachments when saving a file, causing files to be overwritten if two attachments with the same
+                           filename are on an MSG file.
+     --skip-not-implemented, --skip-ni
+                           Skips any attachments that are not implemented, allowing saving of the rest of the message.
+     --out-name OUTNAME    Name to be used with saving the file output. Cannot be used if you are saving more than one file.
+     --glob, --wildcard    Interpret all paths as having wildcards. Incompatible with --out-name.
+     --ignore-rtfde        Ignores all errors thrown from RTFDE when trying to save. Useful for allowing fallback to continue when an exception happens.
+     --progress            Shows what file the program is currently working on during it's progress.
+     -s, --stdin           Read file from stdin (only works with one file at a time).
 
 **To use this in your own script**, start by using:
 
@@ -122,7 +126,7 @@ From there, open the MSG file:
      msg = extract_msg.openMsg("path/to/msg/file.msg")
 
 Alternatively, if you wish to send a msg binary string instead of a file
-to the ``extract_msg.openMsg`` Method:
+to the ``extract_msg.openMsg()`` Method:
 
 ::
 
@@ -238,14 +242,18 @@ Credits
 
 `Philippe Lagadec`_ - Python OleFile module developer.
 
+`Seamus Tuohy`_ - Developer of the Python RTFDE module. Gave first examples of how to use the module and has worked with Destiny to ensure functionality.
+
+`Kadhiraven EG`_ - Discovery of and assistance with vulnerability in PDF conversion, including assistance with updating security policies and notifying the public about the vulnerability.
+
 `Joel Kaufman`_ - First implementations of the json and filename flags.
 
 `Dean Malmgren`_ - First implementation of the setup.py script.
 
-`Seamus Tuohy`_ - Developer of the Python RTFDE module. Gave first examples of how to use the module and has worked with Destiny to ensure functionality.
-
 `Liam`_ - Significant reorganization and transfer of data.
 
+
+
 And thank you to everyone who has opened an issue and helped us track down those pesky bugs.
 
 Extra
@@ -260,12 +268,15 @@ your access to the newest major version of extract-msg.
 .. |License: GPL v3| image:: https://img.shields.io/badge/License-GPLv3-blue.svg
    :target: LICENSE.txt
 
-.. |PyPI3| image:: https://img.shields.io/badge/pypi-0.56.1-blue.svg
-   :target: https://pypi.org/project/extract-msg/0.56.1/
+.. |PyPI3| image:: https://img.shields.io/badge/pypi-0.57.0-blue.svg
+   :target: https://pypi.org/project/extract-msg/0.57.0/
 
 .. |PyPI2| image:: https://img.shields.io/badge/python-3.8+-brightgreen.svg
    :target: https://www.python.org/downloads/release/python-3810/
 
+.. |Downloads| image:: https://img.shields.io/pypi/dm/extract-msg.svg
+   :target: https://pypistats.org/packages/extract-msg
+
 .. |Read the Docs| image:: https://readthedocs.org/projects/msg-extractor/badge/?version=latest
     :target: https://msg-extractor.readthedocs.io/en/stable/?badge=latest
     :alt: Documentation Status
@@ -277,6 +288,7 @@ your access to the newest major version of extract-msg.
 .. _Dean Malmgren: https://github.com/deanmalmgren
 .. _Joel Kaufman: https://github.com/joelkaufman
 .. _Liam: https://github.com/LiamPM5
+.. _Kadhiraven EG: https://github.com/JOKER-53
 .. _Seamus Tuohy: https://github.com/seamustuohy
 .. _Discord: https://discord.com/invite/B77McRmzdc
 .. _Buy Me a Coffee: https://www.buymeacoffee.com/DestructionE