Change log:

1. clean and add comments to code
2. Update README.md file

Author: Wasin Waeosri

README.md

@@ -6,9 +6,9 @@
 
 ## Overview
 
-This example shows how to writing an application to subscribe Machine Readable News (MRN) using [Elektron WebSocket API](https://developers.refinitiv.com/elektron/websocket-api) from Thomson Reuters Enterprise Platform (TREP). The example just connects to TREP via a WebSocket connection, then subscribes and display MRN News data in a console. The project are implemented with Python language, but the main concept for consuming and assembling MRN News messages are the same for all technologies. 
+This example shows how to writing the [Elektron WebSocket API](https://developers.refinitiv.com/elektron/websocket-api) application to subscribe Machine Readable News (MRN) from Thomson Reuters Enterprise Platform (TREP). The example just connects to TREP via a WebSocket connection, then subscribes and display MRN News data in a console. The project are implemented with Python language, but the main concept for consuming and assembling MRN News messages are the same for all technologies. 
 
-*Note:* The news message is in UTF-8 JSON string format. Some news messages that contains special unicode character may not be able to show in Windows OS console (cmd, git bash, powershell, etc) due to the OS limitation. Those messages will be print as *UnicodeEncodeError exception. Cannot decode unicode character* message in a console instead.
+*Note:* The news message is in UTF-8 JSON string format. Some news messages that contains special unicode character may not be able to show in Windows OS console (cmd, git bash, powershell, etc) due to the OS limitation. Those messages will be print as ```UnicodeEncodeError exception. Cannot decode unicode character``` message in a console instead.
 
 ## Machine Readable News Overview
 
@@ -17,23 +17,23 @@ Refinitiv Machine Readable News (MRN) is an advanced service for automating the
 ### MRN Data model
 MRN is published over Elektron using an Open Message Model (OMM) envelope in News Text Analytics domain messages. The Real-time News content set is made available over MRN_STORY RIC. The content data is contained in a FRAGMENT field that has been compressed, and potentially fragmented across multiple messages, in order to reduce bandwidth and message size.
 
-A FRAGMENT field has a different data type based on a connection:
-* RSSL connection: BUFFER type
-* WebSocket connection: Base64 string
+A FRAGMENT field has a different data type based on a connection type:
+* RSSL connection (ESDK [C++](https://developers.refinitiv.com/elektron/elektron-sdk-cc)/[Java](https://developers.refinitiv.com/elektron/elektron-sdk-java)): BUFFER type
+* WebSocket connection: Base64 ascii string
 
 The data goes through the following series of transformations:
 
 1. The core content data is a UTF-8 JSON string
 2. This JSON string is compressed using gzip
-3. The compressed JSON is split into a number of fragments (BUFFER or Base64 string) which each fit into a single update message
+3. The compressed JSON is split into a number of fragments (BUFFER or Base64 ascii string) which each fit into a single update message
 4. The data fragments are added to an update message as the FRAGMENT field value in a FieldList envelope
 
-Therefore, in order to parse the core content data, the application will need to reverse this process. The WebSocket application also need to convert a received Base64 string in a FRAGMENT field to bytes data before further process this field.
+Therefore, in order to parse the core content data, the application will need to reverse this process. The WebSocket application also need to convert a received Base64 string in a FRAGMENT field to bytes data before further process this field. This application uses Python [base64](https://docs.python.org/3/library/base64.html) and [zlib](https://docs.python.org/3/library/zlib.html) modules to decode Base64 string and decompress JSON string.
 
-If you are not familiar with MRN concept, please visit the following resources which will give you a full explanation of the MRN data behavior and how to process it:
+If you are not familiar with MRN concept, please visit the following resources which will give you a full explanation of the MRN data model and implementation logic:
 * [Webinar Recording: Introduction to Machine Readable News](https://developers.refinitiv.com/news#news-accordion-nid-12045)
 * [Introduction to Machine Readable News (MRN) with Elektron Message API (EMA)](https://developers.refinitiv.com/article/introduction-machine-readable-news-mrn-elektron-message-api-ema).
-* [Machine Readable News (MRN) & N2_UBMS Comparison and Migration Guide](https://developers.refinitiv.com/article/machine-readable-news-mrn-n2_ubms-comparison-and-migration-guide).
+* [MRN Data Models and Elektron Implementation Guide](https://developers.refinitiv.com/elektron/elektron-sdk-java/docs?content=8736&type=documentation_item).
 
 ## Prerequisite
 This example requires the following dependencies softwares and libraries.
@@ -63,15 +63,108 @@ Please be informed that your TREP server (ADS and ADH) should have a Service tha
 2. Run ```$> pip install -r requestments.txt``` in a console to install all the dependencies libraries.
 3. Then you can run mrn_console_app.py application with the following command
     ```
-    $> python mrn_console_app.py --hostname <ADS server IP Address/Hostname> --port <WebSocket Port> --item <MRN RIC> --service <ADS Contribution Service name>
+    $> python mrn_console_app.py --hostname <ADS server IP Address/Hostname> --port <WebSocket Port> --ric <MRN RIC name>
     ```
+Optionally, the application subscribes ```MRN_STORY``` RIC code from TREP by default. You can pass your interested MRN RIC code to ```--ric``` parameter on the application command line. The supported MRN RIC codes are ```MRN_STORY```, ```MRN_TRNA```, ```MRN_TRNA_DOC``` and ```MRN_TRSI``` only. the application 
+
+## Example Results
+### Send MRN_STORY request to TREP
+```
+SENT:
+{
+  "Domain":"NewsTextAnalytics",
+  "ID":2,
+  "Key":{
+    "Name":"MRN_STORY"
+  }
+}
+RECEIVED: 
+[
+  {
+    "Domain":"NewsTextAnalytics",
+    "Fields":{
+      "ACTIV_DATE":"2019-07-20",
+      "CONTEXT_ID":3752,
+      "DDS_DSO_ID":4232,
+      "FRAGMENT":null,
+      "FRAG_NUM":1,
+      "GUID":null,
+      "MRN_SRC":"HK1_PRD_A",
+      "MRN_TYPE":"STORY",
+      "MRN_V_MAJ":"2",
+      "MRN_V_MIN":"10",
+      "PROD_PERM":10001,
+      "RDN_EXCHD2":"MRN",
+      "RECORDTYPE":30,
+      "SPS_SP_RIC":".[SPSML2L1",
+      "TIMACT_MS":37708132,
+      "TOT_SIZE":0
+    },
+    "ID":2,
+    "Key":{
+      "Name":"MRN_STORY",
+      "Service":"ELEKTRON_DD"
+    },
+    "PermData":"AwhCEAAc",
+    "Qos":{
+      "Rate":"TickByTick",
+      "Timeliness":"Realtime"
+    },
+    "SeqNumber":32240,
+    "State":{
+      "Data":"Ok",
+      "Stream":"Open",
+      "Text":"All is well"
+    },
+    "Type":"Refresh"
+  }
+]
+```
+
+### Receive Update message and assemble News 
+```
+RECEIVED: 
+[
+  {
+    "DoNotCache":true,
+    "DoNotConflate":true,
+    "DoNotRipple":true,
+    "Domain":"NewsTextAnalytics",
+    "Fields":{
+      "ACTIV_DATE":"2019-07-25",
+      "FRAGMENT":"H4sIAAAAAAAC/71UW2/bNhR+36840MNe5sq2HCcx0W5QbTfW4miO5bRp2mCgRcpmLZEaSVnJhv33HUpyUKAvfSgGCfwOb+c7PLd/PJrbiHnEkxGrx+xyUTOv59GKCS5TbjzyyYtXZBqvQ1xGKZrdx95jz9sq9oy3lrRUmkpYUC0QqKUGYpFTAeHBiiOFt1wbsQeGm9eqKJURRsBKaasylQsF8Fl+liFjwgolaY4aLE33BZfWQIqXthwyVUmGQq5q4k7vrS1Jv1+yzNe8skjgp6pwc8lrc0KfmvI38eZsNBhP0tE2G/DRxfhs+HP1ptKSuBNFTr66T4LBcDK4CMbEeeLc3mbLD47t/2KcTB4Wye7E6L6ZqrY5hzQX6QGUBLvncLdeAt2qIwer4Ch43axSbUWac3+Vc2o4SGVxf08tCIlkklugKQbTgDCg+V+V0Jz5EGXwrCrgTyXXTbSh1AoZi+60kLtG+0lJD8pWf6qkqXLrbmvAjVrpA1BWCCmM1dQqDfhbnu6lSDGmpioxS6x705JabiwcXVbgk1QGIVMY4zBFajRYc8q4bu1EF2EeMM5e3rqavYNM5Nz4ABu0LP9Gm7O309Flj9paKiQqybQqoItkXdc+dcRtHLViVWpNn7ZW9J0GY3kZ+Htb5M7ud/igNXd8DGKMZA+66ExP0VGSdwY0HmLckE+uWB7BAY5hEoXNBAdXTwjJxm3MbzcfEZbNzv0qufJ/v36Er1j5kUoLKy0wRJHMlC6oq5bvsIG87vT9Cli+mdDGTvFx+AqsXZd+rwYXr4LxZjggowkZXfrn5+cPeHKPDsjRaz+kwlGfcHwvDebPNu2DKLhYTy7vlju1+uW5vzCjjTpmwTWbpA+35vj28JT9fThzt6Wx1HWjPzLsR9h8cip3Fd0584RrVwVmK043zyUuBTgXRTfxLH+y/RLNlXgOA30UmBu4Hieulbm1aptYaitsdR7SWFIZik7FHVNtv3BMC9cDQzLElatuDMYtzK8bnEYNtJMExxsyvJ01OApaWDQQvm8gvmvAOXpNugC57hqQMJmH8UmMwpN0vwqnrTy9WcUfW3F+s77qJEygVopmLS5Pws0Ga6YV4/mHdSut591FZHthwVx0fd3SA0+wRbh+4JFhz6v0DmXs9aOe1xXad6TQvz/9B7krulxaBgAA",
+      "FRAG_NUM":1,
+      "GUID":"Idw5d8Hwd_1907252I27R98ULgoP+y/Hs3Tovf2Kd9cZQsvBkxfzk4",
+      "MRN_SRC":"HK1_PRD_A",
+      "MRN_TYPE":"STORY",
+      "MRN_V_MAJ":"2",
+      "MRN_V_MIN":"10",
+      "TIMACT_MS":38378746,
+      "TOT_SIZE":882
+    },
+    "ID":2,
+    "Key":{
+      "Name":"MRN_STORY",
+      "Service":"ELEKTRON_DD"
+    },
+    "PermData":"AwhCEBkrEiFLEkMM",
+    "SeqNumber":32350,
+    "Type":"Update",
+    "UpdateType":"Unspecified"
+  }
+]
+FRAGMENT length = 882
+decompress News FRAGMENT(s) for GUID  Idw5d8Hwd_1907252I27R98ULgoP+y/Hs3Tovf2Kd9cZQsvBkxfzk4
+News = {'altId': 'nIdw5d8Hwd', 'audiences': ['NP:CNRA', 'NP:IDXN'], 'body': 'Laporan Harian atas Nilai Aktiva Bersih dan Komposisi Portofolio  \n\nAdditional attachments can be found below:\n\nhttp://pdf.reuters.com/pdfnews/pdfnews.asp?i=43059c3bf0e37541&u=urn:newsml:reuters.com:20190725:nIdw6tQfLW\n\n\n\nhttp://pdf.reuters.com/pdfnews/pdfnews.asp?i=43059c3bf0e37541&u=urn:newsml:reuters.com:20190725:nIdw99ZHSg\n\n\n\n\n\nDouble click on the URL above to view the article.Please note that internet access is required. If you experience problem accessing the internet, please consult your network administrator or technical support\n\nLatest version of Adobe Acrobat reader is recommended to view PDF files.  The latest version of the reader can be obtained from http://www.adobe.com/products/acrobat/readstep2.html\n\nFor Related News, Double Click on one of these codes:[IDXN] [IDX] [ASIA] [ID] [CNRA] [STX] [EQTY] [LID] [XPSG.JK] \n\nFor Relevant Price Information, Double Click on one of these code:<XPSG.JK> ', 'firstCreated': '2019-07-25T10:39:38.666Z', 'headline': 'Laporan Harian atas Nilai Aktiva Bersih dan Komposisi Portofolio  ', 'id': 'Idw5d8Hwd_1907252I27R98ULgoP+y/Hs3Tovf2Kd9cZQsvBkxfzk4', 'instancesOf': [], 'language': 'id', 'messageType': 2, 'mimeType': 'text/plain', 'provider': 'NS:IDX', 'pubStatus': 'stat:usable', 'subjects': ['A:1', 'G:1', 'G:25', 'G:2EK', 'G:CI', 'G:K', 'G:S', 'M:1QD', 'M:32', 'M:3H', 'M:AV', 'M:NU', 'M:Z', 'R:XPSG.JK', 'N2:ASEAN', 'N2:ASIA', 'N2:ASXPAC', 'N2:CMPNY', 'N2:EMRG', 'N2:EQTY', 'N2:ID', 'N2:LID', 'N2:MTPDF', 'N2:NEWR', 'N2:REG', 'N2:SEASIA', 'N2:STX'], 'takeSequence': 1, 'urgency': 3, 'versionCreated': '2019-07-25T10:39:38.666Z'}
+```
 
 ## References
 * [Refinitiv Elektron SDK Family page](https://developers.refinitiv.com/elektron) on the [Refinitiv Developer Community](https://developers.thomsonreuters.com/) web site.
 * [Refinitiv Elektron WebSocket API page](https://developers.refinitiv.com/websocket-api).
 * [Developer Webinar Recording: Introduction to Electron WebSocket API](https://www.youtube.com/watch?v=CDKWMsIQfaw).
 * [Machine Readable News (MRN) & N2_UBMS Comparison and Migration Guide](https://developers.refinitiv.com/article/machine-readable-news-mrn-n2_ubms-comparison-and-migration-guide).
 * [Introduction to Machine Readable News (MRN) with Elektron Message API (EMA)](https://developers.refinitiv.com/article/introduction-machine-readable-news-mrn-elektron-message-api-ema).
+* [MRN Data Models and Elektron Implementation Guide](https://developers.refinitiv.com/elektron/elektron-sdk-java/docs?content=8736&type=documentation_item).
 * [Refinitiv-API-Samples/Example.WebSocketAPI.Javascript.NewsMonitor](https://github.com/Refinitiv-API-Samples/Example.WebSocketAPI.Javascript.NewsMonitor).
 
 For any question related to this example or Elektron WebSocket API, please use the Developer Community [Q&A Forum](https://community.developers.refinitiv.com/spaces/152/websocket-api.html).

mrn_console_app.py

@@ -28,7 +28,6 @@
 position = socket.gethostbyname(socket.gethostname())
 mrn_domain = 'NewsTextAnalytics'
 mrn_item = 'MRN_STORY'
-#mrn_item = 'MRN_TRNA'
 
 # Global Variables
 web_socket_app = None
@@ -66,8 +65,6 @@ def processRefresh(ws, message_json):
 
 
 def processMRNUpdate(ws, message_json):  # process incoming News Update messages
-    #print("RECEIVED: Update Message")
-    # print(message_json)
 
     fields_data = message_json["Fields"]
     # Dump the FieldList first (for informational purposes)
@@ -88,8 +85,6 @@ def processMRNUpdate(ws, message_json):  # process incoming News Update messages
         #print("FRAG_NUM = %d" % frag_num)
         #print("MRN_SRC = %s" % mrn_src)
 
-        #fragment_decoded = base64.b64decode(fragment)
-
         if frag_num > 1:  # We are now processing more than one part of an envelope - retrieve the current details
             guid_index = next((index for (index, d) in enumerate(
                 _news_envelopes) if d["guid"] == guid), None)
@@ -100,7 +95,7 @@ def processMRNUpdate(ws, message_json):  # process incoming News Update messages
 
                 #print("fragment before merge = %d" % len(envelop["data"]["fragment"]))
 
-                # Merge incoming data to existing envelop and getting FRAGMENT and TOT_SIZE data to local variables
+                # Merge incoming data to existing news envelop and getting FRAGMENT and TOT_SIZE data to local variables
                 fragment = envelop["data"]["fragment"] = envelop["data"]["fragment"] + fragment
                 envelop["data"]["frag_num"] = frag_num
                 tot_size = envelop["data"]["tot_size"]
@@ -110,7 +105,7 @@ def processMRNUpdate(ws, message_json):  # process incoming News Update messages
                 # The multiple fragments news are not completed, waiting.
                 if tot_size != len(fragment):
                     return None
-                # The multiple fragments news are completed, delete assoiclate GUID dictionary
+                # The multiple fragments news are completed, delete assoiclate GUID envelop
                 elif tot_size == len(fragment):
                     del _news_envelopes[guid_index]
             else:
@@ -120,9 +115,10 @@ def processMRNUpdate(ws, message_json):  # process incoming News Update messages
         else:  # FRAG_NUM = 1 The first fragment
             tot_size = int(fields_data["TOT_SIZE"])
             print("FRAGMENT length = %d" % len(fragment))
-            if tot_size != len(fragment):  # Completed News
+            # The fragment news is not completed, waiting and add this news data to envelop object.
+            if tot_size != len(fragment):
                 print("Add new fragments to news envelop for guid %s" % guid)
-                _news_envelopes.append({
+                _news_envelopes.append({  # the envelop object is a Python dictionary with GUID as a key and other fields are data
                     "guid": guid,
                     "data": {
                         "fragment": fragment,
@@ -189,7 +185,6 @@ def process_message(ws, message_json):
 
 def process_login_response(ws, message_json):
     """ Send item request """
-    # send_market_price_request(ws)
     send_mrn_request(ws)
 
 
@@ -257,15 +252,15 @@ def on_open(ws):
     # Get command line parameters
     try:
         opts, args = getopt.getopt(sys.argv[1:], "", [
-                                   "help", "hostname=", "port=", "app_id=", "user=", "position="])
+                                   "help", "hostname=", "port=", "app_id=", "user=", "position=", "ric="])
     except getopt.GetoptError:
         print(
-            'Usage: market_price.py [--hostname hostname] [--port port] [--app_id app_id] [--user user] [--position position] [--help]')
+            'Usage: market_price.py [--hostname hostname] [--port port] [--app_id app_id] [--user user] [--position position] [--ric news RIC name] [--help]')
         sys.exit(2)
     for opt, arg in opts:
         if opt in ("--help"):
             print(
-                'Usage: market_price.py [--hostname hostname] [--port port] [--app_id app_id] [--user user] [--position position] [--help]')
+                'Usage: market_price.py [--hostname hostname] [--port port] [--app_id app_id] [--user user] [--position position] [--ric news RIC name] [--help]')
             sys.exit(0)
         elif opt in ("--hostname"):
             hostname = arg
@@ -277,6 +272,13 @@ def on_open(ws):
             user = arg
         elif opt in ("--position"):
             position = arg
+        elif opt in ("--ric"):
+            if arg not in ["MRN_STORY", "MRN_TRNA", "MRN_TRNA_DOC", "MRN_TRSI"]:
+                print(
+                    "The supported MRN RIC names are MRN_STORY or MRN_TRNA or MRN_TRNA_DOC or MRN_TRSI only")
+                sys.exit(2)
+            else:
+                item = arg
 
     # Start websocket handshake
     ws_address = "ws://{}:{}/WebSocket".format(hostname, port)
@@ -294,6 +296,6 @@ def on_open(ws):
 
     try:
         while True:
-            time.sleep(600)
+            time.sleep(1)
     except KeyboardInterrupt:
         web_socket_app.close()