[ TLS Handshake Data Collected By Lumen ]

Data Format

The data files contain raw binaries of handshake messages (client hellos and server hellos, where applicable) and certificates (full certificate chains, where available) along with some contextual information (excluding session information) collected by Lumen from November 2015 to June 2017.
Additionally, it has device-level information (brand, board, OS version, SDK version, and default and supported fingerprints, where available).

• devices/contains information (in JSON format) about 1378 devices that are unique to this dataset, spanning most versions of Android from 4.x to 7.x.
  
The device information is aslo included in all handshake records where it was collected.
Here's a sample device record:


				
{u'default_ciphers': [u'SSL_RSA_WITH_RC4_128_MD5',
                      u'SSL_RSA_WITH_RC4_128_SHA',
                      u'TLS_RSA_WITH_AES_128_CBC_SHA',
                      u'TLS_RSA_WITH_AES_256_CBC_SHA',
                      u’[list_truncated]'],
 u'default_versions': [u'SSLv3', u'TLSv1'],
 u'device_info': {u'board': u'S1',
                  u'brand': u'zwx',
                  u'osversion': u'4.2.2|REL',
                  u'sdkversion': u'17'},
 u'supported_ciphers': [u'SSL_RSA_WITH_RC4_128_MD5',
                        u'SSL_RSA_WITH_RC4_128_SHA',
                        u'TLS_RSA_WITH_AES_128_CBC_SHA',
                        u'TLS_RSA_WITH_AES_256_CBC_SHA',
                        u’[list_truncated]'],
 u'supported_versions': [u'SSLv3', u'TLSv1', u'TLSv1.1', u'TLSv1.2']}
				
				

• handshakes/ contains all handshake traces (in BSON format), separated using sub-folders named by the date the handshakes occured.
• The cipher field in flow headers refers to the cipher suite that was used in the connection.
  
• No browser fingerprints are collected to preserve user privacy.
  
• Timestamps are fuzzed from nano-seconds down to dates.
  
• You can use our open-source TLS handshake record parser on GitHub to parse binary handshake records.
  
Here’s an example of what a handshake record looks like (this is then encoded as BSON and compressed for release):



{   u'certs': [   u'9f76ba18faf26aa5d4413ddc8f6d4f7a71f64924cd9e9fab18884a96',
                  u'd449059581f1d54b2104db6b4ced2fc0121a38a9093c55538338e21d',
                  u'65b75171e011e1b884c5ccb143bab6ca9943c3d986d4126832613a4f'],
    u'client_hello': “[binary data]",
    u'flow_headers': {   u'APPVERSION': u'22',
                         u'CIPHER': u'TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256',
                         u'CLIENTHELLO_LEN': u'517',
                         u'DSTIP': u'52.84.171.87',
                         u'DSTPORT': u'443',
                         u'PACKAGE': u'com.android.location.fused',
                         u'PROXIED': u'0',
                         u'SERVERCERTS_LEN': u'3397',
                         u'SERVERHELLO_LEN': u'66',
                         u'SNI': u'lepodownload.mediatek.com',
                         u'SRCPORT': u'58066',
                         u'TIMESTAMP': u'2017-08-16'},
    u'os_params': {   u'default_ciphers': [   u'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256',
                                              u'TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384',
                                              u'TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256',
                                              u’[list_truncated]'],
                      u'default_versions': [   u'SSLv3',
                                               u'TLSv1',
                                               u'TLSv1.1',
                                               u'TLSv1.2'],
                      u'device_info': {   u'board': u'MT6580',
                                          u'brand': u'plus_one_japan_ltd',
                                          u'osversion': u'5.1|REL',
                                          u'sdkversion': u'22'},
                      u'supported_ciphers': [   u'SSL_RSA_WITH_RC4_128_MD5',
                                                u'SSL_RSA_WITH_RC4_128_SHA',
                                                u'TLS_RSA_WITH_AES_128_CBC_SHA',
                                                u'TLS_RSA_WITH_AES_256_CBC_SHA',
                                                u’[list_truncated]'],
                      u'supported_versions': [   u'SSLv3',
                                                 u'TLSv1',
                                                 u'TLSv1.1',
                                                 u'TLSv1.2']},
    u'server_hello': '[binary_data]'}
				
				

• certs/ holds server certificates (in binary DER format). The filenames are hashes of the certificate contents, and are referenced in handshake traces using these hashes in an ordered list where the full certificate chain was available.
  You can load these certificates using OpenSSL's command-line tool:
  openssl x509 -text -inform der -in [filename].crt

License Agreement

Access to this data is subject to agreeing to the following Acceptable Use Agreement (inspired by CAIDA's Acceptable Use Agreement).

HAYSTACK ACCEPTABLE USE AGREEMENT for TLS HANDSHAKE DATA COLLECTED BY LUMEN Usage of this dataset is subject to agreeing to the following terms. LICENSE Haystack team's authorization to access the data grants You a limited, non-exclusive, non-transferable, non-assignable, and terminable license to copy, modify, and use the data in accordance with this Public Agreement. No license is granted for any other purpose and there are no implied licenses in this Agreement. Nothing in this License is intended to limit any rights You may have arising from fair use or due to other limitations on Haystack's exclusive rights under copyright law or other applicable laws. Haystack has the authority and reserves the right, in its sole discretion, to discontinue further access and use to anyone who violates this AUA. If You create a publication (including web pages, papers published by a third party, and publicly available presentations) using data from this dataset, You should cite the corresponding paper as follows: Abbas Razaghpanah, Arian Akhavan Niaki, Narseo Vallina-Rodriguez, Srikanth Sundaresan, Johanna Amann, and Phillipa Gill. 2017. "Studying TLS Usage in Android Apps", ACM International Conference on emerging Networking EXperiments and Technologies (CoNEXT) 2017 We encourage You to provide Haystack with a copy of (or a link to) the publication. We use this information in reports to our funding agencies. DISCLAIMER OF WARRANTIES. HAYSTACK USES ITS BEST EFFORTS TO PROVIDE DATA IN ACCORDANCE WITH ETHICAL PRINCIPLES AND SCIENTIFIC INTEGRITY. HOWEVER, THE DATA PROVIDED HEREIN IS ON AN "AS IS" BASIS. NEITHER HAYSTACK, ITS RESEARCHERS, RESEARCH PARTNERS, LICENSORS, AND DATA PROVIDERS, NOR THE UNIVERSITY OF CALIFORNIA AND ITS TRUSTEES, OFFICERS, EMPLOYEES, AND AGENTS MAKE ANYWARRANTY, EITHER IMPLIED OR EXPRESS, OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, INCLUDING, BUT NOT LIMITED TO, THE ACCURACY, TIMELINESS, COMPLETENESS, RELIABILITY, OR AVAILABILITY OF CAIDA DATA, APPLICATIONS, OR SERVICES ACCESSIBLE THROUGH OR MADE AVAILABLE BY HAYSTACK. LIMITATION OF LIABILITY. TO THE EXTENT ALLOWED BY LAW, IN NO EVENT SHALL HAYSTACK AND THE UNIVERSITY OF CALIFORNIA BE LIABLE TO YOU OR ANY THIRD PARTY FOR ANY INDIRECT, CONSEQUENTIAL, INCIDENTAL, SPECIAL OR PUNITIVE DAMAGES, ARISING FROM YOUR USE OF THE DATA. If You have any questions about the data or about this Public Agreement, please email info@haystack.mobi.

Download The Data

You can obtain this dataset from Zenodo (DOI: 10.5281/zenodo.2224273).