[ TLS Handshake Data Collected By Lumen (CoNEXT 2017) ]

Data Format

The data files contain raw binaries of handshake messages (client hellos and server hellos, where applicable) and certificates (full certificate chains, where available) along with some contextual information (excluding session information) collected by Lumen from November 2015 to June 2017.
Additionally, it has device-level information (brand, board, OS version, SDK version, and default and supported fingerprints, where available).

• devices/contains information (in JSON format) about 1378 devices that are unique to this dataset, spanning most versions of Android from 4.x to 7.x.
  
The device information is aslo included in all handshake records where it was collected.
Here's a sample device record:


				
{u'default_ciphers': [u'SSL_RSA_WITH_RC4_128_MD5',
                      u'SSL_RSA_WITH_RC4_128_SHA',
                      u'TLS_RSA_WITH_AES_128_CBC_SHA',
                      u'TLS_RSA_WITH_AES_256_CBC_SHA',
                      u’[list_truncated]'],
 u'default_versions': [u'SSLv3', u'TLSv1'],
 u'device_info': {u'board': u'S1',
                  u'brand': u'zwx',
                  u'osversion': u'4.2.2|REL',
                  u'sdkversion': u'17'},
 u'supported_ciphers': [u'SSL_RSA_WITH_RC4_128_MD5',
                        u'SSL_RSA_WITH_RC4_128_SHA',
                        u'TLS_RSA_WITH_AES_128_CBC_SHA',
                        u'TLS_RSA_WITH_AES_256_CBC_SHA',
                        u’[list_truncated]'],
 u'supported_versions': [u'SSLv3', u'TLSv1', u'TLSv1.1', u'TLSv1.2']}
				
				

• handshakes/ contains all handshake traces (in BSON format), separated using sub-folders named by the date the handshakes occured.
• The cipher field in flow headers refers to the cipher suite that was used in the connection.
  
• No browser fingerprints are collected to preserve user privacy.
  
• Timestamps are fuzzed from nano-seconds down to dates.
  
• You can use our open-source TLS handshake record parser on GitHub to parse binary handshake records.
  
Here’s an example of what a handshake record looks like (this is then encoded as BSON and compressed for release):



{   u'certs': [   u'9f76ba18faf26aa5d4413ddc8f6d4f7a71f64924cd9e9fab18884a96',
                  u'd449059581f1d54b2104db6b4ced2fc0121a38a9093c55538338e21d',
                  u'65b75171e011e1b884c5ccb143bab6ca9943c3d986d4126832613a4f'],
    u'client_hello': “[binary data]",
    u'flow_headers': {   u'APPVERSION': u'22',
                         u'CIPHER': u'TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256',
                         u'CLIENTHELLO_LEN': u'517',
                         u'DSTIP': u'52.84.171.87',
                         u'DSTPORT': u'443',
                         u'PACKAGE': u'com.android.location.fused',
                         u'PROXIED': u'0',
                         u'SERVERCERTS_LEN': u'3397',
                         u'SERVERHELLO_LEN': u'66',
                         u'SNI': u'lepodownload.mediatek.com',
                         u'SRCPORT': u'58066',
                         u'TIMESTAMP': u'2017-08-16'},
    u'os_params': {   u'default_ciphers': [   u'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256',
                                              u'TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384',
                                              u'TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256',
                                              u’[list_truncated]'],
                      u'default_versions': [   u'SSLv3',
                                               u'TLSv1',
                                               u'TLSv1.1',
                                               u'TLSv1.2'],
                      u'device_info': {   u'board': u'MT6580',
                                          u'brand': u'plus_one_japan_ltd',
                                          u'osversion': u'5.1|REL',
                                          u'sdkversion': u'22'},
                      u'supported_ciphers': [   u'SSL_RSA_WITH_RC4_128_MD5',
                                                u'SSL_RSA_WITH_RC4_128_SHA',
                                                u'TLS_RSA_WITH_AES_128_CBC_SHA',
                                                u'TLS_RSA_WITH_AES_256_CBC_SHA',
                                                u’[list_truncated]'],
                      u'supported_versions': [   u'SSLv3',
                                                 u'TLSv1',
                                                 u'TLSv1.1',
                                                 u'TLSv1.2']},
    u'server_hello': '[binary_data]'}
				
				

• certs/ holds server certificates (in binary DER format). The filenames are hashes of the certificate contents, and are referenced in handshake traces using these hashes in an ordered list where the full certificate chain was available.
  You can load these certificates using OpenSSL's command-line tool:
  openssl x509 -text -inform der -in [filename].crt

License Agreement

Download The Data