pcapng_utils.tshark.protocols package

Submodules

pcapng_utils.tshark.protocols.http1 module

class pcapng_utils.tshark.protocols.http1.Http1Traffic(traffic)

Bases: object

Class to represent HTTP1 network traffic.

This class is the entry point for parsing HTTP1 network traffic.

The format of JSON data from tshark is as follows for a single HTTP request:

• GET /spi/v2/platforms/ HTTP/1.1rn: Contains the HTTP method, URI, and version.

• http.request.version: The HTTP version used.

• http.request.line: A list of HTTP headers sent with the request.

• http.host: The Host header value.

• http.request.full_uri: The full URI including the scheme (e.g., https).

• http.request_number: The request number.

• http.response_in: The response number associated with this request.

The format of JSON data from tshark is as follows for a single HTTP response:

• HTTP/1.1 200 OKrn: Contains the HTTP version, status code, and status phrase.

• http.content_type: The Content-Type header value.

• http.response.line: A list of HTTP headers sent with the response.

• http.content_encoding: The Content-Encoding header value.

• http.response_number: The response number.

• http.time: The time taken for the response.

• http.request_in: The request number associated with this response.

• http.response_for.uri: The URI for which this response is generated.

• http.file_data_raw: The data in hexadecimal format (requires -x flag).

get_har_entries()

Convert the HTTP network traffic to HTTP Archive (HAR) format. :return: the HTTP network traffic in HAR format

Return type:

list[dict[str, Any]]

parse_traffic()

Parse the HTTP network traffic and extract the request-response pairs.

Identify each HTTP request and its associated HTTP response by following these steps:

1. Iterate through packets: It loops through all packets obtained from the traffic object.

2. Check protocols: It checks if the packet contains the http protocol by examining the frame.protocols field.

3. Identify http requests: It checks if the packet contains an HTTP request by looking for the http.request key in the http layer.

4. Find associated response: If the packet is an HTTP request and contains the http.response_in key, it retrieves the corresponding response packet using the get_packet_by_number method with the response number.

5. Create conversation: It creates an HttpConversation object with the request and response packets and appends it to the conversations list.

class pcapng_utils.tshark.protocols.http1.HttpConversation(request_layers, response_layers)

Bases: object

Class to represent an HTTP conversation composed of a request and a response.

to_har()

Convert the HTTP conversation to HTTP Archive (HAR) format. :return: the HTTP conversation (request and response) in HAR format

Return type:

dict[str, Any]

property community_id: str

property waiting_duration: float

class pcapng_utils.tshark.protocols.http1.HttpRequest(packet)

Bases: HttpRequestResponse

Class to represent an HTTP request.

to_har()

Convert the HTTP request to HTTP Archive (HAR) format. :return: the HTTP request in HAR format

Return type:

dict[str, Any]

property http_version_method: tuple[str, str]

Get the HTTP version & method from the packet data. :return: tuple with HTTP version & method

property raw_headers: list[bytes]

property sending_duration: float

property uri: str

class pcapng_utils.tshark.protocols.http1.HttpRequestResponse(packet)

Bases: ABC

Base class for HTTP request and response packets. It wraps the packet data and provides methods to access the relevant information.

FALLBACK_CONTENT_TYPE: ClassVar[str] = 'application/octet-stream'

property common_har_props: dict[str, Any]

property community_id: str

property content_length: int

property content_type: str

property dst_host: str

property dst_ip: str

property dst_port: int

property frame_nb: int

property header_length: int

property headers: list[dict[str, str]]

property http_layer: dict[str, Any]

property ip_version_and_layer: tuple[str, dict[str, Any]]

packet: Mapping[str, Any]

property payload: Payload

abstract property raw_headers: Sequence[bytes]

property src_host: str

property src_ip: str

property src_port: int

property timestamp: float

class pcapng_utils.tshark.protocols.http1.HttpResponse(packet)

Bases: HttpRequestResponse

Class to represent an HTTP response.

to_har()

Convert the HTTP response to HTTP Archive (HAR) format. :return: the HTTP response in HAR format

property http_version_status_code_message: tuple[str, int, str]

Retrieve the HTTP version & status code & message. :return: tuple with HTTP version, status code and message

property raw_headers: list[bytes]

property receiving_duration: float

pcapng_utils.tshark.protocols.http2 module

class pcapng_utils.tshark.protocols.http2.Http2Helper

Bases: object

classmethod get_headers_and_data(substreams)

Identify the headers and data substreams and return them.

The substreams are identified by their types: - Headers substream: type 1 - Data substream: type 0 We ignore the rest of the substreams.

Note that (flag & 0x01) identify the end of stream, usually it happens for a data-stream but it may also happen for a header-stream (trailers in gRPC), or even never happen.

Parameters:

substreams (Sequence[Http2Substream]) – the substreams of a HTTP2 stream

Returns:

the headers and data substreams regardless if it is a request or a response

static get_data(data_substreams)

Extract the data from the substreams (precondition: all substreams are data substreams).

Parameters:

data_substreams (Sequence[Http2Substream]) – the data substreams to be analyzed

Returns:

the reassembled data

Return type:

Payload

static get_headers(substream)

Extract the headers from the substream (precondition: it is a header substream).

Parameters:

substream (Http2Substream) – the substream to be analyzed

Returns:

the headers of the substream

Return type:

list[Mapping[str, str]]

static substream_is_data(substream)

Returns whether substream is a data substream.

Return type:

bool

static substream_is_header(substream)

Returns whether substream is a header substream.

Return type:

bool

static to_har(message)

Convert the HTTP2 request or response to a HAR entry.

<!> Some HTTP2 responses are missing

Parameters:

message (Http2RequestResponse) – the HTTP2 request or response to be converted

Returns:

the HAR entry for the HTTP2 request or response

Return type:

dict[str, Any]

class pcapng_utils.tshark.protocols.http2.Http2Request(substreams)

Bases: Http2RequestResponse

Class to represent a HTTP2 request. It contains the headers and data of the request.

property uri: str

class pcapng_utils.tshark.protocols.http2.Http2RequestResponse(substreams)

Bases: object

Base class to represent a HTTP2 request or response. It contains the headers and data of the request or response. Implements the common properties of a HTTP2 request or response.

get_duration_ms()

Return type:

float

FALLBACK_CONTENT_TYPE: ClassVar[str] = 'application/octet-stream'

property body_length: int

This is number of compressed bytes (if any compression)

• http2.length is also populated for header substreams

• we do NOT always have the http2.body.fragments -> http2.body.reassembled.length

property community_id: str

property content_type: str

property dst_host: str

property dst_ip: str

property dst_port: int

property frames_nbs: Sequence[int]

property header_length: int

property headers_map: dict[str, str]

property http_method: str

property http_status: int

property http_version: str

property src_host: str

property src_ip: str

property src_port: int

property timestamp: float

class pcapng_utils.tshark.protocols.http2.Http2Response(substreams)

Bases: Http2RequestResponse

Class to represent a HTTP2 response. It contains the headers and data of the response.

<!> May be empty for convenience (response never received)

class pcapng_utils.tshark.protocols.http2.Http2Stream(tcp_stream_id, http2_stream_id, community_id)

Bases: object

Class to represent an entire HTTP2 stream (multiple substreams). It contains the request and response objects. Http2Stream represents a single HTTP2 stream that can contain multiple substreams as follows:

+-------------------------------------- (tcp stream, http2 stream)
| Http2SubStream 1    | Request headers (type: 1)
| Http2SubStream ...  | Request data (type: 0, flags: 0x0) - partial data
| Http2SubStream 3    | Request data (type: 0, flags: 0x1) - end of stream, contains reassembled data
| (Http2SubStream 4   | Request trailers (type: 1))
+--------------------------------------
| Http2SubStream 5    | Response headers (type: 1)
| Http2SubStream ...  | Response data (type: 0, flags: 0x0) - partial data
| Http2SubStream 7    | Response data (type: 0, flags: 0x1) - end of stream, contains reassembled data
| (Http2SubStream 8   | Response trailers (type: 1))
+--------------------------------------

Each HTTP2 stream is uniquely identified by a tuple (tcp stream index, http2 stream index)
and contains both request and response objects.

classmethod get_raw_data(raw_http2_substreams)

Find the data in the substreams.

Parameters:

raw_http2_substreams (Sequence[Mapping[str, Any]]) – the data substreams to be analyzed

Returns:

the raw reassembled data if it exists, otherwise an empty Payload

Return type:

Payload

__init__(tcp_stream_id, http2_stream_id, community_id)

Defines a HTTP2 stream for the given TCP stream and HTTP2 stream.

Parameters:

• tcp_stream_id (int) – the ID of the TCP stream

• http2_stream_id (int) – the ID of the HTTP2 stream

• community_id (str) – the community ID (i.e. TCP|UDP + ips & ports) for this conversation

append(raw_http2_substream, all_layers)

Append a new substream to the HTTP2 stream.

Parameters:

• substream – the substream to be added

• frame – the frame containing the substream. A frame can contain multiple substreams.

har_entry()

Create a HAR entry for the HTTP2 stream. It contains the request and response objects.

Returns:

the HAR entry for the HTTP2 stream

Return type:

dict[str, Any] | None

process()

Process the substreams and create the request and response objects accordingly. Substreams are processed in order, the first substreams are request headers, followed by request data, and finally the response headers and data. The reassembled data is used to create the request and response objects.

Request substreams are identified by the presence of the ‘http2.request.full_uri’ key in the raw stream. If no response substream is found, the request object is created with the first substreams.

It retrieves the source and destination IP addresses from the first substream to identify the substreams that belong to the request. The response substreams are identified by checking their source IP address matches the destination IP address of the first substream.

property id: tuple[int, int]

property waiting_duration: float

class pcapng_utils.tshark.protocols.http2.Http2Substream(raw_http2_substream, all_layers)

Bases: object

Class to represent a HTTP2 substream. It contains the layers of the packet and the metadata of the substream. Wrap the raw HTTP2 substream and the frame layers to extract the relevant information.

KEEP_LAYERS: ClassVar[Set[str]] = {'frame', 'ip', 'ipv6', 'tcp'}

property community_id: str

property dst_host: str

property dst_ip: str

property dst_port: int

property frame_layer: dict[str, Any]

property frame_nb: int

property http2_flags: int

property http2_type: int

property ip_version_and_layer: tuple[str, dict[str, Any]]

property raw_headers: list[dict[str, Any]]

property src_host: str

property src_ip: str

property src_port: int

property timestamp: float

class pcapng_utils.tshark.protocols.http2.Http2Traffic(traffic)

Bases: object

Class to represent the HTTP2 traffic. It contains the HTTP2 streams and the parsed traffic data.

In HTTP/2, frames are the smallest unit of communication. Each frame has a specific type and can have associated flags.

HTTP/2 frame types and flags:

HTTP/2 Frame Types:

• DATA (0x0): carries arbitrary, variable-length sequences of octets associated with a stream.

• HEADERS (0x1): used to open a stream and carry a header block fragment.

• PRIORITY (0x2): specifies the sender-advised priority of a stream.

• RST_STREAM (0x3): abruptly terminates a stream.

• SETTINGS (0x4): used to communicate configuration parameters.

• PUSH_PROMISE (0x5): used to notify the peer endpoint in advance of streams the sender intends to initiate.

• PING (0x6): used to measure round-trip time and ensure the connection is still active.

• GOAWAY (0x7): informs the peer to stop creating streams on this connection.

• WINDOW_UPDATE (0x8): used to implement flow control.

• CONTINUATION (0x9): used to continue a sequence of header block fragments.

HTTP/2 Frame Flags:

• END_STREAM (0x1): indicates that the frame is the last one for the current stream.

• END_HEADERS (0x4): indicates that the frame contains the entire header block.

• PADDED (0x8): indicates that the frame contains padding.

• PRIORITY (0x20): indicates that the frame contains priority information.

TCP stream ID and the HTTP/2 stream ID The TCP stream ID identifies a unique TCP connection. Each TCP connection is assigned a unique stream ID, which is used to track the packets that belong to that connection. The HTTP/2 stream ID, within a single TCP connection, multiple HTTP/2 streams can exist. Each HTTP/2 stream is identified by a unique stream ID within the context of that TCP connection. These stream IDs are used to multiplex multiple HTTP/2 requests and responses over a single TCP connection.

A single TCP stream (connection) can contain multiple HTTP/2 streams. Each HTTP/2 stream is uniquely identified within the context of its TCP stream. The combination of the TCP stream ID and the HTTP/2 stream ID uniquely identifies an HTTP/2 stream within the network traffic.

get_har_entries()

Convert the HTTP2 traffic to HTTP Archive (HAR) format.

Returns:

the HTTP2 traffic in HAR format

Return type:

list[dict[str, Any]]

get_http2_streams()

parse_traffic()

Parse the traffic and extract the HTTP2 streams. It creates a dictionary for each HTTP2 stream. Each key is a tuple with the TCP stream ID and the HTTP2 stream ID.

Identify each HTTP2 request and its associated HTTP2 response by following these steps:

1. Iterate through packets: it loops through all packets obtained from the traffic object.

2. Extract protocols: for each packet, it extracts the protocols from the frame.protocols field.

3. Check for HTTP2 protocol: it checks if the packet contains the http2 protocol.

4. Extract the TCP stream ID: it retrieves the TCP stream ID from the tcp.stream field.

5. Handle HTTP2 layer: it ensures the http2 layer is a list of HTTP2 stream objects.

6. Process each HTTP2 stream: for each HTTP2 stream in the http2 layer:

   • extract stream information: it retrieves the stream type and stream ID.

   • filter relevant streams: it ignores streams that are not data (type 0) or headers (type 1).

   • create or update stream pair: it creates a new tuple of (tcp_stream_id, http2_stream_id) if it does not exist and appends the substream to the list.

7. Process streams: after assembling the HTTP2 streams, it processes each stream to create the request and response objects.

Module contents

class pcapng_utils.tshark.protocols.Http1Traffic(traffic)

Bases: object

Class to represent HTTP1 network traffic.

This class is the entry point for parsing HTTP1 network traffic.

The format of JSON data from tshark is as follows for a single HTTP request:

• GET /spi/v2/platforms/ HTTP/1.1rn: Contains the HTTP method, URI, and version.

• http.request.version: The HTTP version used.

• http.request.line: A list of HTTP headers sent with the request.

• http.host: The Host header value.

• http.request.full_uri: The full URI including the scheme (e.g., https).

• http.request_number: The request number.

• http.response_in: The response number associated with this request.

The format of JSON data from tshark is as follows for a single HTTP response:

• HTTP/1.1 200 OKrn: Contains the HTTP version, status code, and status phrase.

• http.content_type: The Content-Type header value.

• http.response.line: A list of HTTP headers sent with the response.

• http.content_encoding: The Content-Encoding header value.

• http.response_number: The response number.

• http.time: The time taken for the response.

• http.request_in: The request number associated with this response.

• http.response_for.uri: The URI for which this response is generated.

• http.file_data_raw: The data in hexadecimal format (requires -x flag).

get_har_entries()

Convert the HTTP network traffic to HTTP Archive (HAR) format. :return: the HTTP network traffic in HAR format

Return type:

list[dict[str, Any]]

parse_traffic()

Parse the HTTP network traffic and extract the request-response pairs.

Identify each HTTP request and its associated HTTP response by following these steps:

1. Iterate through packets: It loops through all packets obtained from the traffic object.

2. Check protocols: It checks if the packet contains the http protocol by examining the frame.protocols field.

3. Identify http requests: It checks if the packet contains an HTTP request by looking for the http.request key in the http layer.

4. Find associated response: If the packet is an HTTP request and contains the http.response_in key, it retrieves the corresponding response packet using the get_packet_by_number method with the response number.

5. Create conversation: It creates an HttpConversation object with the request and response packets and appends it to the conversations list.

class pcapng_utils.tshark.protocols.Http2Traffic(traffic)

Bases: object

Class to represent the HTTP2 traffic. It contains the HTTP2 streams and the parsed traffic data.

In HTTP/2, frames are the smallest unit of communication. Each frame has a specific type and can have associated flags.

HTTP/2 frame types and flags:

HTTP/2 Frame Types:

• DATA (0x0): carries arbitrary, variable-length sequences of octets associated with a stream.

• HEADERS (0x1): used to open a stream and carry a header block fragment.

• PRIORITY (0x2): specifies the sender-advised priority of a stream.

• RST_STREAM (0x3): abruptly terminates a stream.

• SETTINGS (0x4): used to communicate configuration parameters.

• PUSH_PROMISE (0x5): used to notify the peer endpoint in advance of streams the sender intends to initiate.

• PING (0x6): used to measure round-trip time and ensure the connection is still active.

• GOAWAY (0x7): informs the peer to stop creating streams on this connection.

• WINDOW_UPDATE (0x8): used to implement flow control.

• CONTINUATION (0x9): used to continue a sequence of header block fragments.

HTTP/2 Frame Flags:

• END_STREAM (0x1): indicates that the frame is the last one for the current stream.

• END_HEADERS (0x4): indicates that the frame contains the entire header block.

• PADDED (0x8): indicates that the frame contains padding.

• PRIORITY (0x20): indicates that the frame contains priority information.

TCP stream ID and the HTTP/2 stream ID The TCP stream ID identifies a unique TCP connection. Each TCP connection is assigned a unique stream ID, which is used to track the packets that belong to that connection. The HTTP/2 stream ID, within a single TCP connection, multiple HTTP/2 streams can exist. Each HTTP/2 stream is identified by a unique stream ID within the context of that TCP connection. These stream IDs are used to multiplex multiple HTTP/2 requests and responses over a single TCP connection.

A single TCP stream (connection) can contain multiple HTTP/2 streams. Each HTTP/2 stream is uniquely identified within the context of its TCP stream. The combination of the TCP stream ID and the HTTP/2 stream ID uniquely identifies an HTTP/2 stream within the network traffic.

get_har_entries()

Convert the HTTP2 traffic to HTTP Archive (HAR) format.

Returns:

the HTTP2 traffic in HAR format

Return type:

list[dict[str, Any]]

get_http2_streams()

parse_traffic()

Parse the traffic and extract the HTTP2 streams. It creates a dictionary for each HTTP2 stream. Each key is a tuple with the TCP stream ID and the HTTP2 stream ID.

Identify each HTTP2 request and its associated HTTP2 response by following these steps:

1. Iterate through packets: it loops through all packets obtained from the traffic object.

2. Extract protocols: for each packet, it extracts the protocols from the frame.protocols field.

3. Check for HTTP2 protocol: it checks if the packet contains the http2 protocol.

4. Extract the TCP stream ID: it retrieves the TCP stream ID from the tcp.stream field.

5. Handle HTTP2 layer: it ensures the http2 layer is a list of HTTP2 stream objects.

6. Process each HTTP2 stream: for each HTTP2 stream in the http2 layer:

   • extract stream information: it retrieves the stream type and stream ID.

   • filter relevant streams: it ignores streams that are not data (type 0) or headers (type 1).

   • create or update stream pair: it creates a new tuple of (tcp_stream_id, http2_stream_id) if it does not exist and appends the substream to the list.

7. Process streams: after assembling the HTTP2 streams, it processes each stream to create the request and response objects.