This is a copy of the discussion thread started on the GitHub. The documentation is only used as a backup reference to the original discussion thread.
As PyPCAPKit reaches its 16k lines of code and 800th commit, I figure it would be a better idea to record the project enchancement proposals here in the discussion thread. The proposals and/or notes will be documented and maintained here.
Pull requests for the existing proposals and any new ideas are highly welcomed and encouraged. Should you have any questions, please leave a note either in this thread or under the Q&A category discussions.
Wish you enjoy PyPCAPKit!!!
More Protocols, More!!!¶
As you may have noticed, there are some protocol-named files under the
NotImplemented folders. These protocols are what I planned to implement
but not yet done. Namely, grouped by each TCP/IP layer and ordered by protocol
Link Layer: DSL, EAPOL, FDDI, ISDN, PPP
Internet Layer: ECN, ESP, ICMP, ICMPv6, IGMP, NDP, Shim6
Transport Layer: DCCP, QUIC, RSVP, SCTP
Application Layer: BGP, DHCP, DHCPv6, DNS, IMAP, LDAP, MQTT, NNTP, NTP, ONC/RPC, POP, RIP, RTP, SIP, SMTP, SNMP, SSH, Telnet, TLS/SSL, XMPP
Specifically, I have attempted to implement ESP several years ago, and I abandoned the implementation in the NotImplemented folder due to some design flaws within PyPCAPKit at that time. But now, the protocol should be able to implement quite smoothly.
MH requires some help to
implement all the message data types, you can find more information in the
Also, for the existing protocols, I am looking for a helping hand to implement
the construction features, as defined in the
method. You can find some reference from the PCAP’s
As mentioned in #35, PyPCAPKit does not support parsing PCAPNG files with its builtin default engine at the moment – partly because I could not understand the file format specifications.
If you are to help with it, please refer to the implementation of PCAP format
Maybe Even Faster?¶
Based on my recent benchmarking, PyPCAPKit’s builtin default engine is only 4 times slower than Scapy and 10 times to DPKT. Considering the general overhead and verbose features provided by PyPCAPKit’s builtin default engine, such performance difference is acceptable.
However, there might still be a way to further accelerate the protocol
implementation – merge and concatenation
_read_xxxxxx methods within one
file.read(), such that we shall decrease the overall number of IO
calls and reduce the duplicated
struct.unpack() calls, etc. I am not yet
confident about the performance improvement, but this is the most efficient way
to accelerate PyPCAPKit at the moment, inspired from the implementation of
Scapy and DPKT themselves.
Specifically, as the following code from
_tsus = self._read_unpack(4, lilendian=True) _ilen = self._read_unpack(4, lilendian=True) _olen = self._read_unpack(4, lilendian=True)
we might be able to rewrite it as
_tsus, _ilen, _olen = self._read_fields(unpack(4, lilendian=True), unpack(4, lilendian=True), unpack(4, lilendian=True))
and the PoC of
_read_fields would be something like
def _read_fields(self, *fields: 'Field') -> 'tuple[Any, ...]': # built template fmt = ''.join(field.template for field in fields) len = sum([field.length for field in fields]) # read from buffer & do unpack buf = self._file.read(fmt) tmp = struct.unpack(fmt, buf) # do post-processing based on field-specific implementations ret =  for field, val in itertools.chain(fields, tmp): ret.append(field.post_process(val)) return ret
As PyPCAPKit now has the
pcapkit.utilities.logging.logger in place, I’m
expecting to fully extend its functionality in the entire module. Ideas and
contributions are welcomed to integrate the logging system into PyPCAPKit.
Although PyPCAPKit already has support for some popular PCAP parsing libraries, I’m expecting to extend the list of supported engines furthermore. The candidate engines include:
Implementation for support of new engines would include adding corresponding
handler methods and code blocks into
(see support for Scapy, DPKT, and/or PyShark), as well as, the unified auxiliary
tools located in