sidescanpy.io.xtf.reader#

Low-level XTF (eXtended Triton Format) reader.

Reads the file header, frames packets, and parses sonar packets into per-ping header records and per-channel sample arrays. The converter to the standard survey model is sidescanpy.io.xtf.converter; read_xtf is the per-file entry point that feeds it.

File layout#

A file starts with a 1024-byte header holding up to six ChanInfo channel descriptions; with more than six channels the remaining ChanInfo entries follow contiguously in further 1024-byte blocks. Packets follow. Each starts with the magic 0xFACE, a header type and NumBytesThisRecord, the packet’s total size including any padding. That byte count is authoritative: the reader steps from packet to packet with it, never with sizes derived from the packet contents. A packet with a bad magic triggers a forward scan for the next 0xFACE (counted, and logged at WARNING); a truncated final packet is logged at WARNING and ends the read.

A sonar packet (header type 0) is a 256-byte ping header followed, for each channel, by a 64-byte channel header and NumSamples samples. The sample type comes from the file header’s ChanInfo entry for that channel (SampleFormat, else BytesPerSample; a SampleFormat the specification does not define, such as 255, counts as unset).

Channel blocks are walked using the packet byte count#

NumChansToFollow in the ping header is treated as an upper bound. Channel blocks are parsed while at least a full 64-byte channel header remains inside the packet; a channel whose samples would run past the packet end is clipped to the samples that fit. Each mismatch kind is logged at WARNING once per file with a count.

Evidence for this rule (EdgeTech 4200 files recorded by Isis, four channels declared in the file header: port/starboard at 120 kHz as channels 0/1 and at 410 kHz as channels 2/3, BytesPerSample 2):

In two such files the first sonar packets (2976 of 25539 in one file, 1680 of 20475 in the other, a single contiguous run at the start) are 4480 bytes long while their ping headers say NumChansToFollow = 4 and the first two channel headers say NumSamples = 1024. Normal packets are 8704 = 256 + 4 * (64 + 1024 * 2) bytes.
4480 = 256 + 2 * (64 + 1024 * 2) exactly: the packet holds two complete channel blocks (channel numbers 0 and 1, the low-frequency pair) and nothing else. The “third channel header” a reader trusting NumChansToFollow would parse starts with 0xFACE: it is the next packet.
The samples are 16-bit, 1024 per channel as declared. Read as uint16, adjacent samples correlate (lag-1 r = 0.96 and 0.79 over the two runs, 0.86 and 0.80 in the same files’ normal pings), values stay within the normal pings’ 0..32767 range, and medians are comparable (1148 vs 1505, 285 vs 731). Read as 8-bit, adjacent samples do not correlate (r = -0.01 and -0.46). The first pings of each run are faint (high byte zero in 88-99.99 % of the first 200 pings), rising to the normal level. Ping numbers are continuous across the switch to four-channel packets, navigation is continuous, and the low-only pings repeat faster (0.08 s vs 0.14 s at 100 m range, 0.21 s vs 0.42 s at 300 m).

So those pings carry only the low-frequency pair; the high-frequency channels are absent, not empty. pyxtf 1.5.0 trusts NumChansToFollow and fails on such packets.

Sample arrays#

Samples keep their recorded dtype (uint8/uint16/uint32/float32), so values are bit-identical to the file. A channel’s array is (n_ping, max_samples) over all sonar pings of the file; pings where the channel is absent or shorter are zero-padded on the right, and the true length is in num_samples (0 where absent, with present False). The converter decides whether padding becomes NaN.

2026 rshomberg@uri.edu

Contents

`XtfFileHeader`	Parsed XTF file header.
`header_size_for_channels`	Return the file-header size in bytes for `n_channels` channels.
`read_xtf_header`	Read the file header of an XTF file.
`XtfPacket`	One framed packet.
`PacketStats`	Counts gathered while framing packets.
`iter_packets`	Stream the packets of an XTF file.
`count_packets`	Frame every packet of a file without parsing them.
`XtfChannel`	Samples and headers of one sonar channel over all pings.
`XtfFrequencyGroup`	A port/starboard channel pair sharing one frequency.
`ChannelBlockStats`	Mismatches between sonar packets' declared and actual channel blocks.
`XtfSegmentInfo`	Where a run of pings sits in its source file.
`XtfPingData`	Sonar pings of one XTF file, as recorded.
`ping_times`	Compute ping times from ping header records.
`read_xtf_pings`	Read every sonar ping of an XTF file.
`read_xtf`	Read one XTF file for conversion.

class sidescanpy.io.xtf.reader.XtfFileHeader(record, chan_info, size)#

Bases: object

Parsed XTF file header.

Attributes:

recordnumpy.void: The first 1024-byte header block as a FILE_HEADER_DTYPE record (its ChanInfo field holds the first six slots).
chan_infonumpy.ndarray: CHAN_INFO_DTYPE array with one entry per declared channel (the sum of the header’s channel counts), including entries from extra header blocks.
sizeint: Header size in bytes, i.e. the offset of the first packet.

property n_channels#: Number of channels declared in the header.

sidescanpy.io.xtf.reader.header_size_for_channels(n_channels)#

Return the file-header size in bytes for n_channels channels.

The first 1024-byte block holds six ChanInfo entries; further entries follow contiguously from byte 1024, padded to whole 1024-byte blocks.

Parameters:

n_channelsint: Declared channel count.

Returns:

int

sidescanpy.io.xtf.reader.read_xtf_header(path)#

Read the file header of an XTF file.

Parameters:

pathstr or Path: XTF file.

Returns:

XtfFileHeader

Raises:

FileNotFoundError: If the file does not exist.
ValueError: If the file is too short or its first byte is not the XTF FileFormat value 0x7B.

class sidescanpy.io.xtf.reader.XtfPacket(header_type, offset, data)#

Bases: object

One framed packet.

Attributes:

header_typeint: HeaderType from the packet start (see structs.HEADER_TYPE_NAMES).
offsetint: Byte offset of the packet in the file.
databytes: The whole packet, starting with its 14 packet-start bytes, of length NumBytesThisRecord.

property start#: The packet start fields as a PACKET_START_DTYPE record.

class sidescanpy.io.xtf.reader.PacketStats(counts=<factory>, skipped=<factory>, unknown=<factory>, resyncs=0, resync_bytes=0, truncated=False)#

Bases: object

Counts gathered while framing packets.

Attributes:

countscollections.Counter: Packets framed, by header type (yielded or skipped).
skippedcollections.Counter: Packets not yielded because of the header_types filter.
unknowncollections.Counter: Packets whose header type the specification does not define.
resyncsint: Bad-magic events that needed a forward scan.
resync_bytesint: Bytes skipped by those scans.
truncatedbool: True if the file ended inside a packet.

sidescanpy.io.xtf.reader.iter_packets(path, header_types=None, stats=None)#

Stream the packets of an XTF file.

Parameters:

pathstr or Path: XTF file.
header_typesiterable of int, optional: Header types to yield. Others are skipped by their byte count without reading their body. Default: yield every packet.
statsPacketStats, optional: Filled in place with counts as packets are framed (useful because a generator has no return value to the caller).

Yields:

XtfPacket: Packets in file order.

Notes

Packets whose header type is not defined by the specification are counted in stats.unknown and logged at DEBUG; they are yielded like any other when no filter is given. A bad magic number or a byte count smaller than the packet start triggers a forward scan for the next 0xFACE (logged at WARNING). A packet extending past the end of the file is logged at WARNING and ends iteration.

sidescanpy.io.xtf.reader.count_packets(path)#

Frame every packet of a file without parsing them.

Parameters:

pathstr or Path: XTF file.

Returns:

PacketStats: Counts by header type, plus resync and truncation information.

class sidescanpy.io.xtf.reader.XtfChannel(channel_number, chan_info, name, side, frequency, samples, num_samples, present, headers)#

Bases: object

Samples and headers of one sonar channel over all pings.

Attributes:

channel_numberint: ChannelNumber from the channel headers (index into the file header’s ChanInfo).
chan_infonumpy.void: The ChanInfo record for this channel.
namestr: ChanInfo.ChannelName.
sidestr: "port", "stbd" or "other" (from TypeOfChannel).
frequencyfloat: ChanInfo.Frequency, or the most common nonzero per-ping channel-header Frequency when that is 0, else NaN. The unit is whatever the recorder wrote (kHz in the files seen so far).
samplesnumpy.ndarray: (n_ping, max_samples) in the recorded sample dtype, zero-padded on the right.
num_samplesnumpy.ndarray: (n_ping,) uint32, samples stored for each ping (0 where absent).
presentnumpy.ndarray: (n_ping,) bool, whether the ping held this channel.
headersnumpy.ndarray: (n_ping,) PING_CHAN_HEADER_DTYPE records; all zero where the channel is absent.

class sidescanpy.io.xtf.reader.XtfFrequencyGroup(frequency, port=None, stbd=None)#

Bases: object

A port/starboard channel pair sharing one frequency.

Attributes:

frequencyfloat: Frequency as recorded (see XtfChannel.frequency); NaN when unknown, in which case channels were paired in channel order.
port, stbdXtfChannel or None: The two sides; one may be missing.

class sidescanpy.io.xtf.reader.ChannelBlockStats(fewer_channels=0, declared=<factory>, found=<factory>, clipped=0, no_chan_info=0, legacy_count=0)#

Bases: object

Mismatches between sonar packets’ declared and actual channel blocks.

Attributes:

fewer_channelsint: Sonar packets holding fewer complete channel blocks than their NumChansToFollow.
declared, foundcollections.Counter: For those packets: declared channel counts, and the number of blocks actually found.
clippedint: Channel blocks declaring more samples than fit in their packet (clipped to what fits).
no_chan_infoint: Channel blocks with no usable ChanInfo entry (rest of the packet skipped).
legacy_countint: Channel blocks whose sample count came from ChanInfo because NumSamples was 0 (pre-X41 files).

class sidescanpy.io.xtf.reader.XtfSegmentInfo(index, count, ping_start, ping_stop, n_pings_file, n_runs_dropped, n_pings_dropped, rtol)#

Bases: object

Where a run of pings sits in its source file.

Set by sidescanpy.io.xtf.split_by_sample_rate().

Attributes:

indexint: 1-based position among the kept runs.
countint: Number of kept runs of the file.
ping_start, ping_stopint: The run’s pings are [ping_start, ping_stop) in the file’s sonar-ping order.
n_pings_fileint: Sonar pings in the whole file.
n_runs_droppedint: Runs of the file shorter than min_pings, dropped.
n_pings_droppedint: Pings in those dropped runs.
rtolfloat: Relative tolerance used for the cuts.

class sidescanpy.io.xtf.reader.XtfPingData(path, file_header, ping_headers, packet_offsets, channels, frequency_groups, stats, block_stats=<factory>, segment=None)#

Bases: object

Sonar pings of one XTF file, as recorded.

Attributes:

pathPath: Source file.
file_headerXtfFileHeader: Parsed file header.
ping_headersnumpy.ndarray: (n_ping,) PING_HEADER_DTYPE records, in file order.
packet_offsetsnumpy.ndarray: (n_ping,) int64 file offset of each sonar packet.
channelsdict[int, XtfChannel]: Channels keyed by ChannelNumber.
frequency_groupslist[XtfFrequencyGroup]: Port/starboard channels grouped by frequency, ascending (unknown frequencies last).
statsPacketStats: Packet counts from the read.
block_statsChannelBlockStats: Channel-block mismatches found while parsing sonar packets.
segmentXtfSegmentInfo or None: Set when these pings are one run of a file split at sample-rate changes (sidescanpy.io.xtf.split_by_sample_rate()); None for a whole file.

property n_pings#: Number of sonar pings.

property times#: Ping times as datetime64[ns] (see ping_times()).

sidescanpy.io.xtf.reader.ping_times(ping_headers)#

Compute ping times from ping header records.

Uses Year, Month, Day, Hour, Minute, Second and HSeconds (hundredths). The time base is whatever the recorder used (normally UTC). Records with an invalid date give NaT.

Parameters:

ping_headersnumpy.ndarray: PING_HEADER_DTYPE records.

Returns:

numpy.ndarray: datetime64[ns], one per record.

sidescanpy.io.xtf.reader.read_xtf_pings(path)#

Read every sonar ping of an XTF file.

Two passes over the file: the first frames sonar packets and records the ping and channel headers and where each channel’s samples are; the second reads the samples straight into preallocated arrays. Other packet types are counted and skipped.

Parameters:

pathstr or Path: XTF file.

Returns:

XtfPingData: Ping headers, per-channel sample arrays (recorded dtype, zero-padded) and port/starboard frequency groups.

Raises:

ValueError: If the file header is invalid or a channel uses an unsupported sample format.

sidescanpy.io.xtf.reader.read_xtf(path)#

Read one XTF file for conversion.

The XTF counterpart of read_jsf / read_sdf: returns what sidescanpy.io.xtf.convert() takes. Equivalent to read_xtf_pings().

Parameters:

pathstr or Path: XTF file.

Returns:

XtfPingData