aisdb.database.decoder module

class aisdb.database.decoder.FileChecksums(*, dbconn)[source]

Bases: object

Initializes a FileChecksums object with a specified database connection.

Parameters:

dbconn – A required parameter of type PostgresDBConn or SQLiteDBConn that represents the database connection.

Returns:

None

checksum_exists(checksum)[source]

Check if a given checksum exists in the database.

Parameters:

checksum – The checksum value to check.

Returns:

True if the checksum exists in the database, False otherwise.

checksums_table()[source]

Creates a checksums table in the database if it doesn’t exist.

This method creates a table named ‘hashmap’ in the database, if it doesn’t already exist. The table contains two columns: ‘hash’ of type TEXT and ‘bytes’ of type BLOB for SQLiteDBConn *, or BYTEA for PostgresDBConn.

param self:

an instance of the current object.

return:

None

get_md5(path, f)[source]

Calculates the MD5 hash digest of a file.

Parameters:

  • path – The path of the file.

  • f – The file object to calculate the digest for.

Returns:

The MD5 hash digest of the file content.

insert_checksum(checksum)[source]

Inserts a checksum and corresponding pickled None value into the hashmap table.

Parameters:

checksum – The checksum to be inserted into the hashmap table.

Returns:

None

aisdb.database.decoder.decode_msgs(filepaths, dbconn, source, vacuum=False, skip_checksum=True, workers=4, type_preference='all', raw_insertion=True, verbose=True, timescaledb=False)[source]

Decode messages from filepaths and insert them into a database.

Parameters:

  • filepaths – list of file paths to decode

  • dbconn – database connection to use for insertion

  • source – source identifier for the decoded messages

  • vacuum – whether to vacuum the database after insertion (default is False)

  • skip_checksum – whether to skip checksum validation (default is True)

  • workers – number of parallel workers to use (default is 4)

  • type_preference – preferred file type to be used (default is “all”)

  • raw_insertion – whether to insert messages without indexing them (default is True)

  • verbose – whether to print verbose output (default is True)

  • timescaledb – whether to insert data to a database with timescale extension (default is False)

Returns:

None

aisdb.database.decoder.decoder(dbpath, psql_conn_string, files, source, verbose)

Parse NMEA-formatted strings, and create databases from raw AIS transmissions

Parameters:

  • dbpath (str) – Output SQLite database path. Set this to an empty string to only use Postgres

  • psql_conn_string (str) – Postgres database connection string. Set this to an empty string to only use SQLite

  • files (array of str) – array of .nm4 raw data filepath strings

  • source (str) – data source text. Will be used as a primary key index in database

  • verbose (bool) – enables logging

Returns:

None

aisdb.database.decoder.fast_unzip(zip_filenames, dirname)[source]

Unzips multiple zip files to a specified directory.

Parameters:

  • zip_filenames – List of zip file names to be extracted.

  • dirname – Directory path where the files should be extracted to.