Archive::Tyd

A while back when I was a little more serious than usual about actually programming a game, I put some time into programming my own archiving format. I called it Tyd (sounds like "tied"). It was pretty simple: basically the contents of multiple files are all thrown together into a single file, and then the whole archive is encrypted using CipherSaber, a symmetric key cipher algorithm.

The idea was that there could be a common archive file format (Tyd) that could be used by multiple applications or games, and each application would have its own password for its Tyd archive, that only the application and the developer knows. This would make it at least a little bit difficult for the app's end users to open up the archive and poke around at its contents. Compare this to Blizzard's MPQ archive format used by all their games, where users can easily open them up and get at their contents. With Tyd, they'd need to reverse engineer each application that uses a Tyd archive to open that app's archives.

You can see Archive::Tyd's CPAN page for more details.

This was limited though because, since the whole entire file was encrypted together, the application would need to load the whole archive into memory to be able to use it. So, while it was fine for small archives containing small files, a larger archive would consume too much memory. So forget about storing a lot of MP3s and MPEGs in a Tyd archive unless you're operating a supercomputer.

Also, there was no way to verify that a password to an archive was entered correctly, short of trying to decode it and see if you only get gibberish out of it.

So I started piecing together ideas for a successor to Tyd, which will still be called Tyd (version 2.0). The basic requirements are:

• It has to support live reading (the calling app should be able to read a file directly out of the archive without extracting it to disk first). Archive::Zip doesn't handle this.
• It has to have some protection of some sort, to ensure that the app that owns it has a much easier time reading it than the app's end users do.
• It has to be able to handle very large archives. Archive::Zip, again, can't handle this.

So I typed up a spec sheet and began programming. The new Tyd archive will come with a complete set of headers (giving the archive a name and comment, and indicating whether the archive is encrypted, compressed, or raw, including some verification checks for decrypting, and a few other things like this). It will also contain a file table in the headers, so that a list of files can be pulled from the archive quickly, along with their attributes. And then the actual contents of the files follow all of that.

To facilitate "streaming", when the archive is encrypted or compressed, each file is only affected one block at a time. By default the block size is 512 bytes, so when a file is added to an encrypted archive, 512 bytes are encrypted at a time and separately. When reading the file back from the archive, one block at a time is read, decrypted, and returned to the caller (the block size after encryption is surely greater than 512 bytes; when compressed, less than 512 bytes).

For the actual encryption and compression algorithms I'll be using existing CPAN modules to implement known algorithms.

The new Archive::Tyd algorithm is intended to have basically these features:

• Archive Headers:
  • A name and comment, author's name, generator's name
  • A flag to define whether the archive is compressed, encrypted, or neither (raw).
  • Block size for reading/writing the archive.
• Encrypted Archives:
  • Supported Encryption Algorithms:
    • CipherSaber (symmetric key; used with old Archive::Tyd)
    • RSA encryption (public-key)
    • AES, DES, 3DES, RC2, RC4
    • Others?
  • If it uses a symmetric-key encryption such as CipherSaber, the MD5 or SHA1 sum of the correct password will be included to verify that the decryption will work. One catch -- the MD5 or SHA1 sum is, itself, encrypted using the same algorithm. So with CipherSaber, you give it the password, it attempts to decode the MD5/SHA1 sum with it, and then compares the sum with the original password you gave it. This way it can verify you have the right password while keeping the password relatively safe from cracking.
• Compressed Archives:
  • Supported Compression Algorithms:
    • Deflate
    • LZF
    • GZip

With public-key encryption used on the archive, this would basically mean that only the application developer would be able to create or modify an archive that their application uses. Instead of a single password being used to both create and read archives (as with CipherSaber), a secret key is needed to create it, and a public key is used to decode it.

This way the application can be built to know the public key so it can read the archive, and any user who reverse engineers the application can only get the public key -- so they can get read-only access to the archive, but have a much harder time modifying it or changing its contents without the secret key. IIRC this would be similar to Blizzard's MPQ, in that the DLL that reads MPQ files for their games doesn't include the functions needed to write/modify MPQ files, giving the end users read-only access to the file's contents.

Anyway, no ETA yet, this is a big project. (Well, not really, the heavy lifting of encryption/compression is done by third party modules, all I need to do is program the wrapper code).