Author: Arvid Norberg, arvid@libtorrent.org
Version: 1.1.2

home

Custom Storage

libtorrent provides a customization point for storage of data. By default, (default_storage) downloaded files are saved to disk according with the general conventions of bittorrent clients, mimicing the original file layout when the torrent was created. The libtorrent user may define a custom storage to store piece data in a different way.

A custom storage implementation must derive from and implement the storage_interface. You must also provide a function that constructs the custom storage object and provide this function to the add_torrent() call via add_torrent_params. Either passed in to the constructor or by setting the add_torrent_params::storage field.

This is an example storage implementation that stores all pieces in a std::map, i.e. in RAM. It's not necessarily very useful in practice, but illustrates the basics of implementing a custom storage.

struct temp_storage : storage_interface
{
        temp_storage(file_storage const& fs) : m_files(fs) {}
        virtual bool initialize(storage_error& se) { return false; }
        virtual bool has_any_file() { return false; }
        virtual int read(char* buf, int piece, int offset, int size)
        {
                std::map<int, std::vector<char> >::const_iterator i = m_file_data.find(piece);
                if (i == m_file_data.end()) return 0;
                int available = i->second.size() - offset;
                if (available <= 0) return 0;
                if (available > size) available = size;
                memcpy(buf, &i->second[offset], available);
                return available;
        }
        virtual int write(const char* buf, int piece, int offset, int size)
        {
                std::vector<char>& data = m_file_data[piece];
                if (data.size() < offset + size) data.resize(offset + size);
                std::memcpy(&data[offset], buf, size);
                return size;
        }
        virtual bool rename_file(int file, std::string const& new_name)
        { assert(false); return false; }
        virtual bool move_storage(std::string const& save_path) { return false; }
        virtual bool verify_resume_data(bdecode_node const& rd
                , std::vector<std::string> const* links
                , storage_error& error) { return false; }
        virtual bool write_resume_data(entry& rd) const { return false; }
        virtual boost::int64_t physical_offset(int piece, int offset)
        { return piece * m_files.piece_length() + offset; };
        virtual sha1_hash hash_for_slot(int piece, partial_hash& ph, int piece_size)
        {
                int left = piece_size - ph.offset;
                assert(left >= 0);
                if (left > 0)
                {
                        std::vector<char>& data = m_file_data[piece];
                        // if there are padding files, those blocks will be considered
                        // completed even though they haven't been written to the storage.
                        // in this case, just extend the piece buffer to its full size
                        // and fill it with zeroes.
                        if (data.size() < piece_size) data.resize(piece_size, 0);
                        ph.h.update(&data[ph.offset], left);
                }
                return ph.h.final();
        }
        virtual bool release_files() { return false; }
        virtual bool delete_files() { return false; }

        std::map<int, std::vector<char> > m_file_data;
        file_storage m_files;
};

storage_interface* temp_storage_constructor(storage_params const& params)
{
        return new temp_storage(*params.files);
}

disk_buffer_holder

Declared in "libtorrent/disk_buffer_holder.hpp"

The disk buffer holder acts like a scoped_ptr that frees a disk buffer when it's destructed, unless it's released. release returns the disk buffer and transfers ownership and responsibility to free it to the caller.

A disk buffer is freed by passing it to session_impl::free_disk_buffer().

get() returns the pointer without transferring responsibility. If this buffer has been released, buffer() will return 0.

struct disk_buffer_holder
{
   disk_buffer_holder (buffer_allocator_interface& alloc, disk_io_job const& j);
   ~disk_buffer_holder ();
   char* release ();
   char* get () const;
   void reset (char* buf = 0);
   void reset (disk_io_job const& j);
   void swap (disk_buffer_holder& h);
   block_cache_reference ref () const;
};

disk_buffer_holder()

disk_buffer_holder (buffer_allocator_interface& alloc, disk_io_job const& j);

construct a buffer holder that will free the held buffer using a disk buffer pool directly (there's only one disk_buffer_pool per session)

~disk_buffer_holder()

~disk_buffer_holder ();

frees any unreleased disk buffer held by this object

release()

char* release ();

return the held disk buffer and clear it from the holder. The responsibility to free it is passed on to the caller

get()

char* get () const;

return a pointer to the held buffer

reset()

void reset (char* buf = 0);
void reset (disk_io_job const& j);

set the holder object to hold the specified buffer (or NULL by default). If it's already holding a disk buffer, it will first be freed.

swap()

void swap (disk_buffer_holder& h);

swap pointers of two disk buffer holders.

file_pool

Declared in "libtorrent/file_pool.hpp"

this is an internal cache of open file handles. It's primarily used by storage_interface implementations. It provides semi weak guarantees of not opening more file handles than specified. Given multiple threads, each with the ability to lock a file handle (via smart pointer), there may be windows where more file handles are open.

struct file_pool : boost::noncopyable
{
   ~file_pool ();
   file_pool (int size = 40);
   file_handle open_file (void* st, std::string const& p
      , int file_index, file_storage const& fs, int m, error_code& ec);
   void release (void* st = NULL);
   void release (void* st, int file_index);
   void resize (int size);
   int size_limit () const;
};

~file_pool() file_pool()

~file_pool ();
file_pool (int size = 40);

size specifies the number of allowed files handles to hold open at any given time.

open_file()

file_handle open_file (void* st, std::string const& p
      , int file_index, file_storage const& fs, int m, error_code& ec);

return an open file handle to file at file_index in the file_storage fs opened at save path p. m is the file open mode (see file::open_mode_t).

release()

void release (void* st = NULL);
void release (void* st, int file_index);

release all files belonging to the specified storage_interface (st) the overload that takes file_index releases only the file with that index in storage st.

resize()

void resize (int size);

update the allowed number of open file handles to size.

size_limit()

int size_limit () const;

returns the current limit of number of allowed open file handles held by the file_pool.

storage_interface

Declared in "libtorrent/storage.hpp"

The storage interface is a pure virtual class that can be implemented to customize how and where data for a torrent is stored. The default storage implementation uses regular files in the filesystem, mapping the files in the torrent in the way one would assume a torrent is saved to disk. Implementing your own storage interface makes it possible to store all data in RAM, or in some optimized order on disk (the order the pieces are received for instance), or saving multifile torrents in a single file in order to be able to take advantage of optimized disk-I/O.

It is also possible to write a thin class that uses the default storage but modifies some particular behavior, for instance encrypting the data before it's written to disk, and decrypting it when it's read again.

The storage interface is based on pieces. Avery read and write operation happens in the piece-space. Each piece fits 'piece_size' number of bytes. All access is done by writing and reading whole or partial pieces.

libtorrent comes with two built-in storage implementations; default_storage and disabled_storage. Their constructor functions are called default_storage_constructor() and disabled_storage_constructor respectively. The disabled storage does just what it sounds like. It throws away data that's written, and it reads garbage. It's useful mostly for benchmarking and profiling purpose.

struct storage_interface
{
   virtual void initialize (storage_error& ec) = 0;
   virtual int writev (file::iovec_t const* bufs, int num_bufs
      , int piece, int offset, int flags, storage_error& ec) = 0;
   virtual int readv (file::iovec_t const* bufs, int num_bufs
      , int piece, int offset, int flags, storage_error& ec) = 0;
   virtual bool has_any_file (storage_error& ec) = 0;
   virtual void set_file_priority (std::vector<boost::uint8_t> const& prio
      , storage_error& ec) = 0;
   virtual int move_storage (std::string const& save_path, int flags
      , storage_error& ec) = 0;
   virtual bool verify_resume_data (bdecode_node const& rd
      , std::vector<std::string> const* links
      , storage_error& ec) = 0;
   virtual void write_resume_data (entry& rd, storage_error& ec) const = 0;
   virtual void release_files (storage_error& ec) = 0;
   virtual void rename_file (int index, std::string const& new_filename
      , storage_error& ec) = 0;
   virtual void delete_files (int options, storage_error& ec) = 0;
   virtual bool tick ();
   aux::session_settings const& settings () const;

   aux::session_settings* m_settings;
};

initialize()

virtual void initialize (storage_error& ec) = 0;

This function is called when the storage is to be initialized. The default storage will create directories and empty files at this point. If allocate_files is true, it will also ftruncate all files to their target size.

If an error occurs, storage_error should be set to reflect it.

writev() readv()

virtual int writev (file::iovec_t const* bufs, int num_bufs
      , int piece, int offset, int flags, storage_error& ec) = 0;
virtual int readv (file::iovec_t const* bufs, int num_bufs
      , int piece, int offset, int flags, storage_error& ec) = 0;

These functions should read and write the data in or to the given piece at the given offset. It should read or write num_bufs buffers sequentially, where the size of each buffer is specified in the buffer array bufs. The file::iovec_t type has the following members:

struct iovec_t { void* iov_base; size_t iov_len; };

These functions may be called simultaneously from multiple threads. Make sure they are thread safe. The file in libtorrent is thread safe when it can fall back to pread, preadv or the windows equivalents. On targets where read operations cannot be thread safe (i.e one has to seek first and then read), only one disk thread is used.

Every buffer in bufs can be assumed to be page aligned and be of a page aligned size, except for the last buffer of the torrent. The allocated buffer can be assumed to fit a fully page aligned number of bytes though. This is useful when reading and writing the last piece of a file in unbuffered mode.

The offset is aligned to 16 kiB boundaries most of the time, but there are rare exceptions when it's not. Specifically if the read cache is disabled/or full and a peer requests unaligned data. Most clients request aligned data.

The number of bytes read or written should be returned, or -1 on error. If there's an error, the storage_error must be filled out to represent the error that occurred.

has_any_file()

virtual bool has_any_file (storage_error& ec) = 0;

This function is called when first checking (or re-checking) the storage for a torrent. It should return true if any of the files that is used in this storage exists on disk. If so, the storage will be checked for existing pieces before starting the download.

If an error occurs, storage_error should be set to reflect it.

set_file_priority()

virtual void set_file_priority (std::vector<boost::uint8_t> const& prio
      , storage_error& ec) = 0;

change the priorities of files. This is a fenced job and is guaranteed to be the only running function on this storage when called

move_storage()

virtual int move_storage (std::string const& save_path, int flags
      , storage_error& ec) = 0;

This function should move all the files belonging to the storage to the new save_path. The default storage moves the single file or the directory of the torrent.

Before moving the files, any open file handles may have to be closed, like release_files().

If an error occurs, storage_error should be set to reflect it.

returns one of: | no_error = 0 | fatal_disk_error = -1 | need_full_check = -2 | file_exist = -4

verify_resume_data()

virtual bool verify_resume_data (bdecode_node const& rd
      , std::vector<std::string> const* links
      , storage_error& ec) = 0;

This function should verify the resume data rd with the files on disk. If the resume data seems to be up-to-date, return true. If not, set error to a description of what mismatched and return false.

The default storage may compare file sizes and time stamps of the files.

If an error occurs, storage_error should be set to reflect it.

This function should verify the resume data rd with the files on disk. If the resume data seems to be up-to-date, return true. If not, set error to a description of what mismatched and return false.

If the links pointer is non-null, it has the same number of elements as there are files. Each element is either empty or contains the absolute path to a file identical to the corresponding file in this torrent. The storage must create hard links (or copy) those files. If any file does not exist or is inaccessible, the disk job must fail.

write_resume_data()

virtual void write_resume_data (entry& rd, storage_error& ec) const = 0;

This function should fill in resume data, the current state of the storage, in rd. The default storage adds file timestamps and sizes.

Returning true indicates an error occurred.

If an error occurs, storage_error should be set to reflect it.

release_files()

virtual void release_files (storage_error& ec) = 0;

This function should release all the file handles that it keeps open to files belonging to this storage. The default implementation just calls file_pool::release_files().

If an error occurs, storage_error should be set to reflect it.

rename_file()

virtual void rename_file (int index, std::string const& new_filename
      , storage_error& ec) = 0;

Rename file with index file to the thame new_name.

If an error occurs, storage_error should be set to reflect it.

delete_files()

virtual void delete_files (int options, storage_error& ec) = 0;

This function should delete some or all of the storage for this torrent. The options parameter specifies whether to delete all files or just the partfile. options are set to the same value as the options passed to session::remove_torrent().

If an error occurs, storage_error should be set to reflect it.

The disk_buffer_pool is used to allocate and free disk buffers. It has the following members:

struct disk_buffer_pool : boost::noncopyable
{
        char* allocate_buffer(char const* category);
        void free_buffer(char* buf);

        char* allocate_buffers(int blocks, char const* category);
        void free_buffers(char* buf, int blocks);

        int block_size() const { return m_block_size; }

        void release_memory();
};

tick()

virtual bool tick ();

called periodically (useful for deferred flushing). When returning false, it means no more ticks are necessary. Any disk job submitted will re-enable ticking. The default will always turn ticking back off again.

settings()

aux::session_settings const& settings () const;

access global session_settings

m_settings
initialized in disk_io_thread::perform_async_job

default_storage

Declared in "libtorrent/storage.hpp"

The default implementation of storage_interface. Behaves as a normal bittorrent client. It is possible to derive from this class in order to override some of its behavior, when implementing a custom storage.

class default_storage : public storage_interface, boost::noncopyable
{
   default_storage (storage_params const& params);
   virtual int move_storage (std::string const& save_path, int flags
      , storage_error& ec) override;
   virtual bool verify_resume_data (bdecode_node const& rd
      , std::vector<std::string> const* links
      , storage_error& error) override;
   virtual void initialize (storage_error& ec) override;
   virtual bool tick () override;
   virtual void delete_files (int options, storage_error& ec) override;
   virtual void rename_file (int index, std::string const& new_filename
      , storage_error& ec) override;
   virtual void set_file_priority (std::vector<boost::uint8_t> const& prio
      , storage_error& ec) override;
   virtual void write_resume_data (entry& rd, storage_error& ec) const override;
   virtual void release_files (storage_error& ec) override;
   virtual bool has_any_file (storage_error& ec) override;
   int readv (file::iovec_t const* bufs, int num_bufs
      , int piece, int offset, int flags, storage_error& ec) override;
   int writev (file::iovec_t const* bufs, int num_bufs
      , int piece, int offset, int flags, storage_error& ec) override;
   file_storage const& files () const;
   static void disk_write_access_log (bool enable);
   static bool disk_write_access_log ();
};

default_storage()

default_storage (storage_params const& params);

constructs the default_storage based on the give file_storage (fs). mapped is an optional argument (it may be NULL). If non-NULL it represents the file mapping that have been made to the torrent before adding it. That's where files are supposed to be saved and looked for on disk. save_path is the root save folder for this torrent. file_pool is the cache of file handles that the storage will use. All files it opens will ask the file_pool to open them. file_prio is a vector indicating the priority of files on startup. It may be an empty vector. Any file whose index is not represented by the vector (because the vector is too short) are assumed to have priority 1. this is used to treat files with priority 0 slightly differently.

files()

file_storage const& files () const;

if the files in this storage are mapped, returns the mapped file_storage, otherwise returns the original file_storage object.

enum move_flags_t

Declared in "libtorrent/storage.hpp"

name value description
always_replace_files 0 replace any files in the destination when copying or moving the storage
fail_if_exist 1 if any files that we want to copy exist in the destination exist, fail the whole operation and don't perform any copy or move. There is an inherent race condition in this mode. The files are checked for existence before the operation starts. In between the check and performing the copy, the destination files may be created, in which case they are replaced.
dont_replace 2 if any file exist in the target, take those files instead of the ones we may have in the source.