DiskBoss Logo
Flexense Data Management Software

Real-Time File Synchronization

DiskBoss Server and DiskBoss Enterprise provide real-time file synchronization capabilities allowing one to synchronize files between local disks, directories, network shares, NAS storage devices and enterprise storage systems. The DiskBoss real-time file synchronization engine is capable of monitoring changes in a disk or directory and synchronizing files on-the-fly as files are changed on the disk without scanning directories, which makes the solution very scalable and capable of almost instantly synchronizing huge file systems containing many millions of files.

DiskBoss Add Real-Time File Synchronization Command

In order to add a real-time file synchronization command, open the DiskBoss client GUI application, press the 'Add' button located on the main toolbar, select the 'Real-Time File Sync Command' menu item and specify a unique command name. On the real-time file sync command dialog, specify a source directory to monitor and a destination directory to synchronize.

DiskBoss Real-Time File Synchronization Command Dialog

Before starting the real-time file sync command, use the regular file sync operations available in the product to perform an initial sync operation. Once the initial sync is completed, select the real-time sync command in the commands view, press the right mouse button and select the 'Start Command' menu item. Now, the real-time sync command will monitor changes in the specified source directory and synchronize all changed files to the specified destination directory. If the real-time sync command should be automatically started after each system restart, open the real-time sync command dialog and select the 'Enable Auto Start' option.

Advanced File Synchronization Options

DiskBoss provides a large number of advanced real-time files synchronization options allowing one to tune the performance of file synchronization operations, control the file copy recovery and fault tolerance options, select which file meta data to copy, etc. In order to access the performance tuning options, open the real-time file sync command dialog, press the 'Options' button and select the 'General' tab.

Real-Time File Synchronization Performance Tuning Options
  • Performance Mode - the performance mode allows one to select one of the following modes: Full Speed, Medium Speed and Low Speed. The performance mode provides the ability to intentionally slow down the performance of the file sync operation in order to minimize the potential impact on the running production applications.
  • Dir Scan Mode - the directory scanning mode allows one to select one of the following directory scanning modes: Sequential or Parallel. In the Sequential directory scanning mode, DiskBoss scans directories one-after-one using a single processing thread. In the Parallel directory scanning mode, DiskBoss scans directories in parallel using a number of processing threads according to the 'Dir Scan Threads' option, which significantly improves the performance of the file synchronization operation when synchronizing large numbers of files via a slow, high-latency network.
  • Dir Scan Threads - this option specified the number of parallel directory scanning threads to use in the Parallel directory scanning mode.
  • File Sync Threads - this option specifies the number of parallel file synchronization threads to use during the file synchronization process. Multi-Stream file synchronization significantly improves the file synchronization performance when synchronizing a large number of files between multiple disks, NAS storage devices or enterprise storage systems.

The 'Advanced' file synchronization options tab provides a number of different file copy modes optimized for different hardware configurations and user-specific needs:

Real-Time File Synchronization Performance
  • Active File Copy Mode - in this file copy mode DiskBoss automatically switches between different file copy modes depending on the size and the type of each file. Small files are transferred using a regular, buffered file copy mode, while large files are transferred using a non-buffered file copy mode and optionally, when the hardware platform supports it, allows to enable the write-through I/O mode.
  • Regular, Buffered File Copy Mode - in this file copy mode DiskBoss performs regular, buffered file copy operations according to the specified file I/O block size and memory alignment.
  • Fault-Tolerant File Copy Mode - in this file copy mode DiskBoss creates a temporary file for each file that should be copied and only after a successful file copy operation replaces the original file with the new one. If a network failure will occur during a file copy operation, original files will remain in place without any changes.
  • Operating System Native File Copy Mode - this file copy mode is optimized for modern operating systems such as Windows Server 2012 and newer. This file copy mode supports SMB direct, direct RDMA file transfers, multi-channel and write-through file I/O operations. In order to take full advantage of all performance optimizations when synchronizing files via the network, Windows Server 2012 or newer should be installed on both sides.

In addition, the 'Advanced' file synchronization options tab provides a number of performance, fault tolerance and recovery options:

  • Verification Mode - the file copy verification mode specifies how to verify files copied to the destination directory. By default, verification of copied files is disabled and in order to enable verification of copied files the user needs to select an appropriate verification mode and then enable verification on the 'Options' tab.
  • Memory Buffer - this option specifies the size of the file copy memory buffer used for each file copy thread.
  • Memory Alignment - this option specifies the alignment of the file copy memory buffer, which is critical for fast, non-buffered file copy operations. By default, the memory alignment is set to 4096 bytes, which is appropriate for modern file systems and storage devices.
  • Copy Retry Count - this option specifies how many times to retry a failed file copy operation. Sometimes, a file may be temporary locked by an application and this option allows one to retry a failed file copy operation after a user specified delay.
  • Copy Retry Delay - this option specifies the time delay (in milliseconds) between failed file copy retries.

The 'Options' tab provides the ability to control which file meta data is copied and allows one to enable/disable various security and verification options.

Real-Time File Synchronization Options
  • Sync File Attributes - when this option is enabled, DiskBoss will copy file attributes for each copied file. By default, this option is disabled and files are created in the destination directory with normal file attributes.
  • Sync User/Group Security Descriptor - when this option is enabled, DiskBoss will copy the file ownership information for each copied file. By default, this option is disabled and files are created in the destination directory with the user account running the DiskBoss application. In addition, in order to be able to use this option, the user needs to have administrative permissions. Finally, when synchronizing files to a remote destination directory located on a NAS device, the NAS device should provide support for the NTFS security capabilities.
  • Sync Discretionary Access Control List (DACL) - when this option is enabled, DiskBoss will copy the DACL security information for each copied file. By default, this option is disabled and files are created in the destination directory with permissions inherited from the parent directory. In addition, in order to be able to use this option, the user needs to have administrative permissions. Finally, when synchronizing files to a remote destination directory located on a NAS device, the NAS device should provide support for the NTFS security capabilities.
  • Sync System Access Control List (DACL) - when this option is enabled, DiskBoss will copy the SACL security information for each copied file. By default, this option is disabled and files are created in the destination directory with permissions inherited from the parent directory. In addition, in order to be able to use this option, the user needs to have administrative permissions. Finally, when synchronizing files to a remote destination directory located on a NAS device, the NAS device should provide support for the NTFS security capabilities.
  • Enable-Bit Level File Synchronization - when this option is enabled, DiskBoss will check which data blocks have changed in each new file and write to the destination file only the changed data blocks.
  • Enable File Write Trough - when this option is enabled and DiskBoss is running on a newer operating system such as Windows Server 2012 or newer, files will be copied using the operating system native RDMA mode.
  • Verify Files After Copy - when this option is enabled, DiskBoss will verify the integrity of each copied file using the file verification mode selected on the 'Advanced' tab.

Synchronizing Specific File Types or Categories of Files

DiskBoss provides the ability to synchronize specific file types or file categories using one or more flexible file matching rules. For example, the user may specify to synchronize all types of documents and digital images with the file size more than 10 MB that were created during this year and skip all other files not matching the specified rules.

Real-Time File Synchronization Rules

In order to add one or more file matching rules, open the real-time file synchronization command dialog, press the 'Options' button, select the 'Rules' tab and press the 'Add' button. On the 'Rules' dialog select an appropriate rule type and specify all the required parameters. During the file synchronization process, DiskBoss will scan the entered source and destination directories and apply the specified file matching rules to all the existing files. Files not matching the specified rules will be just skipped from the file synchronization process.

Real-Time File Synchronization Exclude Files

Another option is to exclude files from the file synchronization process using one or more negative file matching rules. For example, in order to exclude all types of backup and archive files from the file synchronization process, add the 'Category' rule, select the 'Backup and Archive Files' category and set the operator to 'Not Categorized'.

Exclude Directories From File Synchronization

Sometimes, it may be required to exclude one or more subdirectories from the file synchronization process. For example, if you need to synchronize two directories excluding one or two special subdirectories, you may specify the top level directories as the source and destination directories and add the subdirectories that should be skipped to the exclude list.

Real-Time File Synchronization Exclude Directories

In order to add one or more directories to the exclude list, open the file synchronization command dialog, press the 'Options' button, select the 'Exclude' tab and press the 'Add' button. Keep in mind that exclude directories are case sensitive and should be specified with the same case as stored on the disk. All files and subdirectories located in the specified exclude directories will be excluded from the file synchronization process. Select an exclude directory and press the 'Delete' button, to remove the selected directory from the exclude list. In addition, advanced users are provided with a number of exclude directories macro commands allowing one to exclude multiple directories using a single macro command.

DiskBoss provides the following exclude directories macro commands:

  • $BEGINS <Text String> - this macro command excludes all directories beginning with the specified text string.
  • $CONTAINS <Text String> - this macro command excludes all directories containing the specified text string.
  • $ENDS <Text String> - this macro command excludes all directories ending with the specified text string.
  • $REGEX <Regular Expression> - this macro command excludes directories matching the specified regular expression.

For example, the exclude macro command '$CONTAINS Temporary Files' will exclude all directories with 'Temporary Files' in the full directory path and the exclude macro command '$REGEX \.(TMP|TEMP)$' will exclude directories ending with '.TMP' or '.TEMP'.

Real-Time File Synchronization Logs

DiskBoss provides the ability to save different types of file synchronization logs allowing one to keep track of performed file synchronization operations and record file synchronization errors. In order to enable file synchronization logs for a file synchronization command, open the file synchronization options dialog, select the 'Misc' tab, enable one or more file synchronization logs and specify directories where to save the log files.

Real-Time File Synchronization Logs

For each log file, the user is provided with the ability to specify the log file format (plain text or Excel CSV), the log file mode and the log files history. By default, DiskBoss will save an individual log file for each file synchronization operation, but the user can switch to consolidated hourly, daily or monthly logs files, which will be kept in the file synchronization logs directory according to the user-specified number of days.

Real-Time File Synchronization Log Options

In order to open the sync log options dialog, press the 'Options' button located beside each sync log type. On the sync log options dialog, select an appropriate sync log mode, sync log format and press the 'Ok' button.