Data Distribution Service

Table of Contents

Section 1 Introduction
Section 2 Concepts and Terms
Section 3 DDS Components
Section 4 DDS & E-Mail Similarities and Differences
Section 5 DDS Features
Section 6 DDS Installation Types

Section 1
Introduction

Welcome

Welcome to File Express Data Distribution Service (DDS), a data distribution and document storage service. DDS is designed to provide you with traceable, guaranteed data transfer and storage between one or more systems. This overview provides a general introduction to File Express’ Data Distribution Service.

Section 2
Concepts and Terms

Introduction

This section provides concepts and terms to help you get oriented. Specifically, this section:

What is the File Express Data Distribution Service?

The DDS is a data distribution and document storage service specifically designed to provide traceable guaranteed data transfer between one or more systems. The DDS provides:

  • data transfer
  • storage
  • replication
  • archive
  • backup

The DDS also includes tools, applications, and networking services.

The File Express Data Distribution Service is the foundation of File Express’ Inter-enterprise Application Development Suite (IADS) from which File Express Inter-enterprise Applications and Services are built. File Express’ DDS deploys, operates, and manages robust Inter-enterprise Applications.

What is an Inter-enterprise Application?

The term "inter-enterprise application" is a new term many people are not familiar with. An inter-enterprise application is an application capable of spanning multiple organizational or geographic boundaries. Invoicing your customers via EDI or an Internet-based customer order system are examples of inter-enterprise applications. Integrated applications running in multiple locations for a variety of organizations can share information through messages. Tasks related to business transactions that an organization has historically performed manually can be automated through inter-enterprise applications.

Example: Inter-enterprise Application

An inter-enterprise application might allow a carrier to send invoices to ABC Company in an electronic form, such as EDI. Upon receipt of the invoices, ABC Company could have applications that:

  • verify the charge
  • cost allocate the charge
  • send disputed charge information back to the carrier
  • determine if any human review of proposed actions is required
  • send information about cutbacks and payments to the carrier electronically

Upon receipt of payment and cutback information from ABC Company, the carrier could have applications that post the payments and send the cutbacks to an audit department for review. ABC Company could have included an image of their original Bill of Lading with a memo which explained why the cutback occurred. Based on the review of cutback information, the carrier could forward information about their approval or disagreement of ABC Company's adjustments to them automatically. Both organizations could review the same information at the same time during a phone call.

How Integrated and Non-integrated Applications Share Information

The following table shows how inter-enterprise applications can share information.

Application Type How Information is Shared
Fully Integrated Applications All locations share a common set of fully integrated applications.
Common Data Format Applications Each location has its own unique application, but data is shared via commonly understood message or file formats.
Non-integrated Applications Both the data format and the application are unique to each location. Data is converted from one format to another during the data transfer through File Express’ DDS.
Mixed Integration Any combination of the other forms of application integration.

How DDS Supports Integrated and Non-integrated Applications

DDS provides the basic transportation functions that support these types of applications. DDS can convert files and messages from one system into message or file formats for another system as part of the data transfer operation. DDS data conversion capability can support information originating in any system or format, not just systems built with File Express’ Inter-enterprise Application Development Suite (IADS).

Section 3
DDS Components

Introduction

This section contains a description of the components of the DDS.

What are the components of DDS?

The major components of the File Express’ Data Distribution Service include:

What are Data Packs?

Data Packs are structured storage files based on the Microsoft Common Object Model (COM). Data Packs are used by DDS to transfer and store data. Besides providing a data transfer mechanism, Data Packs can also be used as a document-based databases for inter-enterprise applications and their documents.

What are the Characteristics of a Data Pack?

The following list describes the characteristics of Data Packs.

Data Packs:

  • have a globally unique 16-byte identifier based on Microsoft's COM specification for UUIDs
  • can transfer data in any form
  • have no size limitations
  • contain a packing slip which describes any number of other objects in the Data Pack
  • are transferred using Waybills
  • have types and application-specific attributes that can be used to determined where it is stored and what processes should be executed upon its arrival

What do Data Packs Contain?

The following table describes the nine default data types that a Data Pack can contain.

Data Types Description
Packing Slip The packing slip is required in every Data Pack. The packing slip contains information about the process that created the Data Pack, the contents of the Data Pack, and the type of Data Pack.
Operating System File An operating system file is a file from the operating system perspective. By default, all files are compressed, but Data Packs support uncompressed files as well. Using Data Packs, one or more files can be archived, backed up, or transferred from one location to multiple destinations.
Data Object The File Express Inter-enterprise Application Development Suite uses Data Packs to store application data. This application specific data is stored in streams called Data Objects. There are two types of Data Objects: class descriptions containing meta-data and object stores. Data Objects provide the functionality of a table in a Data Pack. Data Objects provide integrated support for multiple coexisting versions of objects, row specific attributes, repeating fields, and a full audit trail of object updates.
System Object System objects are streams that contain application configurations, application rules, meta-data, and screen or report formats for applications developed for or with IADS.
Compressed Image Compressed images are a collection of streams that use the File Express Image Compression System for transferring and storing document images used by applications developed with IADS.
Application Specific Data Integrated applications developed for use with File Express Data Packs can add their own application specific data types to a Data Pack for their own use.
EDI Feed An EDI feed is a file that contains an incoming or outgoing EDI transmission in the ANSI X.12 or EDIFACT format.
E-mail Messages E-mail messages are messages attached to a specific Data Object, System File, or Application Specific Data. These could also be e-mail messages being transferred between one organization’s e-mail system and another.
Fax Image CCITT image which can be faxed or transferred electronically. The DDS can be used to store, print, display, or transmit a Fax Image.

How are Data Packs Transferred?

Data Packs are normally created locally and transferred when a connection to the next system in their route is established. This process continues until the Data Pack reaches its final destination. Data Packs are routed using a Waybill which acts as a routing slip. There is one Waybill for each recipient of a Data Pack.

Data Pack Analogy

The transfer of a Data Pack is similar to shipping a box of parts in a warehouse. The box has a description of its contents, but no shipping information until a waybill is attached. By reading the information on the waybill, the transportation company knows how to get the box to its destination.

What are Distribution Points?

A distribution point provides the capabilities for shipping, receiving, processing, and storage of data. Currently Distribution Points can be installed on Windows 95 or Windows NT machines, usually as part of a LAN or as a host front-end.

A Distribution Point's local store of data is known as a Warehouse. Each Distribution Point can have multiple warehouses. Warehouses provide a way for grouping similar types of data together. Warehouses not only provide backup and archive of Data Packs locally, but they can be used as a simple document database for inter-enterprise applications.

Each Distribution Point consists of a set of applications and warehouses. Though a distribution point is thought of as a single point, it can exist on a network and be composed of many machines.

Distribution Point Components

The following table describes the major components of a distribution point.

Component Description
Warehouse Warehouses provide a convenient way of grouping Data Packs by type that have similar archival or purging requirements. Warehouses are purely logical data stores. Each Warehouse can contains multiple Warehouse Bins. A Warehouse can span multiple disks and disk directories each containing one or more Warehouse Bins.
Warehouse Bins Each Data Pack Warehouse can be divided into one or more sections called a Warehouse Bin. As Data Packs arrive at a Distribution Point, they are placed in the active Bin. Though a Warehouse can have any number of Bins, only one can be active at a time. A new current Warehouse Bin can be created automatically when the existing bin reaches a specific size or when a time event occurs such as the start of a new month.
Shipping Agents Shipping agents are applications that are capable of creating and shipping Data Packs from operating system files or external databases.
Receiving Agents Receiving agents are applications that are capable of unpacking the data in a Data Pack into an operating system file or external database.
Integrated Applications Integrated Applications are applications that:
  • read and write to Data Packs directly
  • use Data Packs as their basic document file or database
Dispatcher A Dispatcher is a File Express program that executes other programs based on an entry in the Task Queue.
Transport Agents Transport Agents provide data transfer services that move Data Packs, Waybills (routing slips), and shipment tracking information from one Distribution Point to another.
Task Scheduler The Task Scheduler is used to schedule tasks to be performed by an agent based on a specific event, a specific time, or time interval.
Distribution Point Database The Distribution Point Database stores information about the Distribution Point, Data Packs, Waybills, and distribution point components.

Note: For performance reasons it is usually best to keep the warehouse bins for a distribution point on one machine.

Data Distribution Point Diagram

The following diagram shows the structure of a File Express Distribution Point.

Diagram to go here

Section 4
DDS & E-Mail Similarities and Differences

Introduction

This section describes the similarities and differences between DDS and an e-mail system.

How is the File Express DDS Similar to an E-mail System?

In some ways, the DDS resembles and e-mail system. This section describes some of the major similarities between an e-mail system and the DDS such as focus, addressing schemes, and data transfer. This section also describes familiar e-mail concepts and their DDS equivalent.

Focus

The primary focus of an e-mail system is to transfer messages between people. The ability to add attached files was an extension of the e-mail system's basic function.

With DDS, the addressees are usually applications. Its primary focus is to provide data transfer and storage services needed to exchange, process, and store business documents electronically. You can think of File Express’ DDS as an e-mail system for inter-enterprise applications.

Addressing Scheme

Like an e-mail system, DDS has an addressing scheme that consists of two parts:

Part Description
global address known as a Distribution Point and is similar to a Post Office in an e-mail system
local address the entity name corresponding to the application or person who is the originator or recipient of the Data Pack

Data Transfer

File Express’ DDS uses a store-and-forward approach to data transfer that is similar to e-mail systems. Data can be sent to multiple recipients, yet it is stored only once at each Distribution Point. In an e-mail system, an attached file being transferred is associated with an e-mail message. The data being transferred by DDS is associated with a File Express Waybill.

E-mail Concepts and the DDS Equivalent

The following table shows familiar e-mail concepts and their DDS equivalent.

E-mail System File Express DDS
Post Office Distribution Point
To: Recipient at a Distribution Point
From: Originator at a Distribution Point
Routing Slip Waybill
Attached File Data Pack
Message No equivalent

How is the File Express DDS Different Than an E-mail System?

Though some of the components of File Express’ DDS can be compared with an e-mail system, DDS can be differentiated in its design. DDS provides the additional functionality needed to develop applications that transfer, process, and store business transactions within and between enterprises. Such functionality includes:

Process Automation

DDS is designed to support inter-enterprise applications which automatically process business transactions without requiring human involvement. To support automatic processing, DDS uses sophisticated scheduling and process queuing functions that allow the central coordination of processes running on networked computers. Processing can be controlled based on events, schedules, or the machine of execution.

Robust Recoverable Processing

All DDS processes are completely recoverable in the event of a system failure. All manually executed or scheduled tasks are executed only after an entry is placed in the Task Queue. After a task is completed, the Task Queue record is updated to indicate the task was processed successfully. If a system failure occurs, tasks which have not completed are reset and restarted.

Many business processes require multiple tasks to be performed in order to complete the business process. Multiple tasks can be placed in the Task Queue in a single commit transaction with each task being contingent on the completion of a prior task. This allows an application to specify several tasks in an order of dependency that controls the order of execution.

Event- and Time-Based Scheduling

In the business world, processing may need to occur as the result of an event rather than the passage of time. DDS provides scheduling based on events -- such as the arrival of a Data Pack in addition to time- or interval-based scheduling. All DDS and IADS processes record events in a central event registry.

There are two types of events:

  • System predefined, for example, a completion of a task.
  • Application specific, for example, the state of an invoice changing from approved to paid.

Using the DDS Scheduler, one or more tasks can be queued, messages can be sent, or additional events generated every time a specific event occurs.

Document Database

DDS Data Packs and Warehouses can be used as a document databases to store business transaction documents in image or data form. File Express’ IADS uses the DDS Warehouses as it basic document database. Data Packs are used as a document database because they have several capabilities which are not found in standard database products. In other standard database products, database tables, queries, and screens must be pre-defined in order to receive and display data.

Data Packs can contain completely self-sufficient data made up of attributes, screens, images, and files in any form. The business documents that are contained in a Data Pack do not need external or preexisting configuration information for them to be viewed, processed, or stored. For example, a Distribution Point which has never been configured to receive purchase orders can receive, store, display, and accept a purchase order without any prior configuration or development.

This data store can be used alone or in conjunction with an external relational database such as Microsoft Access or SQL Server. File Express’ Replication Agent can automatically replicate changes made in a Data Pack to an external database.

Security and Access Rights

DDS restricts each Data Pack’s access to its creator and its recipients. Recipients are the Data Pack users.

The following table describes the five levels of access rights that DDS provides.

Level Description
Owner Owner access rights are the only rights that allow the contents of a Data Pack to be updated or edited. Owner access rights are granted to the Data Pack creator by default. Only one logical user can have Owner access rights at a time.
Extract Extract access rights allow the contents of the data to be extracted by a Receiving Agent from the Data Pack into its native form.
Clone Clone access rights allow the user to create a copy of a Data Pack with a new identity where the user will have Owner access rights.
Ship Ship access rights allow the user to send the Data Pack to other recipients.
Read Only Read Only access rights allow the contents of a Data Pack to be read through IADS applications only. The Data Packs contents cannot be cloned, shipped, or extracted.

Note: Only one logical user can have Owner access rights at any time, but any number of users can have the other access rights.

Advanced Archives

Unlike e-mail systems that delete the mail from the post office once it has been delivered to the recipient address, File Express’ DDS keeps at least two permanent copies of every Data Pack. Archived Data Packs can be retrieved by the origin or destination any number of times. Specific action is required to delete an archived Data Pack. The originator and all local recipients must approve the deletions of inter-enterprise Data Packs from a Distribution Point Server or Hub.

Integrated Data Backup

Because the data handled by DDS may represent mission critical business transactions with legal and financial ramifications, DDS is designed to provide integrated data backup and archiving far superior to standard e-mail systems.

Note: Though DDS can interface with an e-mail system or be used as an e-mail bridge between organizations, it is not a replacement for an e-mail system within the enterprise.

Section 5
Data Distribution Systems Features


Introduction

File Express’ DDS provides powerful features in addition to data storage and transfer that support the development, deployment, operations, and maintenance of inter-enterprise applications. DDS features include:

Pre-requisite Specification and Checking

Inter-enterprise applications may require that each location of the application share a consistent set of software or configuration information. The order in which data is received and processed may also be significant. DDS provides prerequisites that allow applications to specify other Data Packs that must be received and processed prior to a Data Pack being sent to the destination. DDS automatically checks for the presence of prerequisite Data Packs at the destination and transfers them if they are not present. These prerequisite Data Packs can contain:

  • software or configuration information required to process the Data Pack
  • a list of Data Packs which must be processed prior to the new Data Pack

Automatic Data Compression

DDS is frequently used to transfer and store large quantities of business transactions. In order to optimize the use of systems resources, DDS offers automatic data compression for transaction data and images. This compression significantly reduces the time of data transfers and conserves disk space at Distribution Points.

Automatic Transfer and Processing Support

DDS supports inter-enterprise applications that automate the transfer and processing of business transactions. DDS provides a way of activating applications or receiving agents when a Data Pack containing a specific data type is received.

Receiving agents are applications which export data received in a Data Pack to operating system files or some non-integrated application such as a Microsoft Access database. Integrated Applications are applications that use the DDS as a data store and are capable of operating directly on the data contained in a specific type of Data Pack. Parameters that control the operation of Receiving Agents or Applications can be passed in the Data Pack.

Automatic Restart and Recovery

Restart and recovery from system failure is an important part of the DDS architecture. The DDS comes with the File Express Task Queue & Scheduler (TQS) which provides a persistent task queue. The Task Queue provides:

  • restart
  • task priority
  • task dependency
  • task history
  • status information

All DDS processes are idempotent and are automatically restarted if they did not complete successfully due to a system failure. The DDS is designed to handle data transfers of any size over intermittent links. Special attention has been paid to the problems of restart and recovery in dial-up communications over slow intermittent links. Data to be transferred is divided into blocks based on a user-definable block size. If a link failure occurs during a data transfer, the DDS automatically restarts the transfer from the last complete block.

Automatic Program Launch

File Express provides a Dispatcher with each DDS that can be used to start File Express applications and external programs based on entries in the Task Queue. The Dispatcher launches programs in order to perform queued tasks. The Dispatcher can execute any external program -- even programs that are not integrated with the DDS.

Event Registry

Using the Event Queue, the TQS can be used to execute tasks based on events instead of time. DDS has a central Event Registry that is used to record significant events such as:

  • the start or completion of a task
  • errors
  • tasks being queued
  • the arrival of a Data Pack
  • other application specific events

For example, the Event Registry could be used to:

  • start a process when a Data Pack containing a specific data type arrives
  • send an e-mail message to a system operator any time an error occurs

Communications Front-end to Any System

Though the DDS only runs on Windows 95 or NT machines, these machines can be used as a front-end communications device to handle data transfers between mainframes, AS400s, or any type of system. Since almost all computer systems have support for data transfer to microcomputers, the DDS can be used to handle data transfers in almost any environment. The DDS uses third-party products or industry standard file transfer protocols to transfer data to external systems.

Section 6
DDS Installations Types

Introduction

This section describes the different types of DDS installations. Also included is a diagram showing the transfer of a Data Pack through different types of installations.

Three Types of DDS Installations

The following table describes how File Express’ DDS can be installed:

Type Requirements Notes
Client
  • Clients are installed on a Windows 95 or NT machine
  • The Distribution Point Database can be Microsoft Access or SQL Server
  • A backup database is optional
Client DDS installations can perform all of the operations of a Server except they can only transfer data to or from a Server or a Hub. Clients are the only type of DDS installation that do not have to accept a Data Pack when they receive its Waybill. Clients can defer collecting a Data Pack due to local storage constraints or until they are ready to process the data pack.
Server
  • Servers are installed on Windows NT machines only
  • The Distribution Point Database must be SQL Server
  • A backup database is required
Servers can transfer Data Packs to and from any of the other types of DDS installations. Servers cannot push Data Packs to clients -- Data Packs are sent to clients only on request. Servers can provide Data Pack routing for routes that are manually configured and must have a default route to at least one Hub.
Hub
  • Hubs are servers and are installed on Windows NT machines only
  • The Distribution Point Database must be SQL Server
  • A backup database is required
Hubs are Servers that maintain a copy of the Distribution Point Routing Table. This table lists all Distribution Points and enables hubs to route Data Packs to their destination.

Data Pack Transfer Diagram

The following diagram shows Client A transferring a Data Pack through different types of installations to Client D.

Insert Diagram Here (Data Pack Transfer)


Copyright © File Express, Inc.