Fixed structured data keeps businesses running day in and day out. But supporting only fixed data is also extremely limiting to structured data processing. There is no reason that highly principled structured data processing has to be limited to static data structures. Dynamic data structures enable a new set of powerful, unlimited data processing opportunities. As an example, it would be very useful to have a highly flexible and principled hierarchical data processor to support dynamic structured data processing automatically. This article presents an example of such a dynamic structured data processing application and how it operates.
Dynamic Structured Data Processing
Static structured data processing is to dynamic structured data processing what the teletype is to the telephone. Dynamic structure data can be several orders of magnitude greater than static structured data for ease of use, flexibility and solving problems. This opens up an entirely new, faster, easier and more powerful world of uses for structured data. This flexible dynamic structured processing allows changes to the structure of the data as it becomes necessary to solve the problem at hand.
For dynamic structure processing to take advantage of its dynamic processing, it must be able to automatically process its dynamically changing associated metadata. This is needed to keep the dynamic structure processing running at full throttle to avoid being slowed by manual metadata update overhead.
To demonstrate this new dynamic structured processing, I will use a complex distributed peer-to-peer user collaboration example performed freely across remote unrelated peer locations. It transparently supports unpredictable dynamic structured data and data type changes automatically for immediate processing. Such an automatic dynamic structured data collaboration operation is depicted Figure 1 with its dynamic hierarchical data structure being modified at each remote peer site labeled P1 to P4.
Figure 1: Dynamic Structured Data Processing Peer-to-Peer Collaboration
Diagram Processing Description
In Figure 1, distributed peer locations P1, P2, P3, and P4 located anywhere need to collaborate and share their structured data in order to produce the complex distributed result. This will require previously unsupported processing of unpredictable changes in the data structure and data types as it becomes necessary to achieve the desired result. In this example, this process uses SQL’s natural hierarchical processing as a way to demonstrate this dynamic structure data capability that exists today as an experimental prototype. It can be tested here.
Peer 1 starts the data collaboration process by inputting three relational tables – A, B, and C – modeling them into a hierarchical structure and then sending it to Peer 2 for further processing. Concurrently overlapping with Peer 2’s processing, Peer 1 also inputs a XML linear hierarchical structure, XYZ, transforms it into a nonlinear multipath hierarchical structure and sends it to Peer 3 for further processing.
At this point, Peer 2 and Peer 3 are now performing independently and concurrently. Each is retrieving their structure input from Peer 1, inputting additional relational table data from their different peer home locations, and joining this data to their working data structures. On completion, Peer 2 and Peer 3 both send their modified data structures to common Peer 4 for further processing.
Peer 4 accepts the modified data structures from both Peer 2 and Peer 3, which operated concurrently. It then hierarchically joins them together using a hierarchical LEFT JOIN and a matching ON join clause argument, hierarchically linking nodes B and X (B.bb=X.xx). Peer 4 then eliminates unneeded data items from the joined result using SQL’s dynamic SELECT operation to select data items for output from nodes A, B, E, Y and W. This SQL query looks like: SELECT A.a, B.b, E.e, Y.y, W.w FROM P2View LEFT JOIN P3View ON B.b=X.x. The SELECT operation slices out all nodes (C, D, Z, X, V) that were not referenced by the SELECT statement. This automatically aggregates the necessary data as shown in Figure 1. This process is known as projection in relational processing and node promotion in hierarchical processing.
Peer 4’s LEFT JOIN operation hierarchically places P2’s structure over P3’s structure connected by the ON clause specification of: B.b=X.x. and demonstrates how the LEFT JOIN syntax enables keeping track of the data structure. This newly combined hierarchical structure in Peer 4 is sent back to Peer 1 for immediate review and processing where the hierarchical data can be selectively output in different formats, each with different data selections, also shown in Figure 1.
Automatic Metadata Maintenance
The current problem with performing this type of dynamic structured processing is that dynamically generated structured data cannot be automatically handled today. Sharing structured data is performed with shared metadata. The metadata remains the same, so the structure must remain static. But with dynamic structured processing, the data structure can be dynamically and automatically modified as needed to support the required structured operation as shown and described above. This requires Automatic Metadata Maintenance, which previously has not been supported for structured data processing. This automatic process handles all changes to the metadata as necessary, making its operation transparent. This is necessary for dynamic structure processing to operate freely and efficiently.
During the entire collaboration process described above, the changing data structures and data types are automatically maintained and utilized transparently for the user as needed. The user at each receiving peer can also view the current active structure and its data types. But knowledge of the structure is not necessary for the user to specify in the query because the maintained structure is automatically known and used inherently by the query processor. Different working data structure versions can also be saved and restored at each peer by the user.
Integrating Automatic Metadata Maintenance With SQL
The ANSI SQL transparent hierarchical processor prototype mentioned earlier supports the required dynamic and flexible structured data processing necessary for the dynamic structure processing shown in this article. In addition, it uses SQL’s inherent hierarchical data processing capabilities that naturally support full multipath dynamic hierarchical data structures. This allows the most complex hierarchical structured data operations to be performed to always meet the required processing requirements.
The SQL hierarchical prototype already operates structure-aware because of its dynamic processing capability. This is necessary for supporting the required Automatic Metadata Maintenance. The SQL hierarchical prototype’s dynamic hierarchical processing technology can be enhanced to integrate with structured data processing collaboration to eliminate the user control necessary for the dynamically changing metadata. The Automatic Metadata Maintenance automatically supplies the updated current metadata. This metadata accompanies the data when transmitted between peers. This will allow amazingly fast. on-the-fly advanced hierarchical dynamic structured data processing collaboration. This enables unexpected structure processing requests delivered to any peer to be immediately processed by the SQL hierarchical prototype located at the peer location.
Peer-to-Peer Processing
The SQL hierarchical prototype can also be easily updated to control the peer-to-peer process of sending and receiving data structures using new SQL InFile and OutFile keyword operations added to the SQL hierarchical prototype for the purpose of transmitting automatic updated metadata between peers. Password and data encryption can also be supported for data security. Further dynamic processing and data structure modification can be performed at each peer visited, in any order, including in parallel as shown Figure 1. This opens up the new capability of dynamic structured data processing and its automatic and transparent metadata handling.
Advanced SQL Hierarchical Processing Capabilities for Structured Data Collaboration
The SQL hierarchical prototype can also dynamically processes heterogeneous logical flat data such as relational and physical hierarchical data (e.g., XML). This dynamic hierarchical data processing enables logical and physical structures to be hierarchically joined and modeled dynamically. This hierarchical processing significantly increases the power of the data structure and the queries applied to it. This is extremely flexible and powerful and is automatically performed without user navigation or knowledge of the data structure. This operation naturally utilizes the hierarchical semantic information between the pathways to process powerful multipath queries. This allows the most complex data collaboration problems to be solved quickly and easily.
Conclusion
This article demonstrated dynamic structured data processing. This is important because it opens the door to more powerful structured data processing capabilities and their applications. An example of such a powerful application was given that requires dynamic structured data processing. I also covered how Dynamic Metadata Maintenance is necessary for dynamic structured data processing to work. And also important, this article also demonstrates how a powerful dynamic structured data processing and Automatic Metadata Maintenance work together to achieve dynamic structured data processing, which is easier to use and many times more powerful with unlimited new capabilities. If there is any interest or questions about this new dynamic structured data processing technology, I can be reached at: mike@adatinc.com.