What is TDE?

Template Driven Extraction (TDE) allows you to:

• Define a relational lens over your document data, so you can query your data using SQL or the Optic API.
• Use templates to define a semantic lens, specifying which values from a document make up triples in the triple index.
• Generate rows and triples from ingested documents based on predefined templates that describe the following:
  • The input data to match
  • The data transformations that apply to the matched data
  • The final data projections that are translated into indexed data.
• Access the data in your documents in several ways, without changing the documents themselves.

Importance of user roles:

• All operations on Template documents are controlled by:
  • http://marklogic.com/xdmp/tde collection
    • A protected collection that contains TDE template documents
  • tde-admin role
    • Required to access the TDE protected collection
  • tde-view role
    • Required to view documents in the TDE protected collection
• Insertion/Deployment:
  • A Deployment needs a tde-admin role

More about TDE here.

Inserting templates/views: Tutorial

Updating a View:

If you want to create a view to support more than one context or scope, you can update the corresponding view declaration. This can be done either by updating the existing template with the updated view declaration or by creating a new template with the updated view declaration using the same view name.

Before understanding how updating a view is done, one must be familiar with the concept of a 'viewLayout'. A view declaration comes with a viewLayout option which can be set to one of these two options: 'identical' or 'sparse'.

• Identical: A viewLayout is identical by default unless explicitly set to sparse. When a viewLayout is identical, it expects all the templates created in the future using the same view name to be consistent with the view declaration of the existing view
• Sparse: A sparse viewLayout allows you to update an existing view with a new declaration with added/removed columns

While in most cases it is not possible to update a view if the viewLayout is identical, this is not true if a view is associated with only one template. In this specific scenario, you can indeed update an identical viewLayout without error.

In contrast, when a view is shared between multiple templates, the ability to update it either by re-inserting an existing template associated with that view or by creating a new template depends on whether that viewLayout is identical or sparse.

• When the viewLayout is shared between multiple templates and identical:
  • Updating the view is not allowed and results in the error Invalid TDE template: TDE-INCONSISTENTVIEW
  • This is because the new/updated templates view declaration is expected to be consistent with the existing one
• When the viewLayout is shared between multiple templates and sparse:
  • Updating the view is allowed provided it satisfies the following conditions:
    • All 'not-nullable' columns from the existing view must be a part of the updated view
    • The 'nullable' field of all the new columns being added in the updated view declaration must be set to 'true' - this is effectively indicating that these columns are optional

More information about Creating Views from Multiple Templates here

Best Practices:

Given the complexity involved in the process of updating views, it is recommended to follow these best practices when creating views:

• It is very important to understand the goal of creating a view. How will it be used now? How is it expected to be used in the future?
• Make sure to set the viewLayout to the correct option at creation as this ultimately informs whether or not a view can support more than one context or scope in the future.
• If one wishes to opt for a sparse viewLayout, it is very important to set the 'nullable' field to the correct option when declaring columns. It is not possible to remove columns after they have been created unless they're 'nullable'.
• If you have created a view that you'd later like to alter or edit, and that view falls under a scenario where alterations or edits are no longer allowed, to make further changes you will need to delete then recreate the relevant view.