Importer
Importers (Main Menu / Admin / Importer) are used for automated imports from various Data Sources.
Introduction
Importers in Txture are a mighty tool to populate your instance with data from all sorts of data sources. There are four types of importer:
- Asset importer: Used to import assets of any asset type.
- Link importer: Once assets have been imported, a link importer can be used to create links between them.
- Property importer: Oftentimes it is the case that several data sources have information on the same assets (e.g. a Hypervisor and a CMDB). In such a case a property importer can be used to augment data from another source to previously imported assets.
- User importer: User importers allow importing accounts from CSV, Active Directory or other means.
It's critical to understand that importers in Txture are constantly in sync. Txture keeps track of the information source for each asset and each of its properties (e.g. this asset was imported from XY with the ID Z). In the event that the importer runs again, Txture will be able to determine whether the asset is still there in the data source.
- synced: this means an asset was found in the exact same state as already present in Txture. It will remain untouched.
- updated: one or more properties of an asset have changed in the data source. The asset will be updated accordingly.
- removed: the asset is not present in the data source any more and will hence be removed in Txture as well.
For each importer run, a small overview of how many assets have been processed is presented. In addition to above states the importer statistics might also show one of the following two values:
- loaded: number of data sets found at the data source.
- imported: assets that have been imported (created) at the importer run.
Importers and the Last Modified Date
As stated earlier, importers in Txture will always synchronize the data in Txture with the data in your datasource.
Your datasource will be treated as the single (read-only) source of truth in this process.
Each importer run will try to minimize the set of changes to apply in Txture.
This means that if the data in Txture is equal to the data reported by your datasource, the importer will not touch it.
This entails that the Last Modified Date will also remain the same as before.
When importing data, the Last Modified Date therefore (literally) corresponds to the last modification, and not the last synchronization.
Creating an importer
Importers use the connection of Data Sources to extract data and allow the user to map the imported data to the internal structure of the Txture instance. When creating a new importer, you first need to type in a descriptive name and hit save. In the following importer configuration view you need to select the data source and the desired importer type (asset, link, user or property) from a drop-down menu.
In the image below you can see the configuration for a Property importer for Assets that is using a Text Data Source. You can choose whether the content is of CSV, JSON, or Txture Cloud XLSX format for a text data source.
For data sources which are compatible with only one kind of importer, the last step is done automatically in the background.
Type Resolution
For asset and link importers you also need to define the Type Resolution. Constant Type means that the source provides data for a single asset type in Txture, whereas Dynamic Type infers the asset type in Txture by identifying the type from one of the columns.
Depending on the choice of the Type Resolution, the actual mapping of the properties can be made:
- Constant Type: The importer knows the asset type and so in the drop-down menu of the Target property in Txture there are only selectable choices for this specific asset type.
- Dynamic Type: The importer does not know which asset type is imported, so the options in the drop-down menu of the Target property in Txture includes ALL available properties.
Link importers offer a third option, the Inferred Type resolution. When resolving the link type, the “inferred” option looks at the existing link types between two asset types in the Structure. If there is exactly one, this link type will be used. If there is none or multiple candidates, the importer adds an error to the log and doesn’t import the link.
The inferred type resolution is most useful when the datasource for the links doesn’t provide an explicit link type (i.e. it only provides source / target IDs) and the assets belong to different asset types. If all assets belong to the same type, a constant link type should be preferred.
Once the data source and type resolution settings have been made, the preview on the right can be loaded to validate the input.
Transform Data
After an importer has loaded a preview from the data, the processing can begin. The data transformation features offer a solid ETL layer capable of almost all required data transformation processes. Please refer to the dedicated documentation page:
- Resolve product instances: Allows querying Txture's technology databases to enhance imported data
- Parse column to date: Convert the values of a column to a date type (unix timestamp)
- Filter script: Filter scripts provide an option to filter out rows that are not meant to be in Txture by writing conditions in Groovy script
- Dynamic columns: If data does not match the required format, Groovy script in dynamic columns allow to transform it
- Resolve technologies: Allows querying Txture's technology databases to enhance imported data
After configuring the steps, the user can load a transformed preview in which the result of the transformation steps are integrated. For some scenarios a certain order is required for the transformation steps. In this case, you can drag the operations up and down using the dots to the left of each transformation step.
Asset Matching
Link and property importers require an Asset Matching between the data source and the assets within the repository. So we need to define how Txture can identify the assets for which links should be drawn or properties should be added.
In the example below you can see the Asset Matching configuration of a link importer for the link type "Virtual server > uses > Storage".
For the Source asset match an importer match rule has been defined. In this case, we use the asset importer "Azure Assets" to match the source assets. This means that we compare the Asset IDs of all the virtual servers imported via the "Azure Assets" importer with the column "Virtual Server", as defined by the Source ID input. Assets (i.e. virtual servers) with an exact match are the source assets for the links to be drawn.
The Target asset match uses a property match rule. So the importer looks at the property Unique Identifier of all the storages in the repository and compares them to the values in the column "Uses Storage" of the data source. The storages where the Unique Identifier equals the value in the column Uses Storage are selected as target assets.
This creates a link for each row of the data source between the virtual server matched by the importer match rule and the storage matched by the property match rule.
For property importers, only one asset needs to be matched per row. However, the rules themselves work in the same way as for link importers.
Property Mapping
The preview provides a final look up of the extractable data for the Property Mapping. This mapping is the final step to translate data from an external source to properties in the Txture instance.
For asset importers it is necessary to tell the importer which column of the transformed preview holds the UNIQUE ID by selecting the according column from the drop-down menu of the field Column containing Asset ID. This is important as assets are linked to this ID at their data source. If the values in the data source at this ID change, the property values in Txture will as well. If the ID vanishes in the data source, the asset will be deleted. Additionally, when importing links, these IDs can be used for Importer link matching: if a data source returns the same IDs that have been assigned as an ID at source for assets, they can be used to create links.
With a click on + Add mapping you can create a new property mapping. The Input drop-down menu lists all in the transformed preview shown columns and the Target property in Txture drop-down menu lists all properties available in Txture.
In the interest of simplification the mapping is explained on an example:
In the example above, the data source provides two columns (cpu and ram) that we want to map to Txture's data model. Within Txture, we have the corresponding properties CPU Cores and RAM. So we can map the values of the column cpu to the CPU Cores property in Txture. For the property RAM, we also need to define the Unit size of our input, which in our case is given in MiB.
Note that Txture allows to set a priority for each property. The priority comes into play, if several importers have information on the same property. Setting a higher priority will configure Txture to prefer input of this importer over values of other importers. If the higher-rated importers do not provide a value, the values of importers with lower priority will be considered one by one.
A particular case for the priority is the threshold above which importers overwrite manual changes (e.g. from the survey). The default priority for manual changes is 5, but this can be configured in the System Configuration (setting "Priority of Manual Changes").
The button Propose Mapping can be used to get suggestions for the mapping. This feature becomes more intelligent if the properties are tagged accordingly. This can be done in the Structure Editor. If no tags are available, a simple string matching is attempted.
Change data sources of an importer
It is also possible to change the data source of an existing importer. This is useful in case you want to keep all configurations you made to an importer. In the Data Source settings of an importer, you can simply change the selected Data source via the dedicated dropdown.
Note that if you have already imported assets before and you want to update them using the new data source, you need to ensure that the same asset ID is provided at the data source. Otherwise, the importer will not be able to identify already existing assets and therefore creates new ones and deletes old assets.
Available importers
Detailed instructions and peculiarities of importers are documented for the individual importers:
- i-doit
- JSON
- Kubernetes
- Red Hat Virtualization
- SQL
- Google Cloud
- Azure
- AWS
- SSH / WinRM
- VMware
- LDAP / Active directory
- LeanIX
- SCCM
Compatibility: Importer - Data Source
| Importer | Data Source |
|---|---|
| LDAP | LDAP |
| JSON | Folder, HTTP, Text, AWS S3, Google Cloud Storage |
| CSV | Folder, HTTP, Text, AWS S3, Google Cloud Storage |
| SQL | SQL |
| VMware | VMware |
| i-doit | i-doit |
| AWS | AWS |
| Microsoft Azure | Microsoft Azure |
| Microsoft Azure Migrate | Microsoft Azure Migrate |
| SNMP | IP Network |
| OpenStack | OpenStack |
| AlibabaCloud | AlibabaCloud |
| Google Cloud | Google Cloud |
| Kubernetes | Kubernetes |
| Red Hat Virtualization | Red Hat Virtualization |
| Oracle | Oracle |
| SSH | SSH, SSH Asset |
| SSH Server Configuration | SSH, SSH Asset |
| WinRM | WinRM, WinRM Asset |
| WinRM Server Configuration | WinRM, WinRM Asset |
| HyperV | HyperV |
| Jira Software | Jira Software |
| Movere | Movere |
| RVTools | Folder, HTTP, AWS S3, Google Cloud Storage |
| Stratozone | Stratozone |
| CloudScape | CloudScape |
| LeanIX | LeanIX |
| SCCM | SCCM |
| CAST Hightlight | CAST Highlight |
OpenStack Importer
The OpenStack importer has a fixed schema:
| Asset Type | Description |
|---|---|
| FLAVOR | Flavor is an available hardware configuration for a server. It defines the size of a virtual server that can be launched. |
| VIRTUAL_MACHINE | Virtual machines. It is possible to choose "only running", if you do, only the virtual machines which are actually running at the given moment will be imported. |
| IMAGE | All images. |
| DNS | "Domain Name System", all records for the given access data. |
| SUPPORTED_SERVICES | Supported services. |
| LOADBALANCER | Loadbalancer (for a detailled description, please have a look at the documentation of OpenStack.) |
| LOADBALANCER_LISTENER | Listener for loadbalancer. |
| LOADBALANCER_POOL | Pool for loadbalancer. |
| LOADBALANCER_MEMBER | Member of loadbalancer. |
| LOADBALANCER_HEALTHMONITOR | Healthmonitor of loadbalancer. |
| SECURITY_GROUP | Security groups for e.g. grouping virtual machines. |
| Link Type | Description |
|---|---|
| VIRTUAL_MACHINE_TO_FLAVOR | Associations between virtual machines and flavors will be established: which virtual machine is connected to which flavor? |
| VIRTUAL_MACHINE_TO_IMAGE | Associations between virtual machines and images will be established: which virtual machine is connected to which image? |
| VIRTUAL_MACHINE_TO_SECURITY_GROUP | Associations between virtual machines and security groups will be established: which security groups are assigned to the virtual machine? |
| LOADBALANCER_TO_LOADBALANCER_LISTENER | Associations between loadbalancer and listener will be established. |
| LOADBALANCER_POOL_TO_LOADBALANCER_LISTENER | Associations between listener and loadbalancer-pools will be established. |
| LOADBALANCER_POOL_TO_LOADBALANCER_MEMBER | Associations between loadbalancer-pools and members will be established. |
| LOADBALANCER_HEALTHMONITOR_TO_LOADBALANCER_POOL | Associations between loadbalancer-pools and loadbalancer-heathmonitors will be established. |
AlibabaCloud Importer
The AlibabaCloud importer has a fixed schema:
| Asset Type | Description |
|---|---|
| INSTANCE | Instances: virtual machines which have been created in the given region. In addition it is possible to choose "only running", if you do, only the virtual machines which are actually running at the given moment will be imported. |
| IMAGE | Images in the given region. |
| DISK | Disks providing storage in the given region. |
| SECURITY_GROUP | Security groups for e.g. grouping virtual machines. |
| LOADBALANCER | Loadbalancer in the given region. |
| Link Type | Description |
|---|---|
| DISK_TO_INSTANCE | Associations between virtual machines and disks will be established: which disk is connected to which virtual maschine? |
| DISK_TO_IMAGE | Associations between images and disks will be established: which disk is connected to which image? |
| INSTANCE_TO_IMAGE | Associations between virtual machines and images will be established: which virtual machine is connected to which image? |
| INSTANCE_TO_SECURITY_GROUP | Associations between virtual machines and security groups will be established: which security groups are assigned to the virtual machine? |
Oracle Importer
The Oracle importer has a fixed schema:
| Asset Type | Description |
|---|---|
| INSTANCE | Instances: virtual machines which have been created in the given region. In addition it is possible to choose "only running", if you do, only the virtual machines which are actually running at the given moment will be imported. |
| IMAGE | Images in the given region. |
| VNIC | VNICs in the given region. A VNIC is a virtualized Network Interface Card, used by a Virtual Machine as its network interface. |
| BOOT_VOLUME_ATTACHMENT | Boot Volumes in the given region. When launching an instance, a new boot volume for the instance is created in the same compartment. That boot volume is associated with that instance until you terminate the instance. When you terminate the instance, you can preserve the boot volume and its data. |
| VOLUME_ATTACHMENT | Volumes in the given region. A volume attachment is a detachable block storage device that allows you to dynamically expand the storage capacity of an instance. |
| Link Type | Description |
|---|---|
| INSTANCE_CREATED_BY_IMAGE | Associations between instances und images will be established: which instance was created by which image? |
| INSTANCE_TO_VNIC | Associations between instances and VNICs will be established: which VNIC is used by which Virtual Machine? |
| INSTANCE_TO_VOLUME_ATTACHMENT | Associations between instances and volumes will be established: which volume is attached to which Virtual Machine? |
| INSTANCE_TO_BOOT_VOLUME_ATTACHMENT | Associations between instances and boot volumes will be established: which boot volume was created by launching which Virtual Machine? |
| IMAGE_CREATED_BY_IMAGE | Associations between images und images will be established: which image was created by which image? |
SSH Server Configuration Importer
The SSH Server Configuration importer has a fixed schema for server assets. This importer is intended to be used as property importer.
WinRM Server Configuration Importer
The WinRM Server Configuration Importer has a fixed schema for server assets. This importer is intended to be used as property importer.
HyperV Importer
The HyperV importer has a fixed schema:
| Asset Type | Description |
|---|---|
| VIRTUAL_MACHINE | The virtual machines that are known by the HyperV server. |
| HOST | The hosts that are known by the HyperV server. |
| Link Type | Description |
|---|---|
| VM_TO_HOST | Associations between virtual machines and hosts will be established: virtual machines are running on which hosts? |
FAQ
What if my importer has a fixed schema, but the information I want to know is not listed?
The importers with a fixed schema are mostly vendor specific importers and they rely on the information that is accessible by the Data Source. When implementing the importers, we tried to get as many information out of the endpoints as possible, but sometimes the endpoints are updated and the data that can be extracted changes. So if this is the case for you, please contact our Support and tell us what data you would like to import. We will check for updates of the vendor and if possible can add a column for your desired data.
How can I import date ranges?
Import values for date ranges have to be in one of the following formats:
from 02/22/2022 to 02/24/2022
von 22.02.2022 bis 24.02.2022
Both of the above date ranges are the days in February of 2022.