Advanced properties

Note: Advanced properties are only available to Fabric customers.

This page provides additional information about advanced property types available in the Software Catalog. For a general overview of properties, see the Software Catalog overview.

File matching rules

Overview

File matching rules make it possible to read file contents in each repository and extract pattern matches. This allows you to track facts about your services that live directly in source code.

Examples of questions that file matching rules can answer:

  • Does this service have a “Local development” section in its README?
  • Does this service run unit tests in a GitHub Actions workflow?
  • Does this service have an up-to-date version of a critical internal library dependency?
  • Does this service use an internally-managed Docker base image?
  • Does this service still have any function calls out to a vendor we’re migrating away from?

How to set up a rule

Prerequisites

  • Create a Repository Integration for your source code management tool.
  • Ensure your catalog entities have aliases set up so DX can match them to repositories.

Steps

  1. Navigate to Catalog Settings and create a property of type File matching rule.
  2. Click Define rule to open the rule editor.
  3. Choose a rule type: File exists, Substring or Regular expression.
  4. Enter a file path — this must be a static path (glob patterns and recursive searches are not currently supported).
  5. Enter a match expression — the substring or regular expression to look for in the file.
  6. Use the sample file contents field to paste in example file content and test your configuration.
  7. Use the sample entity selector to quickly test configuration with live data in the catalog.
  8. Review the results and confirm the rule behaves as expected.
  9. Click Save changes.

File matching rule editor dialog

How to view results

File matching rule results are visible in the following places:

  • Entity detail page — displayed alongside other property values.
  • Entities list page — available as a column in the entities table.
  • Scorecard checks — use the related_properties column to surface the property in the check result sheet.

The results are displayed in a list, specifying the number of matches (or if the file was not found):

Entities List page, showing list of file matching rule pills

If the result contains one or more matches, clicking on it will open a dialog showing a table with each match:

File matching rule dialog

Rule execution

File matching rules are re-evaluated every 4 hours. Additionally, individual rules are ran immediately after being created or updated.

Scorecard checks with file matching rules

The status field in the entity property value will be one of three strings:

  • FOUND_MATCHES
  • NO_MATCHES
  • FILE_NOT_FOUND

If an entity does not have a repo alias covered by a repository integration, there will be no corresponding record in dx_catalog_entity_properties.

To write a scorecard check based on file matching rules, you can leverage the status field in one of the following two ways:

Identify Good matches — if you want to check that the pattern specified was found (e.g. “README contains a Local development section”):

WITH result_record AS (
  SELECT (ep.value->>'status')::text AS STATUS
  FROM dx_catalog_entity_properties ep
    JOIN dx_catalog_entities e ON e.id = ep.entity_id
    AND e.identifier = $entity_identifier
    JOIN dx_catalog_properties p ON p.id = ep.property_id
    AND p.identifier = 'readme-local-dev-section'
)
SELECT CASE
    WHEN (
      SELECT coalesce(status, 'FILE_NOT_FOUND')
      FROM result_record
    ) = 'FOUND_MATCHES' THEN 'PASS'
    ELSE 'FAIL'
  END AS status,
  'readme-local-dev-section' AS related_properties;

Identify Bad matches — if you want to check that the pattern specified was not found (e.g. “package.json does not contain legacy-vendor-xyz”):

WITH result_record AS (
  SELECT (ep.value->>'status')::text AS STATUS
  FROM dx_catalog_entity_properties ep
    JOIN dx_catalog_entities e ON e.id = ep.entity_id
    AND e.identifier = $entity_identifier
    JOIN dx_catalog_properties p ON p.id = ep.property_id
    AND p.identifier = 'package-references-legacy-vendor-xyz'
)
SELECT CASE
    WHEN (
      SELECT coalesce(status, 'FILE_NOT_FOUND')
      FROM result_record
    ) = 'FOUND_MATCHES' THEN 'FAIL'
    ELSE 'PASS'
  END AS status,
  'package-references-legacy-vendor-xyz' AS related_properties;

Computed properties

Overview

Computed properties use a SQL query to compute a value for each entity of an entity type. Values are re-computed hourly. Creating a computed property or editing its definition also triggers a recompute right away.

Computed property SQL can be written in one of two formats: single-entity or batch format. Both formats will achieve the same result, but batch formats will be processed in an optimized way for complex queries and large datasets. Batch format is recommended if your organization has over 5,000 entities or if your computed property query is complex.

  • Single-entity format — the query runs once per entity and uses the $entity_identifier variable to reference the current entity. Other single-entity variables, such as $entity_github_repo_ids, are also available.
  • Batch format — the query runs once for a batch of entities and uses the $entity_identifiers variable to reference every entity in the batch. Because the work is done in far fewer queries, batch properties can compute much faster, especially for entity types with many entities.

Both formats are supported, and you can choose whichever fits your use case. If your query relies on alias variables like $entity_github_repo_ids, use the single-entity format. Otherwise, consider the batch format for faster computation.

Single-entity format

In the single-entity format, $entity_identifier gets replaced with the identifier of the entity being computed during execution. The query must return an output column - the first row of this output column is used as the computed value.

Single-entity queries can also use the other entity query variables available to scorecard checks, such as alias variables like $entity_github_repo_ids and $entity_github_repo_names. Alias variables follow the pattern $entity_<alias_type>_ids and $entity_<alias_type>_names. These variables can’t be used in batch queries.

SELECT COUNT(o.entity_id) AS output
FROM dx_catalog_entities e
  LEFT JOIN dx_catalog_entity_owners o ON e.id = o.entity_id
WHERE e.identifier = $entity_identifier;

Single-entity queries keep only the first row returned. Any additional rows are ignored. Single-entity queries time out after 5 seconds.

If a query references an alias variable and the entity has no alias of that type, the entity gets a “Missing variables” error instead of a value.

Batch format

In the batch format, $entity_identifiers is replaced with the identifiers of every entity in the batch, as a comma-separated list of quoted string literals (for example, 'service-a','service-b','service-c'). Don’t add your own quotes around $entity_identifiers. The query must return exactly one row per entity.

Each batch holds up to 200 entities, and batch queries time out after 30 seconds.

Batch queries must follow these rules:

  • Return an entity_identifier column along with the output column, so DX can match each row to its entity. If the column is missing, every entity in the batch gets an error.
  • Return one row per entity. If an entity has more than one row, that entity gets a “multiple rows” error.
  • Use $entity_identifiers on its own. It can’t be combined with other variables, such as alias variables like $entity_github_repo_ids. To use alias data in a batch query, join to dx_catalog_entity_aliases and dx_catalog_entity_alias_entries instead, as shown in the Open PR count example.
  • Don’t mix $entity_identifier and $entity_identifiers in the same query.

There are two common ways to work with $entity_identifiers:

  • Filter catalog tables with IN ($entity_identifiers), for example WHERE e.identifier IN ($entity_identifiers).
  • Turn the identifiers into rows with unnest(ARRAY[$entity_identifiers]), which is useful when you don’t need to join to dx_catalog_entities.

Any entity in the batch that has no row in the result gets a “No results returned” error. To make sure every entity gets a row, start from the list of entities and use LEFT JOIN for related data, rather than starting from the related data.

Testing batch queries

The Run button in the query editor needs a sample entity. For batch queries, the preview substitutes only that one identifier for $entity_identifiers, and uses the 5 second single-entity timeout. A batch query can pass the preview and still time out when it runs against a full batch of up to 200 entities, so keep heavy joins scoped to the entities in the batch.

Converting a single-entity query to batch format

  1. Replace = $entity_identifier with IN ($entity_identifiers), or select from unnest(ARRAY[$entity_identifiers]).
  2. Add the entity’s identifier to the SELECT list as entity_identifier.
  3. Add a GROUP BY on the entity identifier if the query aggregates values, so each entity returns exactly one row.

Batch format examples

Owner count

Counts the owners assigned to each entity.

SELECT e.identifier AS entity_identifier,
  COUNT(o.entity_id) AS output
FROM dx_catalog_entities e
  LEFT JOIN dx_catalog_entity_owners o ON e.id = o.entity_id
WHERE e.identifier IN ($entity_identifiers)
GROUP BY e.identifier;

Open PR count

Counts open pull requests across each entity’s repository aliases. The entities CTE narrows the catalog down to the batch first, then the rest of the query joins from there.

WITH entities AS (
  SELECT e.id,
    e.identifier
  FROM dx_catalog_entities e
  WHERE e.identifier IN ($entity_identifiers)
)
SELECT entities.identifier AS entity_identifier,
  COUNT(gp.id) FILTER (
    WHERE gp.merged IS NULL
      AND gp.closed IS NULL
  ) AS output
FROM entities
  LEFT JOIN dx_catalog_entity_aliases dcea ON entities.id = dcea.entity_id
  LEFT JOIN dx_catalog_entity_alias_entries dceae ON dcea.id = dceae.entity_alias_id
  LEFT JOIN repos r ON dceae.identifier = r.external_id
  LEFT JOIN github_repositories gr ON gr.source_id::text = r.external_id
  LEFT JOIN github_pulls gp ON gr.id = gp.repository_id
GROUP BY entities.identifier;

Open bug count

Counts open issues that mention the entity’s identifier in their title and are labeled as bugs.

SELECT id AS entity_identifier,
  COUNT(i.title) AS output
FROM unnest(ARRAY[$entity_identifiers]) AS id
  LEFT JOIN issues i ON i.title ILIKE CONCAT('%', id, '%')
  AND i.title ILIKE '%bug%'
  AND i.completed IS NULL
GROUP BY id;

System properties

The following property types are system properties, meaning they cannot be edited via the UI or the web API:

  • File matching rule — values are computed from repository file contents
  • Computed — values are computed from a SQL query