Managing Job Reports

In this section, you'll find the information you need to create and configure Job Reports, including:

• How to create a Job Report
• How to configure the Report Data Source
• How to add Components to your reports, including:
  • Layout/Structure Components
  • Content Widgets
  • Charts/Data Visualization
• How to export your reports
• How to copy your reports

To create a Job Report

Navigate to the Analyze > Reports page. You can create a Job Report with or without a template file.

• There are two ways to create a blank Job Report:
  • Click Create New Report

    or

  • Expand () the Create New Report menu and select Create Empty Report
• To create a Job Report from a template:
  • Expand () the Create New Report menu and select Create From Template File

    NOTE:  For information on the role of template files in creating Job Reports, see Exporting & Copying Job Reports.

IMPORTANT: Changes to a Job Report do not save automatically. If you navigate to another part of the platform while editing a Job Report, you will be prompted to confirm the loss of unsaved changes. Save your progress often to prevent the loss of a Job Report's configuration.

Configuring the Report Data Source

The report data source is determined by two things: the time range of Job Action results included and, optionally, custom filter rules that determine what Job Action results are included.

Individual components where the Time Range or Filter Rules do not inherit the report's Data Source have a selectable tooltip . You can click the tooltip to view which Data Source rules have been overridden.

A Job Report's Data Source showing example time, rule, and error summary configuration

To Configure the Time Range of a Job Report or Individual Component

By default, all widgets added to a Job Report adhere to the time range you configure in the Report Data Source window. You can also configure a unique data source when editing individual widgets, in the Data Filter tab. Time ranges configured at the individual component level, or the Data Filter tab, override those configured in the Report Data Source window.

Using different time ranges in a single Job Report can help you compare security technology performance between specified time frames. For example, you can add four Stacked Bar Charts and use the Data Filter tab to set each chart's Time Range to a different fiscal quarter. This lets you compare activity trends between multiple quarters.

If you are adding unique Time Range rules in the Data Filter tab, you must clear the checkbox for Use the global time range (uncheck to override).

1. In the Analyze > Reports page, create a new report or use the Actions menu  to edit an existing report.
2. In the Report Data Source window or in a component's Data Filter tab, edit the Job Report's Time Range. Select either
   • Time Until Now (Relative), which lets you specify a window of time (relative to the current date and time) for job results to be pulled from.
   • Custom Range (Absolute), which lets you define an exact window of time for job results to be pulled from.

     By default, both Time Range options pull data from the last two weeks.

3. Click Apply.

To Add Filter Rules to a Job Report or Individual Component

By default, all widgets added to a Job Report adhere to the filter rules you configure in the Report Data Source window. You can also configure a unique data source when editing individual widgets, in the Data Filter tab. If no filter rules are added, the Job Report will use all Action data in the Validation Platform.

Rules configured at the individual component level, or the Data Filter tab, override those configured in the Report Data Source window.

If you are adding Filter Rules in the Data Filter tab, you must clear the checkbox for Use the global filter rules (uncheck to override).

1. In the Analyze > Reports page, create a new report or use the Actions menu  to edit an existing report.
2. In the Report Data Source window or in a component's Data Filter tab, click Add Rule.

3. Select an attribute to match against.

   Available attribute options relate to Action details, Job Action results, and Actors.

4. Select an additional attribute trait to match against. This varies depending on the Job Action attribute you select. See Data Source/Filter Rule Values in Job Reports and the MITRE ATT&CK Dashboard for the full list of options.
5. (Optional) Select either AND to add another condition to the rule or OR (Add Rule) to add another rule.
6. (Optional) Select the vertical ellipses (  ) next to a rule to clone or delete it.
7. Click Continue.

To Include Error Summaries in Report Panels

By default, when a Job Action errors, it is not included in the pass / fail or detected / alerted / missed data for Jobs. This means errored Job Actions are not automatically included in reports you build using Report Builder. If you want to include the errored Job Actions in the report, there is an option you can select.

Error Summaries are only available in the Data Tables.

1. In the Analyze > Reports page, create a new report or use the Actions menu  to edit an existing report.
2. In the Report Data Source window select Include Error Summaries in Report Panels.
3. Click Continue.

To Copy Data Filter Details Used in Another Component

If you have configured a component's rules differently from the Job Report's Data Source, you can copy those rules for easy use in other components. For example, you can choose to copy the rules you configured for a Data Table into your Stacked Bar Chart.

Your report must have at least one component/widget with customized data source rules for you to duplicate rules between components. See Adding Layout/Structure Components to a Job Report and Adding Content Widgets to a Job Report for information.

1. Open a Job Report for editing.
2. Expand a component's action menu  , then select Edit Widget.
3. In the Data Filter tab, clear the checkbox for Use the global filter rules (uncheck to override).
4. Under Copy overridden data rules from component, select another component from the drop-down list.

   Components with zero data rules may still appear in the drop-down list and can be selected. Components that inherit the report's Data Source are unavailable and cannot be selected.

   A message appears and displays the amount of rules that will be inherited from that component.

5. Click Continue.

   Previous rule configuration is overridden and the inherited rule is applied.

To Configure tags in report builder

You can add existing tags to reports or create your own. Tags allow you to search for and filter reports by tag.

1. In the Analyze > Reports page, create a new report or use the Actions menu  to edit an existing report.
2. In the Report Data Source window, click Continue.
3. At the top of the Data Source window, click Tags +.
4. Select tags from the drop-down list by clicking the box next to the tag name.
   

   Select Tags from List

5. If you do not see the tag you want, you can create one by entering the tag name in the field and clicking Create Tag.
   

   Create Tag

6. Click Save.

Adding Layout/Structure Components to a Job Report

To Configure Layout/Structure Components

1. In the Analyze > Reports page, create a new report or use the Actions menu  to edit an existing report.
2. Click Add Component +.
   If there are components already in the report, you may need to hover the cursor above or below a component to see this option.
3. In the Panel Library, add one of the following to your report:
   • Left Weighted Two Column Container
   • Page Break
   • Right Weighted Two Column Container
   • Template Section
   • Three Column Container
   • Two Column Container
4. To add content to a container, click Add Component + inside that container.
   If there are components already in the report, you may need to hover the cursor above or below a component to see the Add Component + option.
5. Select a Content Widget or Chart/Data Visualization component to your report.
6. Optional: To duplicate or delete the container, select the component menu  and select Duplicate or Delete.
7. Optional: To move a Layout/Structure component and its contents, click the component and drag it to the desired location.

A Three Column Container with three Pie/Donut Charts inside

Adding Content Widgets to a Job Report

To Configure Text Content Editor Widgets

1. In the Analyze > Reports page, create a new report or use the Actions menu  to edit an existing report.
2. Click Add Component +.
   If there are components already in the report, you may need to hover the cursor above or below a component to see this option.
3. In the Panel Library, add a Text Content Editor to your report.
4. Enter your text content.
5. (Optional) Use the text formatting toolbar to customize the text.

Two Text Content Editors - one being used to summarize the report and the other to introduce the Pie/Donut charts that follow

To Configure Data Table Widgets

Data Tables can only be added as individual components, not inside column containers.

1. In the Analyze > Reports page, create a new report or use the Actions menu  to edit an existing report.
2. Click Add Component +.
   If there are components already in the report, you may need to hover the cursor above or below a component to see this option.
3. In the Panel Library, add a Data Table to your report.

   The Edit Widget window appears.

4. Configure the Details tab.
   1. (Optional) Enter the table's title.
   2. (Optional) Select a category to group data by.
      Grouping categories organize table data by either Action, Stage of Attack, or Behavior. For example, a table that shows the pass/fail rates of Actions based on ATT&CK Tactics can be further grouped by Stage of Attack. In this example, you would be able to compare the pass/fail performance of your security controls against each Stage of Attack while also viewing which ATT&CK Tactics fall into each category.
      Selecting a grouping category does not add a matching column to the table. By default, a message will display below the table title with an explanation of the data's grouping, but the grouping selection will not be visible in the table. It is recommended that you add a column that matches the grouping category you select.
   3. Select Add Column + and select what information to include in the column.
      Some columns can be used to sort the Data Table's contents and appear with  and  buttons. For example, you can add an Action Name column to your table, then sort the table based on Action Name (ascending or descending). Grouping selections may impact which columns can be used to sort.
      You can add Action characteristics or Job results details (Job Action) as Data Table columns. See Report Data Table Column Options for the full list of options.
5. Configure how the Data Table contents are sorted. Sorting organizes the table based on columns, while grouping sorts the table based on rows. You can either:
   If you grouped your table, sorting based on columns with Job Action Results' start/end times is not supported. Grouping selections may also impact which other columns can be used to sort.
   • Leave Sort results by Job Action run date/time selected.
     1. Choose whether to sort by ascending or descending.
   • Deselect Sort results by Job Action run date/time.
     1. (Optional) Select either  or  next to a column to sort by ascending or descending.
        Selections you make using the  and  buttons can be changed while editing the Data Table or Job Report and while viewing the Job Report.
6. (Optional) Click and drag the columns to determine their order in the table.
7. (Optional) Configure the Data Filter tab.
   See Configuring the Report Data Source for more information.
8. Click Apply.

A Data Table that displays Job Action data, grouped by Stage of Attack

Adding Charts/Data Visualization to a Job Report

To Configure Treemap Charts

1. In the Analyze > Reports page, create a new report or use the Actions menu  to edit an existing report.
2. Click Add Component +.
   If there are components already in the report, you may need to hover the cursor above or below a component to see this option.
3. In the Panel Library, add a Treemap Chart to your report.

   The Edit Widget window appears.

4. Configure the Details tab.
   1. Optional: Enter the Treemap Chart's Title.
   2. Select a data Category for the Treemap Chart to display.
   3. From Scale color by, select a Job Action result for the Treemap Chart to display.
      The color saturation of each rectangle depends on the number of Job Action results that match your selection. Less saturated rectangles represent fewer matching Job Action results, while more saturated rectangles represent higher numbers of matching Job Action results.
      Selecting Passed, Detected, Prevented, or Alerted will display data in various shades of green. Selecting Failed, Not Detected, Not Prevented, Not Alerted, or Missed will display data in various shades of red.
5. Optional: Configure the Data Filter tab.
   See Configuring the Report Data Source for more information.
6. Click Apply.

A Treemap Chart configured to display alerted Job Action Results, organized by MITRE ATT&CK Tactics.

To Configure Single Stats

1. In the Analyze > Reports page, create a new report or use the Actions menu  to edit an existing report.
2. Click Add Component +.
   If there are components already in the report, you may need to hover the cursor above or below a component to see this option.
3. In the Panel Library, add a Single Stat to your report.

   The Edit Widget window appears.

4. Configure the Details tab.
   1. Optional: Enter the Single Stat's Title.
   2. Select the display style, either Text only or With guage/donut.
   3. From Value Type, select the stat you want to display.
   4. Select whether to include a percentage that represents the Single Stat.
5. Optional: Configure the Data Filter tab.
   See Configuring the Report Data Source for more information.
6. Click Apply.

Four Single Stats that display percentages of Actions that were prevented, detected, alerted, and missed

To Configure a Spiral Chart

1. In the Analyze > Reports page, create a new report or use the Actions menu  to edit an existing report.
2. Click Add Component +.
   If there are components already in the report, you may need to hover the cursor above or below a component to see this option.
3. In the Panel Library, add a Spiral Chart to your report.

   The Edit Widget window appears.

4. Configure the Details tab.
   1. Optional: Enter the Spiral Chart's Title.
   2. Select a Value Type.
5. Optional: Configure the Data Filter tab.
   See Configuring the Report Data Source for more information.
6. Click Apply.

A Spiral Chart showing total counts of Actions that were prevented, detected, alerted, and missed

To Configure an Overview Card

1. In the Analyze > Reports page, create a new report or use the Actions menu  to edit an existing report.
2. Click Add Component +.
   If there are components already in the report, you may need to hover the cursor above or below a component to see this option.
3. In the Panel Library, add an Overview Card to your report.

   The Edit Widget window appears.

4. Configure the Details tab.
   1. Optional: Enter the Overview Card's Title.
   2. Optional: Enable Show Job Stats, which displays both the number of Actions and Job Actions in your environment.
   3. For Chart Type, select whether you want to include a chart (Stacked Bar, Job Action Results Line, Trending Threshold) or not (No Visualization).
      If you choose to include a chart in your Overview Card, you'll need to configure that chart. In that case, review the configuration steps for the Stacked Bar Chart, Job Action Results Line Chart, and Trending Threshold Chart.
   4. Select whether to include any or all Single Stat values.
5. Optional: Configure the Data Filter tab.
   See Configuring the Report Data Source for more information.
6. Click Apply.

An Overview Card set to display all available Single Stats together

To Configure a Pie/Donut Chart

1. In the Analyze > Reports page, create a new report or use the Actions menu  to edit an existing report.
2. Click Add Component +.
   If there are components already in the report, you may need to hover the cursor above or below a component to see this option.
3. In the Panel Library, add a Pie/Donut Chart to your report.

   The Edit Widget window appears.

4. Configure the Details tab.
   1. Optional: Enter the Pie/Donut Chart's Title.
   2. Select whether the chart displays as a Pie Chart or Donut Chart.
   3. Select a Value Type.
5. Optional: Configure the Data Filter tab.
   See Configuring the Report Data Source for more information.
6. Click Apply.

A Three Column Container with three Pie/Donut Charts inside

To Configure a Stacked Bar Chart

1. In the Analyze > Reports page, create a new report or use the Actions menu  to edit an existing report.
2. Click Add Component +.
   If there are components already in the report, you may need to hover the cursor above or below a component to see this option.
3. In the Panel Library, add a Stacked Bar Chart to your report.

   The Edit Widget window appears.

4. Configure the Details tab.
   1. Optional: Enter the Stacked Bar Chart's Title.
   2. Select the X Axis Type, which determines whether the X axis displays results over time or results grouped by categories.

      To select the security technologies you want to exclude/include in the report:

      1. Select Security Technologies in the X Axis Type drop-down list.
      2. Deselect/Select the check box next to each security technology you want to exclude from or include in the rendered report (all known security technologies are selected by default in the list).
      3. Search for a specific security technology by entering it in the search field.
      4. Deselect or select all security technologies by clicking the Deselect All/Select All link.
      5. Click Apply.
   3. Select a Value Type, which determines what data is shown in the chart.
   4. Select whether to include a line that shows the total number of Job Actions represented in each bar in the chart.
   5. Select the Chart column density, which allows you to choose how much time is represented by each bar. The drop-down includes the following options:
      1. Automatic (this is the default but it allows for 1 hour bars when the number of days is less than 7)
      2. 1 hour
      3. 1 day
      4. 1 week
      5. 1 month
5. Optional: Configure the Data Filter tab.
   See Configuring the Report Data Source for more information.
6. Click Apply.

A Stacked Bar Chart that shows Pass/Fail results over time and includes total counts with a line

To Configure a Trending Threshold Chart

1. In the Analyze > Reports page, create a new report or use the Actions menu  to edit an existing report.
2. Click Add Component +.
   If there are components already in the report, you may need to hover the cursor above or below a component to see this option.
3. In the Panel Library, add a Trending Threshold Chart to your report.

   The Edit Widget window appears.

4. Configure the Details tab.
   1. Optional: Enter the Trending Threshold Chart's Title.
   2. Define the Desired Rate (Percentage) and the Value Type.
      The desired percentage rate determines the threshold that needs to be met for the chart to display positive or negative trends. For example, if the desired rate is set to 95 and the value type is set to Pass/Fail, the chart displays time ranges where 95% or more of Job Actions passed as green and above the threshold line. The time ranges where less than 95% of Job Actions passed will display as red and below the threshold line. In this example, the threshold line is at 95% on the Y axis.
      
      Annotated Data Table that includes errored Job Action data
   3. Select the Chart column density, which allows you to choose how much time is represented by each bar. The drop-down includes the following options:
      1. Automatic (this is the default but it allows for 1 hour bars when the number of days is less than 7)
      2. 1 hour
      3. 1 day
      4. 1 week
      5. 1 month
5. Optional: Configure the Data Filter tab.
   See Configuring the Report Data Source for more information.
6. Click Apply.

A Trending Threshold Chart where the desired percentage rate (95%) was met in June, but not in July

To Configure a Job Actions Line Chart

1. In the Analyze > Reports page, create a new report or use the Actions menu  to edit an existing report.
2. Click Add Component +.
   If there are components already in the report, you may need to hover the cursor above or below a component to see this option.
3. In the Panel Library, add a Job Actions Line Chart to your report.

   The Edit Widget window appears.

4. Configure the Details tab.
   1. Optional: Enter the Job Actions Line Chart's Title.
   2. Select whether the chart displays counts of or percent of total Job Actions run.
   3. Select whether the chart also displays total Job Actions run in a line and on the the Y axis.
   4. Select the Chart column density, which allows you to choose how much time is represented by each bar. The drop-down includes the following options:
      1. Automatic (this is the default but it allows for 1 hour bars when the number of days is less than 7)
      2. 1 hour
      3. 1 day
      4. 1 week
      5. 1 month
5. Optional: Configure the Data Filter tab.
   See Configuring the Report Data Source for more information.
6. Click Apply.

A Job Action Results Line Chart that includes the totals line and axis

To Configure a Threshold Radar Chart

1. In the Analyze > Reports page, create a new report or use the Actions menu  to edit an existing report.
2. Click Add Component +.
   If there are components already in the report, you may need to hover the cursor above or below a component to see this option.
3. In the Panel Library, add a Threshold Radar Chart to your report.

   The Edit Widget window appears.

4. Configure the Details tab.
   1. Optional: Enter the Threshold Radar Chart's Title.
   2. Select whether the the chart pulls Job Action results in relation to Stages of Attack, MITRE ATT&CK Tactics, NIST 800-53 Control Families, or Attacker Behaviors.
   3. Select whether the chart shows data for Pass/Fail or Prevented/Not Prevented rates.
   4. Set the threshold, or Desired Rate (Percentage), to measure your Job Action performance against. The percentage you define will appear in the chart as a dashed line.
5. Optional: Configure the Data Filter tab.
   See Configuring the Report Data Source for more information.
6. Click Apply.

A Threshold Radar Chart that shows Pass/Fail performance organized by MITRE ATT&CK Tactics, with a 50% passing threshold set

To Configure a bullet Chart

1. In the Analyze > Reports page, create a new report or use the Actions menu  to edit an existing report.
2. Click Add Component +.
   If there are components already in the report, you may need to hover the cursor above or below a component to see this option.
3. In the Panel Library, add a Bullet Chart to your report.

   The Edit Widget window appears.

4. Configure the Details tab.
   1. Optional: Enter the Bullet Chart's Title.
   2. Select whether the the chart pulls Job Action results in relation to Stages of Attack, Attacker Behaviors, Attack Vector, or Zones.
5. Optional: Configure the Data Filter tab.
   See Configuring the Report Data Source for more information.
6. Click Apply.

A Bullet Chart showing the amount of Job Actions at various Stages of Attack

To Configure a Force Field Chart

1. In the Analyze > Reports page, create a new report or use the Actions menu  to edit an existing report.
2. Click Add Component +.
   If there are components already in the report, you may need to hover the cursor above or below a component to see this option.
3. In the Panel Library, add a Force Field Chart to your report.

   The Edit Widget window appears.

4. Configure the Details tab.
   1. Optional: Enter the Force Field Chart's Title.
   2. Select whether the the chart pulls Job Action results in relation to Stages of Attack, Attacker Behaviors, Attack Vector, or Zones.
   3. Select whether the chart shows data for Pass/Fail or Prevented/Not Prevented rates.
   4. Set the Result Display Style to Count or Percentage.
5. Optional: Configure the Data Filter tab.
   See Configuring the Report Data Source for more information.
6. Click Apply.

A Force Field Chart showing the percentage of Job Actions Prevented/Not Prevented at various Stages of Attack.

Exporting Job Reports

Reports created with the Report Builder can be exported as a PDF or as a template file. The PDF will be in Portrait mode with a white background and will contain everything in that Job Report, including unsaved changes, in the format you have configured. The template file is used to easily share reports with other Validation Platform users.

TO EXPORT A JOB REPORT AS A PDF

1. Navigate to the Analyze > Reports page and open a report.

2. Expand () the Export menu.

3. Select Export PDF.

TO EXPORT REPORT CHARTS

1. Navigate to the Analyze > Reports page and open a report.

2. Expand () the Export menu.

3. Select Export Report Charts.

TO EXPORT A JOB REPORT AS A TEMPLATE

1. Navigate to the Analyze > Reports page.
2. Identify the desired Job Report, expand its Actions menu  and select Export Template File.

NOTE: You can also use the Export menu described in the previous section.

Copying Job Reports

Copying your reports lets you quickly repurpose existing reports build out others.

TO SAVE A COPY OF A JOB REPORT

1. Navigate to the Analyze > Reports page.
2. Open or edit a report.
   1. If viewing a report, expand the  drop-down menu and select Save a Copy.
   2. If editing a report, expand the Save drop-down menu and select Save a Copy.
3. Enter a name for the report copy.
4. Click Save Copy. The copied report opens in the edit view.