Work with Active Learning Enabled Review Passes

Set Up Active Learning Enabled Review Passes

Notes:

  • In addition, we recommend that you see Manage Review Passes for a general understanding of the Review Pass process.

  • Active Learning disabled review passes still run Active Learning “in the background”, but it is transparent to reviewers. At some point in the review, the setting could be Active Learning enabled, to start prioritizing predicted positive documents within the review pass, and that would change the document order for reviewers.

  • If you prefer to turn off the Active Learning functionality running “in the background” on Active Learning disabled review passes, the functionality can be turned off across the environment through SQL. For more information, please contact IPRO Support.

Set Up the Review Pass

  1. Make sure that you have performed the steps in “Create a Active Learning enabled Review Pass” in Create Review Passes and Batches

  2. Open the respective case. In the left pane, click the Review Pass Management button . Click the Active Learning enabled review pass you created in Create Review Passes and Batches.

    The Review Pass Details screen for that review pass appears.


  3. In the top right corner of the Review Pass Details screen, click the edit icon . The Edit Review Pass Settings screen appears. This screen contains the settings that display in the Settings tab of the Review Pass Details screen.

  4. Update Review Pass Name, General settings, and Permissions settings.

  5. The following table describes the options you can edit on the Edit Review Pass Settings screen.

    SETTINGS

    DESCRIPTION

    General

    Active Learning

    Switch on - enabled

    Switch off - disabled

    Coding Form Coding form being used for the review pass
    Primary Review Purpose

    Allows IPRO Analytics to learn from the work done by reviewers. If the Primary Review Purpose tag is applied to a document, then the system will learn from that document.

    • If the document is marked "Reviewed" and the Primary Review Purpose is applied, then that document is recorded as a "positive"

    • If the document is marked "Reviewed" and the Primary Review Purpose is not applied, then that document is recorded as a "negative"

    Reviewers should use the On Hold review status when reviewing documents if they are unsure whether the document is a positive or negative example. This prevents the document from being added as an example until a review decision can be made.

    Recall

    The estimate of the percentage of positive documents in the entire case. The desired recall for the review pass can be configured from 70% to 90% in increments of 10%.

    Batch Size

    The number of documents a reviewer will receive at a time

    Layout Style

    Determines the view when the review pass is opened

    Document Scope

    The scope of documents that Active Learning will use train documents. Select from:

    • Entire Case (recommended - Active Learning works best when the entire case is used)

    • Saved Search - (due to the nature of Active Learning, often it is best to not filter the content submitted to a project, but Saved Search can be used if necessary)

    • Custom Index - (intended for advanced users. Allows the user to create an index in Eclipse Admin ahead of time. This gives them additional options for configuration, and then the index can be selected for use in the Review Pass.

    Permissions

    Group(s)

    Contains Groups assigned to the normal batches in the review pass

    Review QC Groups(s)

    Controls the QC batch permissions. Contains Groups assigned to the QC batches within the review pass.

    Note: Not selecting either of these group types allows access to both types of groups.

    If a Review QC Group is selected, users who are part of that group will always receive a QC Batch for the Review pass if one is available. If a QC Batch is not available and the user is part of the regular review group, they will receive a normal batch.

  6. Click . In the left pane, the enabled review pass listing displays with a icon in its top-right corner.

Settings Tab

On the Review Pass Details screen, you can view the values that you set by clicking the Settings tab. See the previous table for a description of these settings.

Reports Tab

Note: All the charts load asynchronously, so they load quickly.

On the Reports tab is the following:

The Relevancy Scores graph, which shows the distribution of documents based on relevancy prediction:

Hovering over a bar shows the Relevancy Index for that score:

When the Relevancy Range is set wide, say -30 to 30, then the Relevancy Index Range Details shows that the documents that fall into the range of being predicted as positive are the same or close to the entire set of review documents. The Relevancy Range can be adjusted as you desire. If you narrow the Relevancy Range, for example, to between 60 and 80, fewer documents will display within the Relevancy Range Details.

Clicking a link will trigger a Visual Search for that particular document set.

The Desired Recall graph. This graph shows for various numbers of documents the estimated number of available desired recall points for recall levels 70%, 80%, and 90%, and recall precision:

Hovering over a bar shows a breakdown of documents and recall precision:

The Estimated Predicted Relevant Documents graph, which shows the estimated number of documents predicted to be relevant over batches:

Hovering over a point shows the manually reviewed documents for the particular batch:

Overview Tab

Note: The Review Progress chart loads asynchronously, so it loads quickly.

The Overview tab can contain the following sections:

Sections

DESCRIPTION

Review Metrics Breakdown of selected tags in checked-in documents

Precision is the percent likelihood that you will hit a positive document as you continue reviewing. For example, if you review 100 documents and 90 of them are positive, then the Precision is 90%.


Review Progress

Manually reviewed checked-in documents by date

Documents Manually Reviewed Listing of manually reviewed documents

Batch Information

Batch details for the review pass

In-app Information bubbles are available next to the Review Metrics for additional information.

See the following example:

Note that the Insights card now contains Prevalence and Responsive Documents values:

Batching Workflow

Note that at the bottom of the Review Metrics for Responsive section there is a card that allows the batching workflow to be switched between a Standard Active Learning workflow and an Ordered workflow.

Example 1 - Switching from Ordered workflow to Standard Active Learning workflow

When an Ordered workflow is set, switching back to the Standard Active Learning workflow will push the higher scoring example documents to the front of the line for Review. Note that the option is not available if no unreviewed example documents are present.

Click Switch to Standard to make the switch. A dialog box displays to make sure this is what you want to do.

Example 2 - Switching from Standard Active Learning workflow to Ordered workflow

When a Standard Active Learning workflow is set, switching to an Ordered workflow will deviate from the Standard Active Learning workflow to push higher scoring documents to the front of the line for Review. Note that adding additional documents to the review pass will automatically revert this selection.

Click Switch to Ordered to make the switch. A dialog box displays to make sure this is what you want to do.

Active Learning Enabled Review Pass QC Batches

QC Batches can be created as a part of an Active Learning enabled review pass. Only records that are part of the original review pass can be added to a QC batch. They can be used for almost any purpose, but most often are used for:

Coding decisions made outside of the Active Learning enabled review pass will not modify the Positive or Negative status of the example, so it is important to use the QC Batch option if the coding change is intended to apply to the Active Learning project.

Using Mass Action, one QC Batch is created for all records submitted with the mass action process. The default batch size selected for the review pass does not apply.

Creating an Active Learning Enabled Review Pass QC Batch

  1. Filter down to the records in your grid view.

  2. Using the Mass Action option, select Create QC Batch.

  3. Select your Active Learning enabled review pass from the selector.

 

Related pages:

Create Review Passes and Batches

Manage Review Passes

Work with Active Learning Disabled Review Passes

Work with Mass Actions

Unlocking the Benefits of Active Learning.pdf