Squirro 3.6.8 LTS Release Notes

Announcements
1

Squirro 3.6.8 long-term support (LTS) has been released :star_struck:.
It highlights the progress since the previous LTS release 3.5.9. Most of these additions were already introduced in the intermediate releases of the 3.6.x series and have been documented in those release notes as well.

As a long-term support (LTS) release, Squirro 3.6.8 will receive updates for security issues and important bug fixes for the next two years. See the Squirro Release Process document for details on Squirro’s versioning.

What’s New :star_struck:

Squirro Monitoring :nerd_face:

Now selectable from the Application Setup page, Squirro Monitoring allows you to review user and project activity data within the Squirro interface. Previously, you needed server SSH access to check logs, which was problematic with installations on multiple nodes.

To learn more, see https://squirro.atlassian.net/wiki/spaces/DOC/pages/2776432645.

The ingester service and the pipeline steps now create structured logs which populate the squirro_v9_data_ingestion index. Two dashboards are available within Squirro Monitoring:

Activity Dashboard

This dashboard shows user activity information and trends.

The Activity Dashboard includes the following improvements:

  • User activity logs for Question Answering (QA) and SimilarSearches widgets.
  • Query context information is forwarded to the query activity log.
  • Configuration options in the Activity Tracking step were exposed to dynamically enrich activity logs with specified User Data. Information is then stored in labels within the Monitoring Project. This can be configured under Setup > Pipeline (edit) > Activity Tracking widget, allowing for customization of collected data from users.

Data Ingestion Logs Dashboard

This dashboard shows project activity information and logs, including errors.

The Data Ingestion Logs Dashboard includes the following improvements:

  • New display names for facets.

  • Step Duration is now measured in milliseconds.

  • Start and finish messages are now more descriptive.

  • New facets were added: Step Name, Step Execution Mode, Successful Items, and Errored Items.

  • Exceptions are included in the body and formatted correctly for the built-in steps executed in the ingester service.

  • Two metrics widgets for number of logs and number of errors were added

Cognitive Search Scoring Profiles

Introduced a feature that allows you to customize document relevancy scoring by ranking on additional signals. For more detailed information, see Scoring Profiles on the Squirro Docs. You can apply Scoring Profiles on project, group, or user levels.

You can think of scoring profiles as a way of building your own “smart results” query-processing workflow. For detailed instructions on using the feature, see How to Use Scoring Profiles to Customize Document Relevancy Scoring.

The figure below shows how Scoring Profiles fit into the overall query-processing workflow:

Additional improvements to Scoring Profiles include:

  • You can now associate scoring roles with Squirro groups, or third-party user metadata (coming from SSO provider).
  • QueryProfile: Personalized document relevancy scoring by allowing user metadata matches. This is done by templating user information in the configured Query Relevancy Scoring Profile.
  • Added popular-item boosting profile. This allows you to use the overall item-popularity (amount of item.read events) as an additional ranking signal.
  • Elasticsearch scripts are now supported to customize document relevancy scoring.
  • Introduced query within the Scoring Profile Execution Stages. This applies profiles on all matching items vs. rescore (rescore only top N ranked items). This is useful if you want to apply more expensive operations on a smaller, more relevant subset.

Spell Check Improvements

The Spell Check Widget suggestion application is now case insensitive and only applies to user input terms, not on other query syntax components such as filters or comparisons.

Also added full support for multilingual spell checking. You don’t have to set the project languages, Squirro will automatically recognize languages used within the project.

Custom Widget Improvements

In late 2021, Squirro began migrating its widgets, including custom widgets, from Backbone.js to React. Now, React widgets are a core part of Squirro’s frontend technology stack.

The most significant functional difference you’ll notice when working with custom widgets is that only specific methods and views are exposed in React widgets. This makes React widgets more resilient to change, and as an end user it’s much simpler to override parts of a widget than it is to copy an entire template, as Backbone.js requires.

And, with Squirro Storybook, you’re able to preview changes and work with widgets dynamically. Learn more by reading the Squirro Storybook Docs Page.

Current LTS additions and improvements to React custom widgets include the following:

  • Now useDrilldown hooks are exposed in React custom widgets. See Available React Hooks.
  • You can now a build custom widget using the Tabs widget as a base.
  • Added a new component preview to Storybook.
  • Similar Searches widget (used for the Other People Ask feature) is now customizable and documented in Squirro Storybook.
  • Added new default templates for React widgets under Setup > Visualization > Custom Widgets > Add.

Document Level Feedback in AI Studio

Document level feedback allows you to apply feedback (:+1: or :-1: ) on the labels associated with individual documents. This differs from typical labels that are applied within the text. Document Level Feedback allows you to assess the validity of the label for the document as a whole in addition to specific sections of the document’s text.

You can find this feature by navigating to Setup > AI Studio > Classification > Start AI Studio > Ground Truths. You’ll need to establish Candidate Sets if you haven’t already.

You can generate feedback on any type of document.

Community Auto-Subscriptions

You can now auto-subscribe to communities using the new Communities Auto-Subscription feature. This is particularly useful for situations where you own or manage a large number of communities you want to follow automatically.

You can configure auto-subscription by navigating to Setup → Settings → Project Configuration → topic.community.auto-subscription as shown below:

For more details on configuring auto-subscription, see Log in with Atlassian account

Lemmatized Search

Search recall has been increased by the introduction of additional language-specific, lemmatized search. This requires an active Query Processing Workflow that uses the LemmaExpander libNLP step that tracks the lemmatized version of relevant query terms during searches.

The LemmaExpander step is now enabled by default within the default query-processing workflow.

General Cognitive Search and Insight Engine

  • You can now clear query search by pressing Enter when your cursor is in the empty global search bar.
  • When searching, the /parse endpoint results are now passed to the item collection requests. This helps Squirro understand the intent of searches and activate relevant features such as Question Answering.
  • Introduced a search query-classifier step that infers query type by detecting whether the query is written in natural language form. Question Answering is then performed on queries in natural language form.
  • Left alignment of the compact global search is now available. To set it, navigate to Setup > Visualization > App & Nav Bar and then select
  • Added document inline search queries to typeahead history. Searches made inside of documents were previously not included in the typeahead history and now they are.
  • Added Create Facet to the pipeline configuration for ML enrichment steps.
  • Added a libNLP step that supports ML models in the ONNX format. You can now load ONNX models using ONNXRuntime.
  • Warning messages in the Pipeline Editor now inform you when steps in a section are missing or when steps in a pipeline workflow are missing required information. The specific error message is shown rather than a generic one.
  • The React Tabs widget now also supports Material icons by default. You can use Material icons by inserting the Material icon name.

Improvements

Dashboards and Widgets

  • The CommunitiesList widget was migrated from Backbone.js to React.
  • Improved React widgets performance and optimized collection fetching. Now a loading bar is shown during the collection fetch of React widgets.
  • Added searchable and visible properties when creating and modifying dashboards.
  • Enhanced context search across all dashboards.
  • Now profile pictures are validated at upload to prevent potential malicious actions. Only JPEG and PNG file types are allowed.

libNLP

  • There is now support for multiline parameters when writing custom libNLP steps.
  • Within libNLP steps, you can now mark parameters as deprecated. This makes the step optional when configured. For default libNLP steps, marking a parameter as deprecated will automatically reflect in the libNLP documentation.

Communities

  • Reordering community types in the Setup space is now allowed. Widgets take the order of communities as defined in the Setup space.
  • Aggregations communities collection factory is now exposed for custom widgets API.
  • Added logic to augment community data using external sources.
  • Added pagination to community subscriptions.
  • Added Community 360 navigation support to subscribed communities list.
  • Added an API to fetch all follower information from a community. This allows for the display of followers’ information in the Community 360 dashboard.

General

  • Improvements to the Items detail SidePanel hide and show logic.
  • ItemsWidget will show Item Read time for last read items instead of Item Created time.
  • Disabling old file removal in pdfconversion is now permitted. This is a storage versus performance issue with caching and is useful in projects where you do not want multiple real-time PDF conversions.
  • Added visibility conditions (NLP Search / is empty / is not empty) to the Global Search state.
  • Now filesystem_plugin has better MIME type detection and will include the body field for text documents.
  • Now ES_TMPDIR allows you specify the default temp directory within the Elasticsearch config file.
  • The available pipeline steps in each section of the Pipeline Editor are now sorted by name.
  • The maxRequests option of the forked FastCGI flup server is now supported. You can use this to limit the total number of requests. It can be defined in the max_requests config option in the server section of a service’s INI file, similar to min_spare and max_spare. It has an effect only when the fork config option is set to true. For more information see the maxRequests documentation on Github (external link).

Documentation

Breaking Changes

3.6.6

  • Implemented pagination to the community subscription endpoint. The default API call will no longer return all subscribed communities.
  • Upgraded squirro-platform to work with Elasticsearch 7.17.6. This requires an Elasticsearch upgrade. This is covered below in the upgrade instructions.
  • Removed the gdfs_plugin package as it is no longer used.

3.6.3

  • Update tensorflow to 2.8.2 and its related dependencies
  • spellcheck parameter in the /items/query endpoint accepts (whether to check the spelling of the provided query) boolean value instead of dictionary value. If a dictionary is passed in, spellchecking is turned on, but previously possible configuration values are ignored.

3.6.1

  • [Search] Introduce a new app step in the ML workflows and use it in the Query Processing workflow. It means that custom query processing workflows won’t longer work. If you have a custom query processing workflow, please delete it and clone the default one. Aggregate into 3.6.0 and remove 3.6.1

3.6.0 - 3.6.1

  • [Search] Configuration: common.ini setting facet_boosting_values was replaced with project configuration-service setting topic.search.field-boosts
  • [Search] The steps for custom query-processing workflow changes and existing custom query workflows are no longer working. To resolve the issue, recreate the workflow by cloning the default workflow.
  • [libNLP] Removal of unused steps:
    • base: batch_randomizer
    • analyzer: classification_analyzer, topic_modeling_analyzer
    • classifiers: cnn_seq2one, rnn_seq2one, seq2one, seq2seq, sequence
    • clusterers: gmm, kmeans, lda
    • embedders: dictionary, doc2vec, glove
    • filters: clear, presence, window
    • recommenders: user_feature
    • tokenizers: word_ngram
  • [libNLP] removal of the aggregation normalizer step + util functionality for reservoir sampling
  • [libNLP] Marking of steps as deprecated (still working by rerouting to a maintained step):
    • base: log
    • embedders: tfidf
    • tokenizers: sentences
  • The cacheable option specifying whether to cache a document in the NLP step is no longer valid. Instead, use the new option called cache_document in the workflow configuration.

Installation and Upgrade

The following config file must be resolved when upgrading from Squirro 3.6.4 or earlier:

  • /etc/nginx/conf.d/frontend.conf

You will have to resolve at least the following config files when upgrading from Squirro 3.3.0.

  • /etc/nginx/conf.d/ssl.inc
  • /etc/squirro/common.ini

For new installations, please follow the Setup on Linux instructions.

To upgrade an existing installation, please consult the Upgrades for Squirro 3.5.3 and later guide.

2 Likes