Release Notes for MongoDB Connector for BI

Improvements

• Now built with Go 1.21.12+ to address CVE-2024-24791 for all operating systems, including the following operating systems:

  • RHEL 7.0

  • RHEL 8.0 zSeries

  • Windows 64

  • macOS 13 arm/OpenSSL 3.0

  • Ubuntu 18.04 container

• MongoDB Connector for BI now supports the upcoming MongoDB version, 8.0.

Improvements

• Now built with Go 1.21.12+ to address CVE-2024-24791 on all operating systems except for the following operating systems:

  • RHEL 7.0

  • RHEL 8.0 zSeries

  • Windows 64

  • macOS 13 arm/OpenSSL 3.0

  • Ubuntu 18.04 container

Improvements

• Internal updates for the MongoDB Connector for BI.

Improvements

• Updates the packaging on Windows.

• Upgrades golang.org/x/crypto from v0.8.0 to v0.17.0.

• Introduces the following SSDLC policy updates:

  • Adds SBOM Lite support.

  • Integrates Silk vulnerability scanning.

  • MongoDB Connector for BI generates and publishes a public third-party dependency report.

Bug Fixes

• Fixes a bug where MongoDB Connector for BI sampling stalled due to aggregate error.

Improvements

• Now built with Go 1.21.0+.

• Adds MacOS 13.0 ARM support.

• Adds Debian 12 support.

Improvements

Adds support for the following distributions:

• Amazon Linux 2023 ARM

• Amazon Linux 2023 x86

• RHEL9 ARM

• RHEL9 x86

• Ubuntu 22.04 ARM

• Ubuntu 22.04 x86

Bug Fixes

• Fixes a bug where MongoDB Connector for BI used the same unwindPaths map for all databases when adding foreign keys during catalog building.

Improvements

• Now built with Go 1.20.7 to fix CVE-2023-29409.

Bug Fixes

• Fixes a bug where schema wouldn't load when a column of a virtual table from an array is renamed.

• Fixes a bug where foreign key calculation returned an error when there was no _id field.

Improvements

• Now built with Go 1.20.6+ to fix CVE-2023-29406.

Bug Fixes

• Fixes a bug where a cursor timeout in mongod didn't result in an error message or error log in MongoDB Connector for BI.

Improvements

• Now built with Go 1.19.10.

Improvements

• Now built with Go 1.19.9+.

Improvements

• Now built with Go 1.19.0.

Bug Fixes

• Fixes an issue where setting the schema mode to auto would always trigger initial sampling even when a preexisting stored schema was available.

Bug Fixes

• Fixes the amazon2-arm URL for the release JSON feed.

Behavioral Changes

• Adds support for MongoDB 6.0.

• Adds MacOS 11.0 ARM support.

• Adds Debian 11 support.

• Adds Enterprise RHEL 8 zSeries support.

• Removes all zSeries/IBM POWER PC platforms support except for RHEL 8 zSeries.

• Removes RHEL 6.x/7.x on S390x support.

Improvements

• Now built with Go 1.16.0.

• Adds support for MongoDB 5.0.

• Adds the RHEL8 ARM distribution to the download center.

Bug Fixes

• Fixes an issue where empty field paths were not skipped when mapping the subpaths of objects that conflict with scalars.

• Fixes an issue where tarballs for the RHEL 8.1 (IBM POWER) 2.14.3 release were incorrectly structured.

Behavioral Changes

• Removes support for MongoDB 3.2, 3.4, and 3.6.

• Removes support for zSeries distributions.

New Features

• Enables the use of externally sourced values in .yaml configuration files.

• Adds support for the following fields in .yaml configuration files:

  • trim

  • type

  • digest

  • digest_key

Bug Fixes

• Fixes an issue where documents with null values were incorrectly returned from some queries.

• Fixes an issue where some queries took longer to complete than in previous versions of BI Connector.

• Fixes an issue where the listDatabases command returned an error when read preference was set to an analytics node in some sharded cluster configurations.

Improvements

• Now built with Go 1.5.0.

• Adds the Amazon Linux 2 ARM 64 distribution to the download center.

New Features

• Detached signatures are now available for release archives.

Bug Fixes

• Fixes an issue that resulted in inefficient translations for in and or expressions.

Improvements

• Now built with Go 1.15.

Bug Fixes

• Fixes an issue in which geospatial data did not map correctly.

• Fixes an issue with regard to DRDL extended character encoding.

• Fixes an issue in which sharding information failed to load for mixed-case databases.

• Fixes an issue in which the response to COM_FIELD_LIST was malformed.

New Features

• Database schemas are now case-sensitive.

• Queries containing UNION can be fully pushed down.

Bug Fix

• Object-array conflict resolution produces NULL data for conflicting object fields nested in array.

Bug Fixes

• Fixes bug in mapping that did not deep copy pipelines during table post-processing, leading to invalid SQL schemas.

Bug Fixes

• Fixes bug that caused panics on document keys containing characters that need to be escaped, such as " or \usb.

Bug Fixes

• Fixes bug that caused index loading to fail with mixed-case databases.

• Fixes connectivity bug caused by non-nil interface in custom go-driver dialer.

• Fixes mongotranslate not including pipelines defined on tables in DRDL file schemas.

• Fixes bug that caused natural left joins to fail in some cases.

Bug Fixes

Fixes the following bugs:

• unix_timestamp returning incorrect results because it did not handle Daylight Savings Time when necessary.

• DRDL tables incorrectly marshalling to BSON if the aggregation pipeline was empty.

Improvements

Version 2.13.0 includes the following improvements:

• Updates an error message for FLUSH SAMPLE authorization that was unclear.

• Updates schema mapping to skip empty field names and field names containing . characters.

• Enables self-join optimization for sharded collections.

• Replaces the max_num_columns_per_table system variable with max_num_fields_per_collection.

• Enhances performance while sampling collections with many fields by reducing the amount of memory used.

• Improves ambiguous column behavior in subqueries, GROUP BY and ORDER BY clauses.

• Adds support for Ubuntu 18.04, SLES 15, and RHEL8.

• Adds non-string support for regexp.

• Improves performance for schemas that contain arrays.

Bug Fixes

Fixes the following bugs:

• mongosqld hangs if an election occurred during sampling.

• Some queries using $lookup fail.

• Panics occur when connecting with a DRDL file that excludes a child table's _id.

• Versions 2.10+ install to the wrong directory on Windows.

• Fixes a bug in the div operator’s in-memory evaluation.

• mongosqld incorrectly kills long-running queries.

• Some queries on ObjectId fields do not push down.

• mongodrdl not parsing replica set seedlists passed to the --host flag.

• timestampadd fails with out-of-range months.

• Allows aggregate functions as grouping expressions.

• datediff fails to push down with polymorphic arguments.

• Displays incorrect foreign key constraints.

• Incorrect pushdown and type conversions for date and datetime types that are too large or too small.

• Aliases for tables in information_schema are ignored.

• Some queries with aggregate functions fail to push down.

• Pipeline parsing error for unwind paths with numeric field names.

• mongodrdl ignores the --gssapiHostName and --gssapiServiceName flags.

• Allows extra arguments to the current_timestamp and trim functions.

• str_to_date returns a date instead of a datetime for some non-constant format-string arguments.

Improvements

Behavioral Changes

• Adds new reserved words to the parser: INSERT, INTO, KEY, FULLTEXT, and PRIMARY.

  Going forward, these words must be wrapped in backticks to be used as identifiers in queries.

• Removes support for altering sampled schema with ALTER TABLE statements.

• Prior to 2.12, mongosqld supported renaming and removing tables and columns via a modified ALTER TABLE syntax when in standalone schema mode. Any customer currently using this feature is advised to migrate their schema management needs to views, DRDL, or custom schema mode. The variable/config parameter that enabled this feature, enable_table_alterations, is still supported but doesn’t do anything in BI Connector 2.12. It will be removed in a future release.

Bug Fixes

Fixes a bug that caused the MySQL schema to include incorrect foreign key constraints.

Fixes a bug that caused incorrect pushdown for some very large or very small date and datetime values.

Fixes a bug that caused aliases to be ignored for tables in the information_schema database.

New Features

• Adds a number of optimizations that are applied to generated aggregation pipelines before sending them to MongoDB.

• Adds new subcommands to mongodrdl for storing a DRDL file in MongoDB and for manipulating stored schemas.

• Enables mongosqld to use stored DRDL schemas instead of sampling. To learn more, see Schema Management Changes in 2.11.

• Introduces the beta version of mongotranslate, a command-line utility that can translate SQL queries to aggregation pipelines without a running BI Connector or MongoDB instance.

Bug Fixes

• Fixes a problem that caused the BI Connector to generate unnecessarily long $group key names.

• Fixes a bug that caused the BI Connector to generate incorrect table pipelines in polymorphic schema mapping mode.

• Fixes a bug that caused the BI Connector to panic when printing some query plans.

• Fixes a bug where the BI Connector’s in-memory evaluation of string-to-date conversions did not match the pushdown behavior.

• Updates the supported Amazon Linux version to match the version supported by MongoDB 4.2.

• Fixes a bug with Active Directory constrained delegation in the BI Connector’s GSSAPI implementation (linux and macOS).

• Introduces a --gssapiConstrainedDelegation flag that must be enabled to ensure existing Kerberos configurations are not broken.

• Fixes a bug that caused Kerberos authentication to fail with a linux mongosqld, MongoDB 4.0, and an Active Directory KDC.

• Fixes a number of small bugs related to correlated subqueries.

Chages/Deprecations

• Changes some schema-management configuration options.

• Deprecates schema-modification support via ALTER TABLE, as well as the enable_table_alterations system variable and setParameter

Bug Fixes

• Fixes a bug where aggregate functions returned NULL in dual queries with the DUAL keyword omitted.

• Fixes a bug where some invalid DUAL queries did not return an error.

• Fixes a bug that prevented pushdown of some joins to derived tables selecting from DUAL.

• Fixes some correctness bugs in the in-memory evaluation for mysql-mode conversions.

• Fixes a bug where some trig functions (ascii, acos, and asin) could return incorrect results due to missing type conversions.

• Fixes a bug where some subquery comparisons could return incorrect results due to missing type conversions.

• Fixes a bug where we lost precision while converting a Uint64 to a Decimal.

• Fixes a correctness bug with the INSERT scalar function's pushdown translation.

• Fixes a bug where unstable sorting caused non-deterministic renaming of colliding collection/field names during schema mapping.

• Fixes a bug where some valid subquery comparisons returned errors.

• Fixes a bug where some valid subquery and tuple comparisons returned errors.

• Fixes a bug where queries using xor returned errors on MongoDB 3.2 because the generated pipelines required features added in MongoDB 3.4.

• Fixes a bug that caused kerberos authentication to fail with MongoDB 4.0.

Date and Time Changes

This release resolves a number of inconsistencies in the BI Connector's date and timestamp semantics. Resolving these inconsistencies resulted in some changes to user-facing behavior:

• Removes inconsistent support for partial dates.

  • MySQL can represent partial dates that are missing month or day components. It is impossible to represent these incomplete dates with MongoDB dates. Attempting to correctly manipulate and render incomplete dates was causing a number of bugs and inconsistencies with complete dates.

  • Starting with version 2.10, the BI Connector does not support operations on or expressions that return incomplete dates. Going forward, we cannot provide guarantees about the behavior of operations that attempt to construct incomplete dates.

• Updates the DATE(a) and TIMESTAMP(a) scalar functions to be equivalent to CAST(a AS DATE), and CAST(a AS DATETIME), respectively.

  • The behavior of the DATE and TIMESTAMP scalar functions will depend on the value of the type_conversion_mode variable. This will change the default behavior for those scalar functions.

  • Queries affected by this change can be rewritten to use a more explicit conversion function (such as str_to_date) that does not depend on type_conversion_mode. If an affected query cannot be rewritten, a user may set type_conversion_mode variable to mysql, which instructs the BI Connector to perform all type conversions with MySQL semantics instead of MongoDB semantics.

    Important

    This is discouraged unless absolutely necessary, as it significantly curtails the ability to push down type conversions across the board.

  • For more information on type conversions and the type_conversion_mode system variable, see Type Conversion Modes.

Features and Performance Improvements

• Includes a sample mongosqld configuration file in the release archive.

• Allows setting commonly used system variables in the configuration file.

• Optimizes conversions of years to days.

• Optimizes join performance.

• General performance improvements.

Bug Fixes

• Improves the BI Connector's resiliency to queries against fields that may contain unsupported BSON types.

Features and Performance Improvements

• Improvements to the BI Connector’s type system and the way it handles polymorphic data.

• Users can alter the type of a given column.

• Users can access “dirty” datasets without query errors.

• Support for pushdown of the GROUP_CONCAT aggregation function.

• Non-admin users can set global variables.

Bug Fixes

• Improvements in parsing strToDateTime to float.

• Fixes to how the BI Connector honors readPreference specifications.

Features and Performance Improvements

• Collapse associative functions

• Support the optimize_view_sampling global variable

• Support the polymorphic_type_conversion_mode session/global variable

• Improve filter optimization for MongoDB versions 3.6 and later

• Expose type_conversion_mode in configuration parameters

• Cache catalog columns for quicker lookup

Bug Fixes

• Fix incorrect semantics for null safe equality operator ( <=> )

• The || operator returns incorrect type (and value) when operating on two strings

• Add DRDL type validation

• Support CAST to unsigned

• Fix WHERE clause removed in certain cross-join queries

• Fix some incorrect left self-join results when dynamic schemas are used

• Support turning off view optimization with $match

• MongoDB BI Connector ODBC Driver on macOS should install in versioned directories

• Fix MongoDB BI Connector ODBC Driver installer issue on macOS that caused unicode driver to be reported as ANSI

Released version 1.2.0 of the MongoDB BI Connector ODBC Driver.

EXPLAIN Support

Users can use EXPLAIN to return details of aggregation pipeline used to generate query results.

Common Table Expression Support

The BI Connector now supports queries that include Common Table Expressions.

New SSPI Support

Windows Clients may now use kerberos to authenticate against the BI Connector.

NULL Valued Types Are Now Cast to Varchar During Sampling

This resolves the issue where queries would fail due to NULL values being ignored during sampling. Now, NULL values will be cast as SQL type VARCHAR.

SIN, COS, TAN Are Now Pushed Down

BI Connector will now push down SIN, COS, and TAN SQL functions. This will improve performance for queries using these functions.

Aggregate Expressions

BI Connector 2.4 improves support for aggregate expressions including math, date, and string functions.

Improved Support for Subqueries and Non-equijoins

With version 2.4, mongosqld has improved support for subqueries and non-equijoins.

Improved Schema Management

In BI Connector 2.3, mongosqld includes the schema mapping functionality previously found in mongodrdl.

Log Rotation

With version 2.3, you can rotate the mongosqld logs. To rotate the logs, issue a command or restart mongosqld.

Authentication Plugins

MongoDB Connector for BI Version 2.0 introduced passthrough authentication, in which clients authenticate as a user defined in MongoDB, without the need to create a separate user within the connector itself.

MongoDB Connector for BI Version 2.2 further increases ease of use with a set of authentication plugins which can authenticate users through the connector without the need for SSL certificates. These plugins perform user authentication via the SASL framework and support the SCRAM-SHA-1 and PLAIN authentication mechanisms. We currently provide authentication plugins in the following languages:

• Java - for use with MySQL’s Connector/J driver

• C - for use with the MySQL ODBC driver and the MySQL Shell

Improved Server-Side Execution

The MongoDB Connector for BI delivers significant performance and scalability improvements by moving as much execution as possible to the underlying MongoDB instance. Pushing execution down to the server reduces network latency and reduces bandwidth consumption.

This approach also reduces memory and CPU consumption in the MongoDB Connector for BI itself.

Pass-through Authentication

The user authentication process is now passed through the MongoDB Connector for BI to the underlying MongoDB instance. Users now authenticate as a user already created within MongoDB.

Simplified Architecture

The MongoDB Connector for BI package is now trimmed down from four separate components to two:

mongodrdl

This tool connects to a MongoDB instance and generates a Document-Relational Definition Language (DRDL) file, which maps a given collection shape to relational schema.

mongosqld

Once installed and run as a daemon, mongosqld responds to SQL queries, requests for DB information, and schema from the requesting BI tool via the MySQL wire protocol.

mongosqld requires at least one DRDL file to start. You will need to have defined your schema with mongodrdl before starting mongosqld.

SQL Compatibility

MongoDB Connector for BI Version 2.x is compatible with SQL-99 SELECT statements, and uses the MySQL wire protocol.

MongoDB Connector for BI 2.0.2

Released Jan 5, 2017

• Hides .pem key file passwords in logs.

• mongodrdl ignores binary fields when generating DRDL files.

• Supports queries that SELECT variables prefixed with "mongodb".

• Fixes incorrect result set from a CROSS JOIN.

• Supports leading "." characters on table names.

MongoDB Connector for BI 2.0.1

Released Dec 13, 2016

• Fixes precedence issue with the NOT operator.

• Fixes error looking up privilege information in databases with capital letters.

• Supports LIKE pattern matching in SHOW COLUMNS queries.

• Improves formatting of aggregation pipeline in logs.

• Fixes substring function when the combined length of the starting index and substring exceeds the length of the string.