fix(MCOL-4628): SET double_col=decimal_col returns an error #3531

AestheticAkhmad · 2025-05-03T15:20:53Z

MCS now does not show the ERROR 1264 (22003): CAL0002: IDB-2025: Data truncated for column 'i'. Behavior now similar to InnoDB, which allows implicit conversions with roundings.

…COL-RETURNS-ERROR

mariadb-LeonidFedorov · 2025-05-23T01:32:34Z

please uncomment roundUp parameter in

uint64_t number_uint_value(const string& data, cscDataType typeCode,
                           const datatypes::SystemCatalog::TypeAttributesStd& /*ct*/, bool& pushwarning,
                           bool /*noRoundup*/)

AestheticAkhmad · 2025-05-23T10:26:11Z

please uncomment roundUp parameter in

uint64_t number_uint_value(const string& data, cscDataType typeCode,
                           const datatypes::SystemCatalog::TypeAttributesStd& /*ct*/, bool& pushwarning,
                           bool /*noRoundup*/)

Done.

…COL-RETURNS-ERROR

mariadb-SergeyZefirov · 2025-05-26T08:10:12Z

I mostly agree.

I made a comment about 2^64-1 and possible overflow. DOUBLE type will not trigger that (it has 52 bits of precision), but wide DECIMAL with precision bigger than 38 would. If potential problem is not triggered or our behavior is in line with server's, then we are good.

AestheticAkhmad · 2025-05-26T08:58:07Z

I also talked about this problem with Roman in Zulip, and he said, "we need to get rid of old and error-prone dataconvert approach."
I haven't received a reply from Roman yet about which method he meant to use instead, however, I assume something like CAST function does in MCS:

case execplan::CalpontSystemCatalog::FLOAT:
case execplan::CalpontSystemCatalog::UFLOAT:
case execplan::CalpontSystemCatalog::DOUBLE:
case execplan::CalpontSystemCatalog::UDOUBLE:
{
return datatypes::TDouble(parm[0]->data()->getDoubleVal(row, isNull)).toMCSSInt64Round();
}
break;

Furthermore, when number_uint_value() used with negative values, the error returned are different:
MCS returns: ERROR 1264 (22003): CAL0002: MCS-2025: Data truncated for column 'i'
InnoDB returns: ERROR 1264 (22003): Out of range value for column 'i' at row 1.

mariadb-SergeyZefirov · 2025-05-27T06:46:24Z

We cannot influence error messages much, mostly because server returns MCS' messages wrapped into its' messages. What we can do is to pass through same range of values and have more or less same errors on why we do not pass erroneous values.

mariadb-SergeyZefirov · 2025-05-27T06:47:08Z

I observe some tests in old regression are failing. I will isolate them and return to you.

mariadb-SergeyZefirov · 2025-05-28T08:25:55Z

Take a look at this SQL code:

use test;
drop table if exists tcs, tid;
create table tcs(csi smallint) engine=columnstore;
insert into tcs(csi) values (-762),(-761),(761),(762);
select * from tcs where csi > -762.2;
create table tid(csi smallint) engine=innodb;
insert into tid(csi) values (-762),(-761),(761),(762);
select * from tid where csi > -762.2;

Please, run it with your changes. Do MCS with your changes and InnoDB return same result set?

This is small reproduction of one of failing tests in mariadb-regression-test suite.

…UBLE_COL-TO-DECIMAL_COL-RETURNS-ERROR

AestheticAkhmad · 2025-05-28T18:49:14Z

Take a look at this SQL code:
use test;
drop table if exists tcs, tid;
create table tcs(csi smallint) engine=columnstore;
insert into tcs(csi) values (-762),(-761),(761),(762);
select * from tcs where csi > -762.2;
create table tid(csi smallint) engine=innodb;
insert into tid(csi) values (-762),(-761),(761),(762);
select * from tid where csi > -762.2;
Please, run it with your changes. Do MCS with your changes and InnoDB return same result set?

This is small reproduction of one of failing tests in mariadb-regression-test suite.

I checked it and the results are different.

Columnstore returns basically an empty result:
MariaDB [test]> select * from tcs where csi > -762.2;
+------+
| csi |
+------+
+------+

InnoDB returns correct result:
MariaDB [test]> select * from tid where csi > -762.2;
+------+
| csi |
+------+
| -762 |
| -761 |
| 761 |
| 762 |
+------+

It is clear that Columnstore gives the wrong result.
The following query also returned wrong reuslt:

drop table if exists t1;
create table t1(i smallint) engine=columnstore;
insert into t1(i) values (-762),(-761),(761),(762);
select * from t1 where i > cast(-762.2 as int);

MariaDB [test]> select * from t1 where i > cast(-762.2 as int);
+------+
| i |
+------+
+------+

InnoDB result differs from its result when used with CAST:
MariaDB [test]> select * from t1 where i > cast(-762.2 as int);
+------+
| i |
+------+
| -761 |
| 761 |
| 762 |
+------+

Furthermore, for some reason I cannot see CI page of MCS anymore. Whenever I click "View details" on build info, the page always shows "Error 404." Previously, I could see what is happening with CI in order to check for failures.

mariadb-SergeyZefirov · 2025-05-28T20:25:16Z

I will ask Leonid about access to CI.

We need to figure out what is not right in execution of the query by MCS. My more-or-less recent build of stable-23.10 gives same result as InnoDB.

AestheticAkhmad · 2025-05-28T20:55:32Z

I will ask Leonid about access to CI.

We need to figure out what is not right in execution of the query by MCS. My more-or-less recent build of stable-23.10 gives same result as InnoDB.

Okay, could you please suggest approximately which files should I look at? Or the problem is only with the commit I made?

mariadb-SergeyZefirov · 2025-05-29T10:37:14Z

Okay, could you please suggest approximately which files should I look at? Or the problem is only with the commit I made?

It is most probably has to do with interference with other parts of the engine.

Myself, I'd like to start with SELECT calsettrace(1); then select * from tcs where csi > -762.2; then SELECT calgettrace(1) \G;

The last one should print what engine is doing.

Most probable, you will see a scanning operation with filtering.

Myself, I'd log constant values in filters (what is going on in PrimProc) to understand how -760.2 gets translated into unsigned int. The translation itself most probable occur next to the creation of SimpleFilter somewhere in dbcon/mysql/ha_mcs_execplan.cpp.

AestheticAkhmad · 2025-05-29T16:34:16Z

I guess the error is not because of my PR. I tried building using stable-23.10, and it still returned the empty result for the given query. What I checked in multiple places, the value of -762.2 is preserved. Also, this value is converted into DECIMAL, basically it has a scale of 1. So, when I logged it a string, it returned -7622 and when logged as int, it returned -762. Currently, I tried to find out where to go from here and I assume the calls happen in the following order:
-> getSelectPlan
-> buildReturnedColumn
-> buildReturnedColumnBody
-> buildConstantColumnNotNullUsingValNative
-> newConstantColumnNotNullUsingValNativeNoTz
-> buildDecimalColumn

I still cannot pinpoint where the error happens.

mariadb-SergeyZefirov · 2025-05-29T21:09:35Z

This is what I get on most recent stable-23.10:

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 5
Server version: 11.4.6-4-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> use test;
Database changed
MariaDB [test]> drop table if exists tcs, tid;
Query OK, 0 rows affected, 1 warning (0.111 sec)

MariaDB [test]> create table tcs(csi smallint) engine=columnstore;
Query OK, 0 rows affected (0.614 sec)

MariaDB [test]> insert into tcs(csi) values (-762),(-761),(761),(762);
Query OK, 4 rows affected (0.511 sec)
Records: 4  Duplicates: 0  Warnings: 0

MariaDB [test]> select * from tcs where csi > -762.2;
+------+
| csi  |
+------+
| -762 |
| -761 |
|  761 |
|  762 |
+------+
4 rows in set (0.183 sec)

MariaDB [test]> create table tid(csi smallint) engine=innodb;
Query OK, 0 rows affected (0.011 sec)

MariaDB [test]> insert into tid(csi) values (-762),(-761),(761),(762);
Query OK, 4 rows affected (0.002 sec)
Records: 4  Duplicates: 0  Warnings: 0

MariaDB [test]> select * from tid where csi > -762.2;
+------+
| csi  |
+------+
| -762 |
| -761 |
|  761 |
|  762 |
+------+
4 rows in set (0.001 sec)

I will check against your changes tomorrow.

AestheticAkhmad · 2025-05-29T22:50:33Z

I guess we have different MariaDB Server versions. Mine is 12.1.0.
Do you think this could be the problem?

Server version: 12.1.0-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> use test;
Database changed
MariaDB [test]> drop table if exists tcs, tid;
i smallint) engine=columnstore;
insert into tcs(csi) values (-762),(-761),(761),(762);
select * from tcs where csi > -762.2;Query OK, 0 rows affected, 1 warning (0.025 sec)

MariaDB [test]> create table tcs(csi smallint) engine=columnstore;
Query OK, 0 rows affected (0.071 sec)

MariaDB [test]> insert into tcs(csi) values (-762),(-761),(761),(762);
Query OK, 4 rows affected (0.007 sec)
Records: 4  Duplicates: 0  Warnings: 0

MariaDB [test]> select * from tcs where csi > -762.2;
+------+
| csi  |
+------+
+------+
0 rows in set (0.051 sec)

My git log is fresh (I have no idea why the dates are messed up, they are literally 10 days behind):

~/Projects/server/storage/columnstore/columnstore$ git log
commit 35578f5f050ccedc40106f03d72714bf1ac5135f (HEAD -> stable-23.10, origin/stable-23.10, origin/HEAD)
Author: drrtuy <drrtuy@gmail.com>
Date:   Mon May 19 19:38:54 2025 +0000

    fix(): fix API return type difference b/w server versions.

commit 7cf3003727d3318c649a08c758131e48e38fcc34
Author: drrtuy <drrtuy@gmail.com>
Date:   Mon May 19 18:45:12 2025 +0000

    fix(formating): return to the original formatting

And when I build it shows: MariaDB-Columnstore 23.10.4

mariadb-SergeyZefirov · 2025-05-30T07:40:26Z

Mine is 12.1.0.

It can very well be.

AestheticAkhmad · 2025-06-06T18:00:11Z

Mine is 12.1.0.

It can very well be.

I found out the problem. I described them here in Zulip MCOL-4628.

mariadb-SergeyZefirov · 2025-06-09T08:15:29Z

We have a problem here: build fails due to old regression suite execution timing out.

I've asked my colleagues whether two hours timeout is intended and is it normal for old regression to timeout.

Last time I saw many regressions there.

mariadb-SergeyZefirov · 2025-06-09T15:41:28Z

select * from region where not r_regionkey > (select avg(n_regionkey) from nation);
select * from region where not r_regionkey in (select n_regionkey from nation);

One of these two queries (they are executed over TPCH1 100g database, or so) return additional rows.

mariadb-SergeyZefirov · 2025-06-09T15:47:25Z

Also, there is slight typo in your MTR test file - you compare smallint column against -762.5, but it should be compared against 762.2 (-762.2). This is where I observe errors.

Do you have access to CI results?

AestheticAkhmad · 2025-06-09T15:54:47Z

Also, there is slight typo in your MTR test file - you compare smallint column against -762.5, but it should be compared against 762.2 (-762.2). This is where I observe errors.

Do you have access to CI results?

Oh, okay, I will add the test case with -762.2. Btw, this case works as well.
No, the CI website still shows Error 404, so I got no access to those tests.

…COL-RETURNS-ERROR

AestheticAkhmad · 2025-06-09T16:11:01Z

select * from region where not r_regionkey > (select avg(n_regionkey) from nation);
select * from region where not r_regionkey in (select n_regionkey from nation);
One of these two queries (they are executed over TPCH1 100g database, or so) return additional rows.

Missed this comment. Can the CI's be stopped, so they don't run for nothing. I will try to check the table's types, maybe it will give some hint which I need to address in SimpleFilter.

AestheticAkhmad · 2025-06-09T16:32:10Z

select * from region where not r_regionkey > (select avg(n_regionkey) from nation);
select * from region where not r_regionkey in (select n_regionkey from nation);
One of these two queries (they are executed over TPCH1 100g database, or so) return additional rows.

I was able to reproduce the error by making small dataset with the schema from bug 7116. InnoDB returned 2 rows, while Columnstore returned 5 rows for the first query. Second query returned same empty results for both Engines. So, the problem again is with comparison.

mariadb-SergeyZefirov · 2025-06-09T20:23:50Z

CI cannot be stopped. ;)

Try to look at 404 page and find there a sign-in/sign-up link. Then you might be able to log there again and see the results.

AestheticAkhmad · 2025-06-09T20:36:08Z

CI cannot be stopped. ;)

Try to look at 404 page and find there a sign-in/sign-up link. Then you might be able to log there again and see the results.

I tried logging in but it requires: "User must be a member of an approved organization." So, it is probably now only for who work at MariaDB.

Is there any way to know if CONSTCOLUMN is a value that comes from literal like -762.2 and not result of any aggregate or subquery?

mariadb-SergeyZefirov · 2025-06-10T05:17:10Z

select sum(lo_ordtotalprice), count(*) from lineorder where lo_custkey <= (       select max(c_custkey) from customer where c_name < 'Customer#000000084' ) and lo_partkey <= (select avg(p_partkey) from part where p_partkey between 1 and 100000);

This query (it uses Star Schema Benchmark (SSB) database schema) returns this:

sum(lo_ordtotalprice)   count(*)
309345764094.00 16348

while our code expects this:

sum(lo_ordtotalprice)   count(*)
78819825239.00  4218

This also has something to do with comparison after aggregation.

AestheticAkhmad · 2025-06-10T10:04:26Z

select sum(lo_ordtotalprice), count(*) from lineorder where lo_custkey <= (       select max(c_custkey) from customer where c_name < 'Customer#000000084' ) and lo_partkey <= (select avg(p_partkey) from part where p_partkey between 1 and 100000);

This query (it uses Star Schema Benchmark (SSB) database schema) returns this:

sum(lo_ordtotalprice)   count(*)
309345764094.00 16348

while our code expects this:

sum(lo_ordtotalprice)   count(*)
78819825239.00  4218

This also has something to do with comparison after aggregation.

Yes, I also noticed that the problem is in comparison. I always tries to make integer comparison, but why? It should instead convert to decimal if the other operand is decimal. Right now it always converts decimal to integral type for comparisons. Am I right? I guess we have to change the logic of comparisons and conversions, I just couldn't find the place where it is decided in the code.

mariadb-SergeyZefirov · 2025-06-11T10:21:30Z

Yes, I also noticed that the problem is in comparison. I always tries to make integer comparison, but why?

What is the version of the server?

Sometimes in the translation process (dbcon/mysql/ha_mcs_execplan.cpp) we rely on types provided in the server's internal structures (look at server's sql/ durectory, search for SELECT_LEX, if you wonder).

Server changes types from time to time, and it can be that we take server types.

I suspect so because all types selections (dbcon/execplan/predicateoperator.cpp, selOpType()) are more or less symmetric and follows right logic.

Look at buildEqualityPredicate in ha_mcs_execplan.cpp where setOpType is called. There can be a problem.

I would set there a breakpoint (mariadbd process) and look into operands.

AestheticAkhmad and others added 5 commits May 3, 2025 17:18

fix(MCOL-4628): SET double_col=decimal_col returns an error

8f5293b

Merge branch 'stable-23.10' into MCOL-4628-SET-DOUBLE_COL-TO-DECIMAL_…

789d3d1

…COL-RETURNS-ERROR

Merge branch 'stable-23.10' into MCOL-4628-SET-DOUBLE_COL-TO-DECIMAL_…

092b0f5

…COL-RETURNS-ERROR

Merge branch 'stable-23.10' into MCOL-4628-SET-DOUBLE_COL-TO-DECIMAL_…

a959018

…COL-RETURNS-ERROR

Merge branch 'stable-23.10' into MCOL-4628-SET-DOUBLE_COL-TO-DECIMAL_…

64f011a

…COL-RETURNS-ERROR

mariadb-LeonidFedorov requested a review from mariadb-SergeyZefirov May 23, 2025 00:39

Uncomment noRoundup var in number_uint_value

68b2f3e

Merge branch 'stable-23.10' into MCOL-4628-SET-DOUBLE_COL-TO-DECIMAL_…

33c821f

…COL-RETURNS-ERROR

Merge branch 'mariadb-corporation:stable-23.10' into MCOL-4628-SET-DO…

5149be1

…UBLE_COL-TO-DECIMAL_COL-RETURNS-ERROR

Perform conversion to DECIMAL if constant column has scale > 0

3000a2d

Added additional testcase

ac11b01

AestheticAkhmad force-pushed the MCOL-4628-SET-DOUBLE_COL-TO-DECIMAL_COL-RETURNS-ERROR branch from e151a93 to ac11b01 Compare June 9, 2025 16:02

Merge branch 'stable-23.10' into MCOL-4628-SET-DOUBLE_COL-TO-DECIMAL_…

eb3c89d

…COL-RETURNS-ERROR

fix(MCOL-4628): SET double_col=decimal_col returns an error #3531

Are you sure you want to change the base?

fix(MCOL-4628): SET double_col=decimal_col returns an error #3531

Uh oh!

Conversation

AestheticAkhmad commented May 3, 2025

Uh oh!

mariadb-LeonidFedorov commented May 23, 2025

Uh oh!

AestheticAkhmad commented May 23, 2025

Uh oh!

mariadb-SergeyZefirov commented May 26, 2025

Uh oh!

AestheticAkhmad commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mariadb-SergeyZefirov commented May 27, 2025

Uh oh!

mariadb-SergeyZefirov commented May 27, 2025

Uh oh!

mariadb-SergeyZefirov commented May 28, 2025

Uh oh!

AestheticAkhmad commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mariadb-SergeyZefirov commented May 28, 2025

Uh oh!

AestheticAkhmad commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mariadb-SergeyZefirov commented May 29, 2025

Uh oh!

AestheticAkhmad commented May 29, 2025

Uh oh!

mariadb-SergeyZefirov commented May 29, 2025

Uh oh!

AestheticAkhmad commented May 29, 2025

Uh oh!

mariadb-SergeyZefirov commented May 30, 2025

Uh oh!

AestheticAkhmad commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mariadb-SergeyZefirov commented Jun 9, 2025

Uh oh!

mariadb-SergeyZefirov commented Jun 9, 2025

Uh oh!

mariadb-SergeyZefirov commented Jun 9, 2025

Uh oh!

AestheticAkhmad commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AestheticAkhmad commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AestheticAkhmad commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mariadb-SergeyZefirov commented Jun 9, 2025

Uh oh!

AestheticAkhmad commented Jun 9, 2025

Uh oh!

mariadb-SergeyZefirov commented Jun 10, 2025

Uh oh!

AestheticAkhmad commented Jun 10, 2025

Uh oh!

mariadb-SergeyZefirov commented Jun 11, 2025

Uh oh!

Uh oh!

AestheticAkhmad commented May 26, 2025 •

edited

Loading

AestheticAkhmad commented May 28, 2025 •

edited

Loading

AestheticAkhmad commented May 28, 2025 •

edited

Loading

AestheticAkhmad commented Jun 6, 2025 •

edited

Loading

AestheticAkhmad commented Jun 9, 2025 •

edited

Loading

AestheticAkhmad commented Jun 9, 2025 •

edited

Loading

AestheticAkhmad commented Jun 9, 2025 •

edited

Loading