Skip to content

NH-4026 - Upgrade Firebird driver and use server #639

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
Jun 10, 2017
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
7c8aa5a
NH-4026 - upgrading Firebird driver and using server instead of embed…
fredericDelaporte Jun 2, 2017
4d36a19
NH-4026 - A connection string parameter is missing for addressing Fir…
fredericDelaporte Jun 2, 2017
fd84733
NH-4026 - some other changes just to try, although my laptop does not…
fredericDelaporte Jun 3, 2017
fe0e8ac
NH-4026 - fix the test database setup (or at least tries to).
fredericDelaporte Jun 3, 2017
728ede7
NH-4026 - let the TestDatabaseSetup create the db.
fredericDelaporte Jun 3, 2017
704c122
NH-4026 - trying to document the relevant things I have done to final…
fredericDelaporte Jun 3, 2017
96ac0d3
NH-4026 - Firebird locks table till connection are actually closed, n…
fredericDelaporte Jun 4, 2017
ab0410c
NH-4026 - some more explanations.
fredericDelaporte Jun 4, 2017
da88449
Revert "NH-4026 - some more explanations."
fredericDelaporte Jun 4, 2017
d75f7ef
NH-4026 - some more explanations.
fredericDelaporte Jun 4, 2017
8b832cf
NH-4026 - fixing remaining failing tests.
fredericDelaporte Jun 5, 2017
e794efe
NH-4026 - remove ignoring previously failed.
fredericDelaporte Jun 5, 2017
d04fafa
NH-4026 - fix a typo in default test config file. Align naming on oth…
fredericDelaporte Jun 6, 2017
3a5395e
NH-4026 - disabling forced writes and simplifying setup.
fredericDelaporte Jun 7, 2017
d71f510
NH-4026 - Updating instruction file
fredericDelaporte Jun 7, 2017
b42cfb3
NH-4026 - relocating the db, and ensuring forced write option by drop…
fredericDelaporte Jun 7, 2017
907b5cf
NH-4026 - removing explicit dependency on D: from TestDatabaseSetup.
fredericDelaporte Jun 8, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions ShowBuildMenu.bat
Original file line number Diff line number Diff line change
Expand Up @@ -91,14 +91,12 @@ goto test-setup-generic
set CONFIG_NAME=FireBird
set PLATFORM=x86
set LIB_FILES=lib\teamcity\firebird\*.dll
set LIB_FILES2=lib\teamcity\firebird\x86\*
goto test-setup-generic

:test-setup-firebirdx64
set CONFIG_NAME=FireBird
set PLATFORM=x64
set LIB_FILES=lib\teamcity\firebird\*.dll
set LIB_FILES2=lib\teamcity\firebird\x64\*
goto test-setup-generic

:test-setup-sqlitex86
Expand Down
Binary file modified lib/teamcity/firebird/FirebirdSql.Data.FirebirdClient.dll
Binary file not shown.
19,131 changes: 0 additions & 19,131 deletions lib/teamcity/firebird/NHibernate.Test.last-results.xml

This file was deleted.

34 changes: 34 additions & 0 deletions lib/teamcity/firebird/firebird_installation.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@

Installation steps for Firebird for NH TeamCity:

1. Download Firebird (Firebird-3.0.2.32703_0_x64): https://www.firebirdsql.org/en/server-packages/;
2. Run the installer AS ADMINISTRATOR... Use the default firebird password when prompted: masterkey.
3. Leave other settings with their defaults.
4. The setup should install Firebird on the machine;
5. Go into Firebird folder (c:\program files\firebird\) and create a folder named Data;
6. Go in Firebird installation directory and open databases.conf;
7. Add in "Live Databases" section:
nhibernate = D:\SqlData\Firebird\nhibernate.fdb
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not make it local?

Copy link
Member Author

@fredericDelaporte fredericDelaporte Jun 8, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Local to the database server? Would be on C:, which is not local to the host. Local to the NHibernate test client? The path seems not frozen, only embedded mode can do that, where the client is the server too. I think Firebird mainstream usage is as an actual server, not as embedded mode. So I would rather test it as an actual server. And as showcase by Nathan branch, Firebird v3 embedded mode seems to need a new dialect.

When using a database server, does it really make sens to locate the database relatively to the client? That should be the server choice, not the client. Currently this is done with a configured alias in FireBird_3_0\databases.conf file, which decides where to put the file for the alias nhibernate.

8. Open firebird.conf;
9. Ensure AuthClient, AuthServer and UserManager are set to Srp only:
AuthServer = Srp
AuthClient = Srp
UserManager = Srp
10. Ensure WireCrypt is set to Enabled.
WireCrypt = Enabled
11. Restart Firebird service.

The TestDatabaseSetup will create the database on the Teamcity agent. It will create the folder
D:\SqlData\Firebird if it is missing. Firebird is particularly sensitive to disk performances,
and D: is supposed to be local to the host, thus this choice.
For manual testing, take care of not creating it with inadequate acl on the file. This may happen
if you use ISQL with a connection string causing it to create it in embedded mode, without actually
using the server. Prefixing your path with "localhost:" should avoid that.

For tests performances, and since it is just an expandable test database, better disable forced writes.
Since those tests drop/create schema constantly, they are quite heavy on writes and this single setting
can have a six fold impact on their duration. For changing it, do:
a. Stop Firebird service.
b. From Firebird installation folder, run:
gfix -w async nhibernate -user SYSDBA
c. Restart Firebird service.
Binary file removed lib/teamcity/firebird/x64/fbembed.dll
Binary file not shown.
Binary file removed lib/teamcity/firebird/x64/firebird.msg
Binary file not shown.
Binary file removed lib/teamcity/firebird/x64/ib_util.dll
Binary file not shown.
Binary file removed lib/teamcity/firebird/x64/icudt30.dll
Binary file not shown.
Binary file removed lib/teamcity/firebird/x64/icuin30.dll
Binary file not shown.
Binary file removed lib/teamcity/firebird/x64/icuuc30.dll
Binary file not shown.
Binary file removed lib/teamcity/firebird/x64/msvcp80.dll
Binary file not shown.
Binary file removed lib/teamcity/firebird/x64/msvcr80.dll
Binary file not shown.
Binary file removed lib/teamcity/firebird/x86/fbembed.dll
Binary file not shown.
Binary file removed lib/teamcity/firebird/x86/firebird.msg
Binary file not shown.
Binary file removed lib/teamcity/firebird/x86/ib_util.dll
Binary file not shown.
Binary file removed lib/teamcity/firebird/x86/icudt30.dll
Binary file not shown.
Binary file removed lib/teamcity/firebird/x86/icuin30.dll
Binary file not shown.
Binary file removed lib/teamcity/firebird/x86/icuuc30.dll
Binary file not shown.
Binary file removed lib/teamcity/firebird/x86/msvcp80.dll
Binary file not shown.
Binary file removed lib/teamcity/firebird/x86/msvcr80.dll
Binary file not shown.
7 changes: 4 additions & 3 deletions src/NHibernate.Config.Templates/FireBird.cfg.xml
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,10 @@ for your own use before compile tests in VisualStudio.
<session-factory name="NHibernate.Test">
<property name="connection.driver_class">NHibernate.Driver.FirebirdClientDriver</property>
<property name="connection.connection_string">
Server=localhost;
Database=C:\nhibernate.fdb;
User=SYSDBA;Password=masterkey
DataSource=localhost;
Database=nhibernate;
User ID=SYSDBA;Password=masterkey;
MaxPoolSize=200;
</property>
<property name="show_sql">false</property>
<property name="dialect">NHibernate.Dialect.FirebirdDialect</property>
Expand Down
35 changes: 23 additions & 12 deletions src/NHibernate.Test/DriverTest/FirebirdClientDriverFixture.cs
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
using System.Data;
using System;
using System.Data;
using System.Data.Common;
using NHibernate.Driver;
using NHibernate.SqlCommand;
Expand All @@ -17,48 +18,58 @@ public class FirebirdClientDriverFixture
public void ConnectionPooling_OpenThenCloseThenOpenAnotherOne_OnlyOneConnectionIsPooled()
{
MakeDriver();

_driver.ClearPool(_connectionString);

var allreadyEstablished = GetEstablishedConnections();

var connection1 = MakeConnection();
var connection2 = MakeConnection();

//open first connection
connection1.Open();
VerifyCountOfEstablishedConnectionsIs(1);
VerifyCountOfEstablishedConnectionsIs(allreadyEstablished + 1, "After first open");

//return it to the pool
connection1.Close();
VerifyCountOfEstablishedConnectionsIs(1);
VerifyCountOfEstablishedConnectionsIs(allreadyEstablished + 1, "After first close");

//open the second connection
connection2.Open();
VerifyCountOfEstablishedConnectionsIs(1);
VerifyCountOfEstablishedConnectionsIs(allreadyEstablished + 1, "After second open");

//return it to the pool
connection2.Close();
VerifyCountOfEstablishedConnectionsIs(1);
VerifyCountOfEstablishedConnectionsIs(allreadyEstablished + 1, "After second close");
}

[Test]
public void ConnectionPooling_OpenThenCloseTwoAtTheSameTime_TowConnectionsArePooled()
{
MakeDriver();

_driver.ClearPool(_connectionString);

var allreadyEstablished = GetEstablishedConnections();

var connection1 = MakeConnection();
var connection2 = MakeConnection();

//open first connection
connection1.Open();
VerifyCountOfEstablishedConnectionsIs(1);
VerifyCountOfEstablishedConnectionsIs(allreadyEstablished + 1, "After first open");

//open second one
connection2.Open();
VerifyCountOfEstablishedConnectionsIs(2);
VerifyCountOfEstablishedConnectionsIs(allreadyEstablished + 2, "After second open");

//return connection1 to the pool
connection1.Close();
VerifyCountOfEstablishedConnectionsIs(2);
VerifyCountOfEstablishedConnectionsIs(allreadyEstablished + 2, "After first close");

//return connection2 to the pool
connection2.Close();
VerifyCountOfEstablishedConnectionsIs(2);
VerifyCountOfEstablishedConnectionsIs(allreadyEstablished + 2, "After second close");
}

[Test]
Expand Down Expand Up @@ -163,10 +174,10 @@ private DbConnection MakeConnection()
return result;
}

private void VerifyCountOfEstablishedConnectionsIs(int expectedCount)
private void VerifyCountOfEstablishedConnectionsIs(int expectedCount, string step)
{
var physicalConnections = GetEstablishedConnections();
Assert.That(physicalConnections, Is.EqualTo(expectedCount));
Assert.That(physicalConnections, Is.EqualTo(expectedCount), step);
}

private int GetEstablishedConnections()
Expand All @@ -178,7 +189,7 @@ private int GetEstablishedConnections()
using (var cmd = conn.CreateCommand())
{
cmd.CommandText = "select count(*) from mon$attachments where mon$attachment_id <> current_connection";
return (int)cmd.ExecuteScalar();
return Convert.ToInt32(cmd.ExecuteScalar());
}
}
}
Expand Down
8 changes: 8 additions & 0 deletions src/NHibernate.Test/Hql/HQLFunctions.cs
Original file line number Diff line number Diff line change
Expand Up @@ -733,6 +733,14 @@ public void Cast()
if (!ex.InnerException.Message.StartsWith("ORA-00979"))
throw;
}
else if (Dialect is FirebirdDialect)
{
string msgToCheck =
"not contained in either an aggregate function or the GROUP BY clause";
// This test raises an exception in Firebird for an unknown reason.
if (!ex.InnerException.Message.Contains(msgToCheck))
throw;
}
else
{
string msgToCheck =
Expand Down
21 changes: 2 additions & 19 deletions src/NHibernate.Test/NHSpecificTest/NH1908ThreadSafety/Fixture.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using NHibernate.Util;
using NHibernate.Dialect;
using NUnit.Framework;

namespace NHibernate.Test.NHSpecificTest.NH1908ThreadSafety
Expand All @@ -12,29 +12,12 @@ public class Fixture : BugTestCase
{
protected override bool AppliesTo(Dialect.Dialect dialect)
{
return !(dialect is Dialect.Oracle8iDialect);
return !(dialect is Oracle8iDialect);
// Oracle sometimes causes: ORA-12520: TNS:listener could not find available handler for requested type of server
// Following links bizarrely suggest it's an Oracle limitation under load:
// http://www.orafaq.com/forum/t/60019/2/ & http://www.ispirer.com/wiki/sqlways/troubleshooting-guide/oracle/import/tns_listener
}

protected override void OnTearDown()
{
base.OnTearDown();

if (!(Dialect is Dialect.FirebirdDialect))
return;

// Firebird will pool each connection created during the test and will not drop the created tables
// which will result in other tests failing when they try to create tables with same name
// By clearing the connection pool the tables will get dropped. This is done by the following code.
var fbConnectionType = ReflectHelper.TypeFromAssembly("FirebirdSql.Data.FirebirdClient.FbConnection", "FirebirdSql.Data.FirebirdClient", false);
var clearPool = fbConnectionType.GetMethod("ClearPool");
var sillyConnection = Sfi.ConnectionProvider.GetConnection();
clearPool.Invoke(null, new object[] { sillyConnection });
Sfi.ConnectionProvider.CloseConnection(sillyConnection);
}

[Test]
public void UsingFiltersIsThreadSafe()
{
Expand Down
30 changes: 20 additions & 10 deletions src/NHibernate.Test/NHSpecificTest/NH3374/FixtureByCode.cs
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
using System.Linq;
using NHibernate.Cfg.MappingSchema;
using NHibernate.Linq;
using NHibernate.Cfg.MappingSchema;
using NHibernate.Mapping.ByCode;
using NUnit.Framework;

Expand All @@ -15,9 +13,11 @@ protected override HbmMapping GetMappings()
mapper.Class<Document>(rc =>
{
rc.Id(x => x.Id, idMapper => idMapper.Generator(Generators.Identity));
rc.ManyToOne(x => x.Blob, m =>
rc.ManyToOne(x => x.Blob,
m =>
{
m.Cascade(Mapping.ByCode.Cascade.All);
m.Column("`Blob`");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the correct way to do this is to mark "blob" as a keyword for Firebird dialect.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

	protected virtual void RegisterKeywords()
	{
		RegisterKeyword("date");
	}

Maybe a lot more are missing...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And does not work anyway. Test failing again.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those keywords are seldom used in the code base. Only the Template class uses them, and not for quoting.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why "date"? it's "blob"

Copy link
Member Author

@fredericDelaporte fredericDelaporte Jun 6, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is the current only registered word. For my local test I have added "blob". But that has no effect on column quoting.

});
rc.Property(x => x.Name);
});
Expand All @@ -30,21 +30,31 @@ protected override HbmMapping GetMappings()
y.Length(int.MaxValue);
y.Lazy(true);
});
map.Table("`Blob`");
});

return mapper.CompileMappingForAllExplicitlyAddedEntities();
}

private int _blobId;
private int _docId;

protected override void OnSetUp()
{
using (ISession session = OpenSession())
using (ITransaction transaction = session.BeginTransaction())
{
var e1 = new Document { Name = "Bob" };
e1.Blob = new Blob { Bytes = new byte[] { 1, 2, 3 } };
var e1 = new Document
{
Name = "Bob",
Blob = new Blob {Bytes = new byte[] {1, 2, 3}}
};

session.Save(e1);

session.Flush();

_blobId = e1.Blob.Id;
_docId = e1.Id;
transaction.Commit();
}
}
Expand All @@ -71,7 +81,7 @@ public void TestNoTargetException()
document.Blob = blob;

using (ISession session = OpenSession())
using (ITransaction transaction = session.BeginTransaction())
using (session.BeginTransaction())
{
session.Merge(document);
}
Expand All @@ -82,7 +92,7 @@ private Blob LoadDetachedBlob()
using (ISession session = OpenSession())
using (session.BeginTransaction())
{
var blob = session.Get<Blob>(1);
var blob = session.Get<Blob>(_blobId);
NHibernateUtil.Initialize(blob.Bytes);
return blob;
}
Expand All @@ -93,7 +103,7 @@ private Document LoadDetachedEntity()
using (ISession session = OpenSession())
using (session.BeginTransaction())
{
return session.Get<Document>(1);
return session.Get<Document>(_docId);
}
}
}
Expand Down
13 changes: 12 additions & 1 deletion src/NHibernate.Test/TestCase.cs
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,9 @@
using NHibernate.Tool.hbm2ddl;
using NHibernate.Type;
using NUnit.Framework;
using NHibernate.Hql.Ast.ANTLR;
using NUnit.Framework.Interfaces;
using System.Text;
using NHibernate.Driver;

namespace NHibernate.Test
{
Expand Down Expand Up @@ -269,6 +269,17 @@ protected virtual void CreateSchema()

protected virtual void DropSchema()
{
if (Sfi.ConnectionProvider.Driver is FirebirdClientDriver fbDriver)
{
// Firebird will pool each connection created during the test and will marked as used any table
// referenced by queries. It will at best delays those tables drop until connections are actually
// closed, or immediately fail dropping them.
// This results in other tests failing when they try to create tables with same name.
// By clearing the connection pool the tables will get dropped. This is done by the following code.
// Moved from NH1908 test case, contributed by Amro El-Fakharany.
fbDriver.ClearPool(null);
}

new SchemaExport(cfg).Drop(OutputDdl, true);
}

Expand Down
8 changes: 4 additions & 4 deletions src/NHibernate.TestDatabaseSetup/TestDatabaseSetup.cs
Original file line number Diff line number Diff line change
Expand Up @@ -102,17 +102,17 @@ private static void SetupSqlServerOdbc(Cfg.Configuration cfg)

private static void SetupFirebird(Cfg.Configuration cfg)
{
Directory.CreateDirectory(@"D:\SqlData\Firebird");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like references to a specific folder. I think we need to use a local (relative) folder. Building with VS and NAnt+MSBuild will produce different results. When building with NAnt the output folder would be /build/version/net40 (or something like this). Building with VS will produce tests assemblies in different folders (/src/NHibernate.Test/bin/Debug-2.0/, /src/NHibernate.TestDatabaseSetup/bin/Debug-2.0/, etc). We need to choose some common folder, probably "curent-test-configuration", or a repository root.

I think we shall/should parse connection string to find where the file sits.

Copy link
Member Author

@fredericDelaporte fredericDelaporte Jun 8, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The connection string does currently only hold the alias nhibernate. The actual path is in databases.conf file, in Firebird installation directory. Yes this CreateDirectory code line is a crutch for accounting the possible loss of D: content. The server do not create the directory itself. We may instead just drop the db on the root, D:\, avoiding this above line of code.

When using a database server, it does not really make sens for me to locate the database relatively to the client, that is usually the server choice. Firebird allows to specify the database location from the connection string, which is a strange model for a server, not at all common. So we may do that anyway, by detecting where the test assemblies are then tweaking the connection string for telling the server where to put the file. But that would depart from what is and can be done for other actual servers like Oracle, Sql-Server, PostgreSql, which gives no option to the client about where to locate the database.
And we would have to do that both in TestDatabaseSetup and the test project.

Otherwise we should go back to testing Firebird in embedded mode, with all its additional binaries and apparently some new behaviors in v3 requiring a dedicated dialect. (There are seven hundred failing tests when running in embedded mode with Firebird v3, most solved by redefining the boolean type as have done Nathan. I have not yet investigated the other cases, since I do not consider we should test Firebird as an embedded server.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem, that people might not necessarily have a "D:" drive, and this would fail. I usually use TestDb to create a fresh copy of a database (including Firebird) myself.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I may just locate the file on d:\ root if that prove to be any better than putting it within Firebird installation folder. That would avoid that line. Actually I was seeing TestDatabaseSetup as a Teamcity exclusive thing, managing myself my own local db. So I would avoid to do such changes now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It still be executed if you run from ShowBuildMenu (release option, for ex)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I have missed that, sorry.

var connStr = cfg.Properties[Cfg.Environment.ConnectionString];
try
{
if (File.Exists("NHibernate.fdb"))
File.Delete("NHibernate.fdb");
FbConnection.DropDatabase(connStr);
}
catch (Exception e)
{
Console.WriteLine(e);
}

FbConnection.CreateDatabase("Database=NHibernate.fdb;ServerType=1");
FbConnection.CreateDatabase(connStr, forcedWrites:false);
}

private static void SetupSqlServerCe(Cfg.Configuration cfg)
Expand Down
Loading