Implementing Multiple Memory Orders

[MadAlex1997]

If we want to match features from numpy, and ensure wide interoperability, we need to implement at least C and F ordering.

Problems:

• SIMD loads will get different data so SIMD load and store dependent functions would need to be modified and either have two versions or some other way to ensure correct behavior
• Ops between C and F order arrays currently would not work and we would have to implement checks in each function to confirm order match.

Proposed Solution

• Implement an order parameter.
• Make a MemoryOrder trait that Order structs must conform to.
• Implement COrder and FOrder structs.
• Have the parts of the code that are affected by the order contained in the Order structs.
• Propigate this across the entire library.

Benefits:

• Makes memory order type level checked.
• Enables future Orders to be defined (might be useful for GPU).
• Very explicit and carries its implementation rather than having an if statement check in every function.

Downsides:

• A lot of work.
• Probably not going to be ready for v0.1

I think we should wait until after v0.1 to implement multiple memory ordering.

Thoughts, on the proposal, any other ideas?

[shivasankarka]

I have implemented a Column major ('F') NDArray before switching to our current Row major ('C') NDArray, so it won't take long to implement the column major arrays.

My thoughts:

1. I agree with implementing 'C' and 'F' order structs, that will probably be a cleaner version to maintain in the future.
2. I too believe that this should be added in v0.2 or later. We should concentrate on other basic functionalities in v0.1 which can be easily extended to Column major.

Proposal for v0.1

• A clean wrapper function for numpy.save, numpy.load with all arguments, until we figure out our own file types in later version (like .npy)
• Work on creating proper suite of test functions / test protocols that we can easily expand upon as we add features. We could look into mojo stdlib tests for reference.

Proposal for v0.1 onwards

• Broadcasting: This is a very powerful feature that lets user work on arrays with different sizes.
• Compile time: Slowly start moving basic NDArray information to compile time - Calculate it in compile time and pass it as parameters.
• Better memory management:
  • Currently we pass the NDArray instance to functions as input arguments, we should change our functions to take DTypePointer as the input instead.
  • In conjunction with above point, we could pass every other NDArray information such as NDArrayShape & NDArrayStrides as compile time parameters. This reduces quite a lot of overhead.
• Linear algebra: Currently our linalg has less implementations, we should start implementing basic algorithms for methods like solve, eig etc and later look into adding more algorithms.

[forfudan]

Here is an example when I incorporated the order argument when initializing an ND array #37. The order argument will only impact the calculation of the strides at the initialization. The strides will impact the get_index function, which translate a multidimensional indices into a single index of the data buffer.

The test uses two matrices: A 200x100 and B 100x 200, both saved in both "C" and "F" order. They are multiplied 10000 times to get the execution time, using both matmul and Matmul paralelled tiled unrolling, and using CxC, CxF, FxF settings. So there will be 6 tests in total.

Array1
[[      1       2       3       ...     98      99      100     ]
 [      101     102     103     ...     198     199     200     ]
 [      201     202     203     ...     298     299     300     ]
...
 [      19701   19702   19703   ...     19798   19799   19800   ]
 [      19801   19802   19803   ...     19898   19899   19900   ]
 [      19901   19902   19903   ...     19998   19999   20000   ]]
Shape: [200, 100]  DType: int32

Array2
[[      1       2       3       ...     198     199     200     ]
 [      201     202     203     ...     398     399     400     ]
 [      401     402     403     ...     598     599     600     ]
...
 [      19401   19402   19403   ...     19598   19599   19600   ]
 [      19601   19602   19603   ...     19798   19799   19800   ]
 [      19801   19802   19803   ...     19998   19999   20000   ]]
Shape: [100, 200]  DType: int32
==================================================

Matmul CxC
0.0013327014999999999 s

[[      66665050        66670100        66675150        ...     67659900        67664950        67670000        ]
 [      165675050       165690100       165705150       ...     168639900       168654950       168670000       ]
 [      264685050       264710100       264735150       ...     269619900       269644950       269670000       ]
...
 [      -1903201430     -1901226380     -1899251330     ...     -1514116580     -1512141530     -1510166480     ]
 [      -1804191430     -1802206380     -1800221330     ...     -1413136580     -1411151530     -1409166480     ]
 [      -1705181430     -1703186380     -1701191330     ...     -1312156580     -1310161530     -1308166480     ]]
Shape: [200, 200]  DType: int32

==================================================
Matmul CxF
0.0014273411 s

[[      66665050        66670100        66675150        ...     67659900        67664950        67670000        ]
 [      165675050       165690100       165705150       ...     168639900       168654950       168670000       ]
 [      264685050       264710100       264735150       ...     269619900       269644950       269670000       ]
...
 [      -1903201430     -1901226380     -1899251330     ...     -1514116580     -1512141530     -1510166480     ]
 [      -1804191430     -1802206380     -1800221330     ...     -1413136580     -1411151530     -1409166480     ]
 [      -1705181430     -1703186380     -1701191330     ...     -1312156580     -1310161530     -1308166480     ]]
Shape: [200, 200]  DType: int32

==================================================
Matmul FxF
0.0013825701999999999 s

[[      66665050        66670100        66675150        ...     67659900        67664950        67670000        ]
 [      165675050       165690100       165705150       ...     168639900       168654950       168670000       ]
 [      264685050       264710100       264735150       ...     269619900       269644950       269670000       ]
...
 [      -1903201430     -1901226380     -1899251330     ...     -1514116580     -1512141530     -1510166480     ]
 [      -1804191430     -1802206380     -1800221330     ...     -1413136580     -1411151530     -1409166480     ]
 [      -1705181430     -1703186380     -1701191330     ...     -1312156580     -1310161530     -1308166480     ]]
Shape: [200, 200]  DType: int32

==================================================
Matmul paralelled tiled unrolling CxC
0.0015150579000000001 s

[[      66665050        66670100        66675150        ...     67659900        67664950        67670000        ]
 [      165675050       165690100       165705150       ...     168639900       168654950       168670000       ]
 [      264685050       264710100       264735150       ...     269619900       269644950       269670000       ]
...
 [      -1903201430     -1901226380     -1899251330     ...     -1514116580     -1512141530     -1510166480     ]
 [      -1804191430     -1802206380     -1800221330     ...     -1413136580     -1411151530     -1409166480     ]
 [      -1705181430     -1703186380     -1701191330     ...     -1312156580     -1310161530     -1308166480     ]]
Shape: [200, 200]  DType: int32

==================================================
Matmul paralelled tiled unrolling CxF
0.0014466614999999999 s

[[      66665050        66670100        66675150        ...     67659900        67664950        67670000        ]
 [      165675050       165690100       165705150       ...     168639900       168654950       168670000       ]
 [      264685050       264710100       264735150       ...     269619900       269644950       269670000       ]
...
 [      -1903201430     -1901226380     -1899251330     ...     -1514116580     -1512141530     -1510166480     ]
 [      -1804191430     -1802206380     -1800221330     ...     -1413136580     -1411151530     -1409166480     ]
 [      -1705181430     -1703186380     -1701191330     ...     -1312156580     -1310161530     -1308166480     ]]
Shape: [200, 200]  DType: int32

==================================================
Matmul paralelled tiled unrolling FxF
0.0019108816999999999 s

[[      66665050        66670100        66675150        ...     67659900        67664950        67670000        ]
 [      165675050       165690100       165705150       ...     168639900       168654950       168670000       ]
 [      264685050       264710100       264735150       ...     269619900       269644950       269670000       ]
...
 [      -1903201430     -1901226380     -1899251330     ...     -1514116580     -1512141530     -1510166480     ]
 [      -1804191430     -1802206380     -1800221330     ...     -1413136580     -1411151530     -1409166480     ]
 [      -1705181430     -1703186380     -1701191330     ...     -1312156580     -1310161530     -1308166480     ]]
Shape: [200, 200]  DType: int32

[forfudan]

Another example is to compare float 32 between naive matmul and paralleled matmul.

Initialize an ND arrays

[[      -1.2397359609603882     1.6120094060897827      1.3828016519546509      ...     0.17709298431873322     -0.82231569290161133    0.32886722683906555     ]
 [      -1.0566920042037964     -1.3322204351425171     0.12518756091594696     ...     -0.78803342580795288    0.028025612235069275    -1.4970443248748779     ]
 [      -0.89967185258865356    0.17051012814044952     -1.0652971267700195     ...     1.4729944467544556      -0.17861346900463104    -0.60788214206695557    ]
...
 [      0.15020164847373962     -0.92272746562957764    -1.5714602470397949     ...     0.73758959770202637     -2.2215306758880615     -0.69156730175018311    ]
 [      -0.57092255353927612    -1.1619268655776978     0.44973188638687134     ...     0.44896170496940613     -0.42239445447921753    -0.70498615503311157    ]
 [      -0.62394857406616211    0.035806018859148026    -1.3667380809783936     ...     -2.5047941207885742     -0.45198026299476624    1.0350886583328247      ]]
Shape: [200, 100]  DType: float32


[[      -1.2397359609603882     1.6120094060897827      1.3828016519546509      ...     0.17709298431873322     -0.82231569290161133    0.32886722683906555     ]
 [      -1.0566920042037964     -1.3322204351425171     0.12518756091594696     ...     -0.78803342580795288    0.028025612235069275    -1.4970443248748779     ]
 [      -0.89967185258865356    0.17051012814044952     -1.0652971267700195     ...     1.4729944467544556      -0.17861346900463104    -0.60788214206695557    ]
...
 [      0.15020164847373962     -0.92272746562957764    -1.5714602470397949     ...     0.73758959770202637     -2.2215306758880615     -0.69156730175018311    ]
 [      -0.57092255353927612    -1.1619268655776978     0.44973188638687134     ...     0.44896170496940613     -0.42239445447921753    -0.70498615503311157    ]
 [      -0.62394857406616211    0.035806018859148026    -1.3667380809783936     ...     -2.5047941207885742     -0.45198026299476624    1.0350886583328247      ]]
Shape: [200, 100]  DType: float32

==================================================
Matmul CxC
0.00063080000000000005 s

[[      -8.0847139358520508     -11.96202278137207      -1.5802438259124756     ...     2.0025465488433838      8.7340116500854492      3.1775879859924316      ]
 [      18.910671234130859      -0.26866933703422546    0.51059097051620483     ...     5.807584285736084       6.9785099029541016      -4.3984193801879883     ]
 [      -7.1395716667175293     -6.6501965522766113     -17.403676986694336     ...     -5.0788745880126953     13.05000114440918       0.43890839815139771     ]
...
 [      -2.2573254108428955     2.3464453220367432      9.5264759063720703      ...     8.740142822265625       12.814759254455566      3.7495174407958984      ]
 [      0.35877308249473572     6.4928913116455078      -9.1665258407592773     ...     1.5036945343017578      0.67588865756988525     10.788858413696289      ]
 [      -11.922613143920898     0.2933214008808136      7.7781648635864258      ...     -4.6804723739624023     -4.6456022262573242     5.7972369194030762      ]]
Shape: [200, 100]  DType: float32

==================================================
Matmul CxF
0.00063624800000000007 s

[[      -8.0847139358520508     -11.96202278137207      -1.5802438259124756     ...     2.0025465488433838      8.7340116500854492      3.1775879859924316      ]
 [      18.910671234130859      -0.26866933703422546    0.51059097051620483     ...     5.807584285736084       6.9785099029541016      -4.3984193801879883     ]
 [      -7.1395716667175293     -6.6501965522766113     -17.403676986694336     ...     -5.0788745880126953     13.05000114440918       0.43890839815139771     ]
...
 [      -2.2573254108428955     2.3464453220367432      9.5264759063720703      ...     8.740142822265625       12.814759254455566      3.7495174407958984      ]
 [      0.35877308249473572     6.4928913116455078      -9.1665258407592773     ...     1.5036945343017578      0.67588865756988525     10.788858413696289      ]
 [      -11.922613143920898     0.2933214008808136      7.7781648635864258      ...     -4.6804723739624023     -4.6456022262573242     5.7972369194030762      ]]
Shape: [200, 100]  DType: float32

==================================================
Matmul FxF
0.00066282700000000006 s

[[      -8.0847139358520508     -11.96202278137207      -1.5802438259124756     ...     2.0025465488433838      8.7340116500854492      3.1775879859924316      ]
 [      18.910671234130859      -0.26866933703422546    0.51059097051620483     ...     5.807584285736084       6.9785099029541016      -4.3984193801879883     ]
 [      -7.1395716667175293     -6.6501965522766113     -17.403676986694336     ...     -5.0788745880126953     13.05000114440918       0.43890839815139771     ]
...
 [      -2.2573254108428955     2.3464453220367432      9.5264759063720703      ...     8.740142822265625       12.814759254455566      3.7495174407958984      ]
 [      0.35877308249473572     6.4928913116455078      -9.1665258407592773     ...     1.5036945343017578      0.67588865756988525     10.788858413696289      ]
 [      -11.922613143920898     0.2933214008808136      7.7781648635864258      ...     -4.6804723739624023     -4.6456022262573242     5.7972369194030762      ]]
Shape: [200, 100]  DType: float32

==================================================
Matmul paralelled tiled unrolling CxC
0.00066195700000000002 s

[[      -8.0847139358520508     -11.96202278137207      -1.5802438259124756     ...     2.0025465488433838      8.7340116500854492      3.1775879859924316      ]
 [      18.910671234130859      -0.26866933703422546    0.51059097051620483     ...     5.807584285736084       6.9785099029541016      -4.3984193801879883     ]
 [      -7.1395716667175293     -6.6501965522766113     -17.403676986694336     ...     -5.0788745880126953     13.05000114440918       0.43890839815139771     ]
...
 [      -2.2573254108428955     2.3464453220367432      9.5264759063720703      ...     8.740142822265625       12.814759254455566      3.7495174407958984      ]
 [      0.35877308249473572     6.4928913116455078      -9.1665258407592773     ...     1.5036945343017578      0.67588865756988525     10.788858413696289      ]
 [      -11.922613143920898     0.2933214008808136      7.7781648635864258      ...     -4.6804723739624023     -4.6456022262573242     5.7972369194030762      ]]
Shape: [200, 100]  DType: float32

==================================================
Matmul paralelled tiled unrolling CxF
0.00066143399999999992 s

[[      -8.0847139358520508     -11.96202278137207      -1.5802438259124756     ...     2.0025465488433838      8.7340116500854492      3.1775879859924316      ]
 [      18.910671234130859      -0.26866933703422546    0.51059097051620483     ...     5.807584285736084       6.9785099029541016      -4.3984193801879883     ]
 [      -7.1395716667175293     -6.6501965522766113     -17.403676986694336     ...     -5.0788745880126953     13.05000114440918       0.43890839815139771     ]
...
 [      -2.2573254108428955     2.3464453220367432      9.5264759063720703      ...     8.740142822265625       12.814759254455566      3.7495174407958984      ]
 [      0.35877308249473572     6.4928913116455078      -9.1665258407592773     ...     1.5036945343017578      0.67588865756988525     10.788858413696289      ]
 [      -11.922613143920898     0.2933214008808136      7.7781648635864258      ...     -4.6804723739624023     -4.6456022262573242     5.7972369194030762      ]]
Shape: [200, 100]  DType: float32

==================================================
Matmul paralelled tiled unrolling FxF
0.00066363099999999992 s

[[      -8.0847139358520508     -11.96202278137207      -1.5802438259124756     ...     2.0025465488433838      8.7340116500854492      3.1775879859924316      ]
 [      18.910671234130859      -0.26866933703422546    0.51059097051620483     ...     5.807584285736084       6.9785099029541016      -4.3984193801879883     ]
 [      -7.1395716667175293     -6.6501965522766113     -17.403676986694336     ...     -5.0788745880126953     13.05000114440918       0.43890839815139771     ]
...
 [      -2.2573254108428955     2.3464453220367432      9.5264759063720703      ...     8.740142822265625       12.814759254455566      3.7495174407958984      ]
 [      0.35877308249473572     6.4928913116455078      -9.1665258407592773     ...     1.5036945343017578      0.67588865756988525     10.788858413696289      ]
 [      -11.922613143920898     0.2933214008808136      7.7781648635864258      ...     -4.6804723739624023     -4.6456022262573242     5.7972369194030762      ]]
Shape: [200, 100]  DType: float32

==================================================
(base) ZHU@MBA-Yuhao-Zhu NuMojo-Examples-and-Benchmarks % mojo run -I "../NuMojo" ndarray.mojo
264036340280000
Initialize an ND arrays

[[      -1.2397359609603882     1.6120094060897827      1.3828016519546509      ...     0.17709298431873322     -0.82231569290161133    0.32886722683906555     ]
 [      -1.0566920042037964     -1.3322204351425171     0.12518756091594696     ...     -0.78803342580795288    0.028025612235069275    -1.4970443248748779     ]
 [      -0.89967185258865356    0.17051012814044952     -1.0652971267700195     ...     1.4729944467544556      -0.17861346900463104    -0.60788214206695557    ]
...
 [      0.15020164847373962     -0.92272746562957764    -1.5714602470397949     ...     0.73758959770202637     -2.2215306758880615     -0.69156730175018311    ]
 [      -0.57092255353927612    -1.1619268655776978     0.44973188638687134     ...     0.44896170496940613     -0.42239445447921753    -0.70498615503311157    ]
 [      -0.62394857406616211    0.035806018859148026    -1.3667380809783936     ...     -2.5047941207885742     -0.45198026299476624    1.0350886583328247      ]]
Shape: [200, 100]  DType: float32


[[      -1.2397359609603882     1.6120094060897827      1.3828016519546509      ...     0.17709298431873322     -0.82231569290161133    0.32886722683906555     ]
 [      -1.0566920042037964     -1.3322204351425171     0.12518756091594696     ...     -0.78803342580795288    0.028025612235069275    -1.4970443248748779     ]
 [      -0.89967185258865356    0.17051012814044952     -1.0652971267700195     ...     1.4729944467544556      -0.17861346900463104    -0.60788214206695557    ]
...
 [      0.15020164847373962     -0.92272746562957764    -1.5714602470397949     ...     0.73758959770202637     -2.2215306758880615     -0.69156730175018311    ]
 [      -0.57092255353927612    -1.1619268655776978     0.44973188638687134     ...     0.44896170496940613     -0.42239445447921753    -0.70498615503311157    ]
 [      -0.62394857406616211    0.035806018859148026    -1.3667380809783936     ...     -2.5047941207885742     -0.45198026299476624    1.0350886583328247      ]]
Shape: [200, 100]  DType: float32

==================================================
Matmul naive CxC
0.0077978319999999993 s

[[      -8.0847148895263672     -11.96202278137207      -1.5802450180053711     ...     2.0025463104248047      8.7340126037597656      3.1775884628295898      ]
 [      18.910671234130859      -0.26866775751113892    0.51059114933013916     ...     5.8075847625732422      6.9785099029541016      -4.3984203338623047     ]
 [      -7.1395711898803711     -6.6501979827880859     -17.403676986694336     ...     -5.0788745880126953     13.050002098083496      0.4389081597328186      ]
...
 [      -2.2573251724243164     2.3464457988739014      9.5264768600463867      ...     8.740142822265625       12.814759254455566      3.7495179176330566      ]
 [      0.35877272486686707     6.4928913116455078      -9.1665239334106445     ...     1.5036945343017578      0.6758882999420166      10.788859367370605      ]
 [      -11.922611236572266     0.2933221161365509      7.7781648635864258      ...     -4.6804733276367188     -4.6456012725830078     5.7972378730773926      ]]
Shape: [200, 100]  DType: float32

==================================================
Matmul naive CxF
0.0076951190000000003 s

[[      -8.0847148895263672     -11.96202278137207      -1.5802450180053711     ...     2.0025463104248047      8.7340126037597656      3.1775884628295898      ]
 [      18.910671234130859      -0.26866775751113892    0.51059114933013916     ...     5.8075847625732422      6.9785099029541016      -4.3984203338623047     ]
 [      -7.1395711898803711     -6.6501979827880859     -17.403676986694336     ...     -5.0788745880126953     13.050002098083496      0.4389081597328186      ]
...
 [      -2.2573251724243164     2.3464457988739014      9.5264768600463867      ...     8.740142822265625       12.814759254455566      3.7495179176330566      ]
 [      0.35877272486686707     6.4928913116455078      -9.1665239334106445     ...     1.5036945343017578      0.6758882999420166      10.788859367370605      ]
 [      -11.922611236572266     0.2933221161365509      7.7781648635864258      ...     -4.6804733276367188     -4.6456012725830078     5.7972378730773926      ]]
Shape: [200, 100]  DType: float32

==================================================
Matmul naive FxF
0.0077039460000000006 s

[[      -8.0847148895263672     -11.96202278137207      -1.5802450180053711     ...     2.0025463104248047      8.7340126037597656      3.1775884628295898      ]
 [      18.910671234130859      -0.26866775751113892    0.51059114933013916     ...     5.8075847625732422      6.9785099029541016      -4.3984203338623047     ]
 [      -7.1395711898803711     -6.6501979827880859     -17.403676986694336     ...     -5.0788745880126953     13.050002098083496      0.4389081597328186      ]
...
 [      -2.2573251724243164     2.3464457988739014      9.5264768600463867      ...     8.740142822265625       12.814759254455566      3.7495179176330566      ]
 [      0.35877272486686707     6.4928913116455078      -9.1665239334106445     ...     1.5036945343017578      0.6758882999420166      10.788859367370605      ]
 [      -11.922611236572266     0.2933221161365509      7.7781648635864258      ...     -4.6804733276367188     -4.6456012725830078     5.7972378730773926      ]]
Shape: [200, 100]  DType: float32

==================================================
Matmul paralelled tiled unrolling CxC
0.00066171800000000005 s

[[      -8.0847139358520508     -11.96202278137207      -1.5802438259124756     ...     2.0025465488433838      8.7340116500854492      3.1775879859924316      ]
 [      18.910671234130859      -0.26866933703422546    0.51059097051620483     ...     5.807584285736084       6.9785099029541016      -4.3984193801879883     ]
 [      -7.1395716667175293     -6.6501965522766113     -17.403676986694336     ...     -5.0788745880126953     13.05000114440918       0.43890839815139771     ]
...
 [      -2.2573254108428955     2.3464453220367432      9.5264759063720703      ...     8.740142822265625       12.814759254455566      3.7495174407958984      ]
 [      0.35877308249473572     6.4928913116455078      -9.1665258407592773     ...     1.5036945343017578      0.67588865756988525     10.788858413696289      ]
 [      -11.922613143920898     0.2933214008808136      7.7781648635864258      ...     -4.6804723739624023     -4.6456022262573242     5.7972369194030762      ]]
Shape: [200, 100]  DType: float32

==================================================
Matmul paralelled tiled unrolling CxF
0.00066179500000000007 s

[[      -8.0847139358520508     -11.96202278137207      -1.5802438259124756     ...     2.0025465488433838      8.7340116500854492      3.1775879859924316      ]
 [      18.910671234130859      -0.26866933703422546    0.51059097051620483     ...     5.807584285736084       6.9785099029541016      -4.3984193801879883     ]
 [      -7.1395716667175293     -6.6501965522766113     -17.403676986694336     ...     -5.0788745880126953     13.05000114440918       0.43890839815139771     ]
...
 [      -2.2573254108428955     2.3464453220367432      9.5264759063720703      ...     8.740142822265625       12.814759254455566      3.7495174407958984      ]
 [      0.35877308249473572     6.4928913116455078      -9.1665258407592773     ...     1.5036945343017578      0.67588865756988525     10.788858413696289      ]
 [      -11.922613143920898     0.2933214008808136      7.7781648635864258      ...     -4.6804723739624023     -4.6456022262573242     5.7972369194030762      ]]
Shape: [200, 100]  DType: float32

==================================================
Matmul paralelled tiled unrolling FxF
0.00066425700000000002 s

[[      -8.0847139358520508     -11.96202278137207      -1.5802438259124756     ...     2.0025465488433838      8.7340116500854492      3.1775879859924316      ]
 [      18.910671234130859      -0.26866933703422546    0.51059097051620483     ...     5.807584285736084       6.9785099029541016      -4.3984193801879883     ]
 [      -7.1395716667175293     -6.6501965522766113     -17.403676986694336     ...     -5.0788745880126953     13.05000114440918       0.43890839815139771     ]
...
 [      -2.2573254108428955     2.3464453220367432      9.5264759063720703      ...     8.740142822265625       12.814759254455566      3.7495174407958984      ]
 [      0.35877308249473572     6.4928913116455078      -9.1665258407592773     ...     1.5036945343017578      0.67588865756988525     10.788858413696289      ]
 [      -11.922613143920898     0.2933214008808136      7.7781648635864258      ...     -4.6804723739624023     -4.6456022262573242     5.7972369194030762      ]]
Shape: [200, 100]  DType: float32

[forfudan]

• Ops between C and F order arrays currently would not work and we would have to implement checks in each function to confirm order match.

I do not think that a matched order is needed. Actually CxF is an efficient way of doing multiplication to exploit the continuous memory allocation. The order information will be eventually reflected on the stride info.

Implement an order parameter.

I agree with this. Making the order known at compiled time is good. We can also add a function to change the order of an array.

Make a MemoryOrder trait that Order structs must conform to.
Implement COrder and FOrder structs.

It is good idea. Maybe we can also look into numpy and see how they differentiate the two orders.